# A Comprehensive Survey on Image Fusion: Which Approach Fits Which Need

Gwendal Bernardi, Godefroy Brisebarre, Sébastien Roman, Mohsen Ardabilian, Emmanuel Dellandrea

► To cite this version:

Gwendal Bernardi, Godefroy Brisebarre, Sébastien Roman, Mohsen Ardabilian, Emmanuel Dellandrea. A Comprehensive Survey on Image Fusion: Which Approach Fits Which Need. Information Fusion, 2025, 126, pp.103594. 10.1016/j.inffus.2025.103594. hal-05015011

HAL Id: hal-05015011

https://hal.science/hal-05015011v1

Submitted on 23 Jan 2026

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons CC BY 4.0 - Attribution - International License
<figure><img src="image_2.png" /></figure>

# A comprehensive survey on image fusion: Which approach fits which need

Gwendal Bernardi a,b ID,*, Godefroy Brisebarre a, Sébastien Roman a, Mohsen Ardabilian b ID,
Emmanuel Dellandrea b ID

<figure><img src="image_3.png" /></figure>

## ARTICLE INFO

**Keywords:**

Image fusion

Multi-view

Multi-modal

Multi-exposure

Multi-focus

Task-driven

Fusion category

## ABSTRACT

Image fusion is a fundamental task in computer vision that involves combining information from multiple images to produce a more informative and consistent representation. Once the relevant features are identified, they are fused to achieve specific application goals. The field of image fusion encompasses several categories, including multi-focus, multi-exposure, multi-modal, and multi-view fusion. Most state-of-the-art solutions focus on optimizing methods to address a specific fusion category (e.g., multi-view, multi-modal, multi-exposure, or multi-focus). However, some use cases require universal methods capable of handling all these challenges. While recent advancements, particularly in deep learning, have achieved remarkable results within individual categories, the growing need for general-purpose solutions across diverse fusion tasks calls for a broader perspective. This survey provides a comprehensive and unified review of image fusion techniques, systematically covering all four major categories. Special attention is given to deep learning-based methods, which have become dominant in recent years across various fusion types. A key contribution of this work is the integration of multi-view image fusion, often overlooked in prior surveys, with other fusion approaches. We introduce a novel taxonomy that distinguishes between mono-category methods, which target a single fusion domain, and multi-category methods, which are capable of addressing multiple fusion types. Particular emphasis is placed on generalist multi-category approaches, which handle cross-domain scenarios and represent a promising line of research. Additionally, this survey provides practical guidance through summaries of available datasets, evaluation metrics, and representative methods. By offering this structured overview and highlighting unexplored directions, the survey serves as both a foundational reference and a roadmap for future research on unified and adaptive deep learning-based image fusion techniques.

## 1. Introduction

Image fusion is a specialized subfield of computer vision. It involves extracting and aligning relevant features from multiple images to perform tasks that benefit from the combined information, these images provide. Performing a task on a single image might not yield satisfactory results. Therefore, the goal is to combine different images to increase the amount of available information, thus enhancing the system performance. Image fusion is used with the aim of augmenting and complementing the data. Thus, an effective image fusion method should accumulate the specific information from each image while eliminating redundant information. For example, in the context of using visible and infrared images for autonomous driving [1,2], image fusion proves to be crucial. Visible images provide information on

the texture of a person, a car, or a sign, helping to identify them. In contrast, infrared images offer information on object contours and help distinguish the foreground from the background, even under challenging imaging conditions. Using both types of images for object detection shows significant improvements compared to using a single image. Similarly, in the case of using images from multiple points of view for autonomous driving [3,4], image fusion allows comprehensive information around the vehicle or provides a dimensional perspective on various objects [5], aiding in spatial recognition.

Image fusion has emerged as an increasingly important field of research in recent years. To illustrate its growing relevance within the scientific community, Fig. 1 presents the annual number of publications related to image fusion from 2018 to 2024. These data were obtained

<footnote>ª Tiama, 215 chemin du Grand Revoyet, Saint-Genis Laval, 69230, France</footnote>

<footnote>b Ecole Centrale de Lyon, CNRS, INSA Lyon, Université Claude Bernard Lyon 1, Université Lumière Lyon 2, LIRIS, UMR5205, 36 avenue Guy de Collongue, 69130 Ecully, France</footnote>

<footnote>* Corresponding author at: Ecole Centrale de Lyon, CNRS, INSA Lyon, Université Claude Bernard Lyon 1, Université Lumière Lyon 2, LIRIS, UMR5205, 36 avenue Guy de Collongue, 69130 Ecully, France.</footnote>

<footnote>E-mail addresses: g.bernardi@tiama.com (G. Bernardi), g.brisebarre@tiama.com (G. Brisebarre), s.roman.ext@tiama.com (S. Roman), mohsen.ardabilian@ec-lyon.fr (M. Ardabilian), emmanuel.dellandrea@ec-lyon.fr (E. Dellandrea).</footnote>

<footnote>https://doi.org/10.1016/j.inffus.2025.103594</footnote>

<footnote>Received 20 March 2025; Received in revised form 20 July 2025; Accepted 30 July 2025</footnote>

<footnote>Available online 9 August 2025</footnote>

<footnote>1566-2535/© 2025 The Authors. Published by Elsevier B.V. This is an open access article under the CC BY license (http://creativecommons.org/licenses/by/4.0/).</footnote>
<figure><img src="image_4.png" /><figcaption>**Fig. 1.** The figure presents the yearly evolution of scholarly publications related to image fusion from 2018 to 2024. The data were collected by retrieving articles mentioning image fusion on Google Scholar and organizing them by year of publication. The data reveal a consistent and substantial increase in research output, rising from 7380 publications in 2018 to 17,300 in 2024. This upward trend underscores the growing scientific and industrial interest in image fusion technologies, which are pivotal in a variety of fields including remote sensing, medical imaging, surveillance, and computer vision. The steady rise in publication volume indicates that image fusion has become an increasingly prominent research domain, reflecting its expanding application potential and the increasing availability of complex multi-modal datasets.</figcaption></figure>

by querying Google Scholar for articles related to image fusion and categorizing the results by year of publication. This continuing upward trend highlights the need for a comprehensive and up-to-date survey on image fusion, which can serve as a valuable resource to help researchers navigate the expanding body of literature and identify key developments in the field.

Traditional image fusion methods relied on rule-based systems, employing expert-designed image processing algorithms. These “non-learning” methods offered the advantages of interpretability and explainability. Additionally, they did not impose data constraints as there is no learning involved. However, these methods are complex to develop and often lack generalization they are manually designed for very specific use cases. Examples of non-learning fusion methods include multi-scale transforms [6], sparse representation [7], hybrid [8] and subspace [9] methods. Nowadays, the most commonly used methods are learning-based. These methods have the advantage of offering generality and ease of implementation, but often require large amounts of data and are not always easily explainable or interpretable. There are several ways to categorize learning-based image fusion methods [10–14]. Classifying methods by type of learning allows the grouping of methods with similar functionalities, thereby better understanding the state-of-the-art in image fusion. Three types of learning are represented in image fusion methods [15]: supervised, unsupervised, and task-driven. Supervised methods are used when the ground truth is known to generate a fused image. Unsupervised methods aim to generate a fused image from multiple images without prior knowledge of ground truth, relying on customized cost functions or specific unsupervised architectures. Lastly, task-driven methods use images of different modalities to accomplish a specific task (such as classification, 2D or 3D object detection, segmentation, etc.). This task has a ground truth that enables the model to learn both the specific task and the image fusion model. The tasks performed within a task-driven architecture are Fusion Tasks. Unsupervised and supervised methods also carry out Fusion Tasks, but are more specific to their architecture. These tasks can include generating an image that fuses visible and infrared data, creating a well-exposed image of multi-exposure data, or constructing a 3D model from multiple multi-view images.

Multi-Exposure, Multi-Focus, and Multi-Modal are three commonly defined subsets of fusion categories within the field of image fusion [16]. Multi-exposure involves combining images with different lighting conditions to reconstruct an image with optimal exposure conditions. Multi-focus aims to merge images of the same scene taken

at different focal points, resulting in a single image where all planes are in focus. Finally, multi-modal refers to the fusion of images from different modalities, particularly from different light spectra, such as visible and infrared.

Another possibility in image fusion is the fusion of images taken from multiple different viewpoints, known as multi-view image fusion. In the various classical states of the art in image fusion [10–13], multi-view is never addressed. However, multi-view image fusion is indeed a form of image fusion, but has some differences compared to other categories. Typically, multi-view image fusion involves images that are more or less aligned and do not have significant angular differences between captures. In contrast, multi-view image fusion involves significant angular differences and covers a scene from many angles. This constraint necessitates the use of implicit or explicit three-dimensional representations, requiring specialized architectures.

Integrating multi-view fusion methods with other commonly presented techniques in surveys is not straightforward but rather complex. Given the constraints previously discussed, multi-view fusion may appear distinct, yet this survey aims to highlight the interconnections between various image fusion categories. Additionally, numerous applications may require both multi-view and multi-modal methods. For instance, in autonomous driving, vehicles may be equipped with multiple cameras operating across different light spectra, such as visible and infrared [1], as well as cameras with different viewing angles [5]. This scenario necessitates the use of multiple geometrically spaced cameras with different spectral sensitivities. Similarly, in the medical field, detecting diseases such as breast cancer often requires the use of images from different modalities and viewing angles [17], thus necessitating multi-category methods.

In recent years, numerous surveys have explored image fusion methods across a variety of contexts and application domains. Table 1 provides an overview of the main state-of-the-art contributions. However, most of these works adopt a specialized or restricted focus, either concentrating on a specific type of fusion such as multi-focus or multi-modal, or being limited to particular application areas like medical imaging or hyperspectral analysis. For example, one recent study highlights the contrast between model-driven and data-driven strategies in spectral fusion, but remains confined to hyperspectral and multispectral domains [18]. Other surveys focus on individual fusion tasks, including multi-exposure image fusion [20], multi-focus fusion [24], and multimodal medical image fusion [21], each offering detailed coverage within its respective subdomain.

While these contributions are thorough within their scope, they often fail to address the broad spectrum of fusion categories. Some works have attempted to provide a more global perspective by reviewing deep learning-based approaches across several types of fusion [11,22,23]. Nevertheless, these surveys generally maintain a strong focus on the more conventional categories, particularly multi-focus, multi-modal, and multi-exposure fusion. In contrast, multi-view fusion, which is gaining importance in fields such as remote sensing, robotics, and autonomous navigation, is frequently overlooked or absent. Even recent efforts that propose updated taxonomies and explore transformer-based architectures [15] offer limited discussion of multi-view fusion and provide minimal insights into general-purpose models capable of adapting across diverse fusion tasks.

This survey aims to fill these gaps by presenting a comprehensive and unified overview of all major image fusion categories, including multi-focus, multi-exposure, multi-modal, and multi-view. Special attention is given to multi-category and general-purpose fusion methods, which are often neglected in the current literature. To support this broader analysis, we introduce a structured taxonomy that organizes traditional, machine learning, and deep learning methods across all fusion types. This structure is intended to guide researchers and practitioners in selecting appropriate methods based on their specific application needs and technical constraints. We also compile comparative tables that summarize evaluation metrics, datasets, and
<figure><img src="image_5.png" /><figcaption>Fig. 2. Overview of image fusion approaches presented in this paper: The structure of this document begins by detailing mono-category image fusion methods (multi-focus, multi-exposure, multi-modal, and multi-view). The methods, along with metrics, datasets, and applications, will be examined to understand the challenges and requirements of each fusion category. A comparison of methods within each fusion category is then conducted to identify the most effective architectures or learning systems. Following this, multi-category methods, including both classical and generalist approaches, are explored. Experiments were conducted to compare multi-category methods against mono-category approaches. The findings provide a basis for identifying research opportunities and addressing challenges to advance the field.</figcaption></figure>

<table><caption>**Table 1**

Overview and scope of recent image fusion surveys.</caption><thead><tr><td>Survey</td><td>Description</td><td>Year</td></tr></thead><tbody><tr><td>Hyperspectral and multispectral image fusion: When model-driven meet data-driven strategies [18]</td><td>Specializes in hyperspectral and multispectral image fusion, contrasting model-driven and data-driven approaches.</td><td>2025</td></tr><tr><td>Image Fusion Survey: A Novel Taxonomy Integrating Transformer and Recent Approaches [15]</td><td>Focuses on multi-focus, multi-modal, and multi-exposure fusion, with a strong emphasis on recent deep learning techniques including transformers.</td><td>2024</td></tr><tr><td>Multi-modal image fusion: A systematic review [19]</td><td>Centers primarily on multispectral and multi-modal fusion methods.</td><td>2023</td></tr><tr><td>Current advances and future perspectives of image fusion: A comprehensive review [11]</td><td>Provides a broad overview with focus on multi-focus, multi-modal, and multi-exposure fusion, summarizing recent DL-based advancements.</td><td>2023</td></tr><tr><td>Multi-exposure image fusion techniques: A comprehensive review [20]</td><td>Dedicated specifically to multi-exposure fusion methods.</td><td>2022</td></tr><tr><td>A review on multimodal medical image fusion: Compendious analysis of medical modalities, multimodal databases, fusion techniques and quality metrics [21]</td><td>Focused on medical image fusion, covering databases, techniques, and evaluation protocols.</td><td>2022</td></tr><tr><td>A survey on deep multimodal learning for computer vision: advances, trends, applications, and datasets [13]</td><td>Reviews deep multimodal learning strategies, emphasizing fusion levels and categorization in computer vision applications.</td><td>2022</td></tr><tr><td>Image fusion meets deep learning: A survey and perspective [22]</td><td>Broad review on DL-based image fusion, particularly in multi-focus, multi-modal, and multi-exposure domains.</td><td>2021</td></tr><tr><td>Image Fusion Techniques: A Survey [23]</td><td>Covers traditional and some learning-based methods for multi-modal and multi-focus fusion.</td><td>2021</td></tr><tr><td>Multi-focus image fusion: A survey of the state of the art [24]</td><td>Focused exclusively on multi-focus fusion techniques.</td><td>2020</td></tr><tr><td>Fusion of medical imaging and electronic health records using deep learning: a systematic review and implementation guidelines [12]</td><td>Specializes in fusing medical images with electronic health records using DL approaches.</td><td>2020</td></tr><tr><td>A survey on region based image fusion methods [25]</td><td>Concentrated on region-based image fusion strategies.</td><td>2019</td></tr><tr><td>Multimodal machine learning: A survey and taxonomy [10]</td><td>Lays the foundational taxonomy of multi-modal learning, offering a broad and generalized perspective on data fusion.</td><td>2017</td></tr></tbody></table>
methodological characteristics from the literature, providing centralized access to resources that are often scattered or incomplete in previous surveys.

At the core of this work is a novel taxonomy that distinguishes between two main methodological groups. The first consists of mono-category image fusion methods, which are specifically developed for a single type of fusion, such as multi-focus, multi-exposure, multi-modal, or multi-view. The second group comprises multi-category image fusion methods, designed to handle multiple fusion types within a unified framework. Within this group, methods that address only the traditional categories of multi-focus, multi-exposure, and multi-modal fusion are referred to as classical multi-category methods. In contrast, those that also include multi-view fusion are considered more general and are referred to as generalist multi-category methods, as they aim to operate across a wider range of fusion tasks.

The objectives of this survey are multiple. The first aim is to present the most recent and notable methods in each area of image fusion, including multi-focus, multi-exposure, multi-modal, and multi-view fusion. This highlights not only the differences between these methods but also their similarities. The second aim is to provide a comprehensive overview of image fusion methods across these various fusion categories. The third objective is to introduce methods that can address all these fusion categories: Multi-category Image Fusion Methods. Although these methods are few, they have numerous applications in diverse fields. This presentation aims to open the discussion on the challenges and future development pathways for Multi-category Image Fusion Methods.

**Motivation and Contributions**-This survey makes several significant contributions to the field of image fusion:

* It presents a recent and comprehensive state-of-the-art review covering the main domains of image fusion, including multi-view, multi-modal, multi-exposure, and multi-focus fusion.

* It integrates multi-view image fusion into the broader landscape of image fusion categories, offering a unified and holistic perspective of the field.

* It provides a detailed description of fusion methods, helping researchers and practitioners identify suitable techniques based on specific problem contexts and application needs.

* It introduces a general taxonomy that classifies fusion methods across categories, and highlights emerging multi-category approaches, generalist fusion methods capable of addressing various fusion categories within a unified framework.

The remainder of this survey is structured as follows (See Fig. 2.). We begin by reviewing state-of-the-art methods for mono-category image fusion. Each category, including multi-focus, multi-exposure, multi-modal, and multi-view, is examined in detail, covering methodological principles, evaluation metrics, datasets, and real-world applications. The second part of the survey presents experimental comparisons between representative methods to support informed decision-making. We then examine multi-category fusion methods, divided into two subsections: classical multi-category approaches, which address the three main types, and generalist approaches, which aim to unify all categories. Finally, we discuss current challenges and future research directions for generalist methods, concluding with a summary of the key contributions of this work.

## 2. Mono-category image fusion methods

Mono-category fusion methods are applications dedicated to a single type of fusion category, such as multi-focus, multi-exposure, multi-modal, or multi-view fusion. The term “fusion category” refers to the nature of the input data and the specific problem being addressed. Each of these fusion categories will be detailed individually, highlighting the associated methods, metrics, and datasets used in the state-of-the-art, as well as the potential applications across various domains.

### 2.1. Multi-focus image fusion

Multi-focus image fusion encompasses a range of techniques aimed at extending the depth-of-field (DOF) of optical systems. By merging partially focused images of the same scene, these methods produce a single, all-in-focus image. This capability is vital in fields such as digital photography, optical microscopy, and integral imaging, where precise focus is critical for accurate analysis and interpretation. The concept and application of multi-focus image fusion have evolved significantly over the past three decades. The early methods for multi-focus image fusion primarily focused on spatial domain techniques [26–28], which involve directly manipulating the pixel values of the images to enhance focus. These methods were relatively simple and computationally less intensive but often struggled with issues related to noise and misalignment between images. As the field progressed, transform domain methods [29–31] gained prominence. These methods work by transforming the images into a different domain, such as the frequency domain, where the fusion process can be performed more effectively. This approach often results in better performance in terms of preserving important features and reducing artefacts. In recent years, the integration of deep learning techniques has revolutionized multi-focus image fusion. Deep learning models, particularly CNNs, have demonstrated superior capabilities in handling complex fusion tasks by learning intricate patterns and features from large datasets [24]. This has led to the development of more robust and efficient fusion algorithms that can produce higher quality all-in-focus images.

#### 2.1.1. Multi-focus methods

The method for multi-focus image fusion presented by Liu et al. [32] utilizes a CNN to automate the process of activity level measurement and fusion rule generation, which are traditionally handled separately and manually in image fusion tasks. The process begins with the detection of focus by inputting the source images into a pre-trained CNN model that generates a focus score map. This map, indicating the clarity information of the source images, is derived from the CNN’s evaluation of overlapping patches. Next, an initial segmentation transforms the focus score map into a binary focus map using a threshold. To ensure consistency, the binary map is refined through small region suppression and guided image filtering, resulting in the final decision map. The fusion of the source images is then achieved by applying a weighted average strategy pixel-by-pixel according to the final decision map. This CNN-based method simplifies the design process, improves fusion quality, and ensures computational efficiency, leveraging parallel computing for practical applications. By integrating activity level measurement and fusion rule generation within the CNN, the method overcomes the complexity and limitations of traditional manual designs.

A common approach to addressing this problem involves supervised methods, which benefit from the availability of ground truth data that can be easily generated in many cases. For example, IFCNN [33] is a supervised image fusion method specifically developed for multi-focus data fusion. The approach involves generating a dataset by simulating varying focus levels from a single image, with the original image serving as ground truth. The IFCNN architecture consists of three stages: feature extraction, feature fusion, and feature reconstruction. The feature extraction stage involves an encoder for each image, composed of two convolutional blocks. Feature matching is performed before each block. The feature fusion stage is executed using element-wise fusion rules, such as element-wise maximum, element-wise sum, and element-wise mean. Finally, the feature reconstruction stage, which consists of two convolutional blocks, aims to recreate an RGB image that closely matches the ground truth representing the fusion of the focuses from the input images. The cost function employed is a perceptual loss [34,35], which optimizes the image fusion by ensuring the fused image is as close as possible to the ground truth.
Another approach is proposed by MADCNN [36]. Initially, the two source images are preprocessed by converting them to greyscale and normalizing them. They are then concatenated into a single vector, which is fed into the MADCNN model composed of MA Blocks. These blocks consist of Multi-scale Feature Extraction (MFE) Units and Attention Units. The Attention Unit is made up of attention cells, while the MFE Unit employs standard convolution kernels and dilated convolution kernels to extract features at different scales. Finally, a convolutional layer and a sigmoid activation function generate a focus map. Using this focus map, a well-focused image is reconstructed and compared to a ground truth image using MSE. Additionally, an SSIM (9) is incorporated into the loss function to help in the learning process.

Another possibility is to use unsupervised methods such as DRPL [37]. The architecture consists of two stages: a weight map estimation stage for a given pair of input images, and an image fusion stage. The weight map estimation is a pixel-to-pixel regression that determines, for each pixel, whether it is blurred or not. This binary mask helps avoid the difficulty of estimating blur levels at the boundaries of focused and unfocused regions within the image. The estimation is performed using a fully convolutional network shared between the two images. The fusion stage is based on the weight maps and is carried out using the dot product and weighted summation operations. Finally, a loss function is computed by comparing the initial images with the resulting fused image. This metric is composed of two measures: a Structural Similarity Index Measure (SSIM) [38] and an edge-preserving measure.

A final type of approach is proposed by FuseGAN [39], a multi-focus image fusion method based on a GAN architecture, comprising a generator and a discriminator. To meet the requirements of fusion, the generator consists of two stages: an encoder that extracts features from the images and a decoder that reconstructs the desired output. Each image utilizes its own encoder, which consists of a series of convolutional layers. The extracted features are then concatenated before being passed to the decoder, which generates a confidence map indicating the focus areas of each image. The discriminator is based on a PatchGAN architecture [40], which divides the binary mask into patches to determine whether each patch is real or fake. These two models are trained concurrently within the adversarial framework, and the least squares GAN [41] loss is used to enhance training stability and generate better results. Additionally, a binary cross-entropy loss is applied to the generator. The generator is ultimately used for inference, and the final image is reconstructed based on its outputs.

To summarize, multi-focus image fusion methods can be categorized into two main types. The first category includes supervised methods, such as IFCNN [33] and MADCNN [36], which utilize datasets with ground truth or generated datasets. The second category consists of unsupervised methods, which employ image quality metrics to evaluate the fusion results [37], or more complex systems such as GAN-based architectures [39], to address multi-focus image fusion challenges.

### 2.1.2. Multi-focus metrics

In the realm of multi-focus image fusion, a variety of objective metrics are utilized to quantitatively evaluate the quality of fused images in the absence of a ground-truth reference image. These metrics are generally classified into four main categories: information theory-based metrics, image feature-based metrics, image structural similarity-based metrics, and human perception-inspired metrics. Notable metrics include normalized mutual information ($Q_{MI}$), nonlinear correlation information entropy ($Q_{NCIE}$), gradient-based metric ($Q_G$), phase congruency-based metric ($Q_P$), structural similarity-based metrics ($Q_E$ and $Q_Y$), and human perception-inspired metrics ($Q_{CB}$ and $Q_{CV}$).

Normalized Mutual Information ($Q_{MI}$) quantifies the information shared between the source and fused images. It is calculated using the formula

$$Q_{MI} = \frac{2 * I(F, A) * I(F, B)}{I(F, A) + I(F, B)} \qquad (1)$$

where I(F, A) and I(F, B) are the mutual information between the fused image F and the source images A and B, respectively.

$Q_{NCIE}$ evaluates the entropy of non-linear correlations in the fused image, indicating how well the fused image retains the information from the source images. It is defined as:

$$Q_{NCIE} = H(F) - H(F|AB) \qquad (2)$$

where H(F) is the entropy of the fused image, and H(F|AB) is the conditional entropy of the fused image given the source images.

The gradient-based metric ($Q_G$) assesses the preservation of edge and texture information in the fused image. It is calculated as:

$$Q_G = \frac{\sum_{i=1}^{N} (|G_F(i)| * \max(|G_A(i)|, |G_B(i)|))}{\sum_{i=1}^{N} (|G_A(i)| + |G_B(i)|)} \qquad (3)$$

where GF, GA and GB are the gradient magnitudes of the fused image and the source images, respectively, and N is the number of pixels.

The phase congruence-based measure ($Q_P$) evaluates the alignment of phase congruency between the fused image and the source images. It is defined as:

$$Q_P = \frac{\sum_{i=1}^{N} (PC - F(i) * \max(PC_A(i), PC_B(i)))}{\sum_{i=1}^{N} (PC_A(i) + PC_B(i))} \qquad (4)$$

where PC_F, PC_A, and PC_B are the phase congruence values of the fused image and the source images, respectively.

$Q_E$ and $Q_Y$ assess the structural similarity between the source images and the fused image. They are calculated as follows:

$$Q_E = \frac{1}{N} \sum_{i=1}^{N} SSIM(F(i), A(i)) * SSIM(F(i), B(i)) \qquad (5)$$

$$Q_Y = \frac{1}{N} \sum_{i=1}^{N} \frac{2 * SSIM(F(i), A(i)) * SSIM(F(i), B(i))}{SSIM(F(i), A(i)) + SSIM(F(i), B(i))} \qquad (6)$$

where SSIM(F(i), A(i)) and SSIM(F(i), B(i)) are the structural similarity indices between the fused image and the source images at pixel i.

$Q_{CB}$ and $Q_{CV}$ are designed to align with human perceptual assessments of image quality. They are defined as follows:

$$Q_{CB} = \frac{1}{N} \sum_{i=1}^{N} CB(F(i), A(i), B(i)) \qquad (7)$$

$$Q_{CV} = \frac{1}{N} \sum_{i=1}^{N} CV(F(i), A(i), B(i)) \qquad (8)$$

where CB(F(i), A(i), B(i)) and CV(F(i), A(i), B(i)) are the colour-based and visual-based quality assessments at pixel i, respectively.

### 2.1.3. Multi-focus datasets

As observed in the literature, many methods rely on existing state-of-the-art datasets and apply transformations to generate multi-focus image pairs. These datasets include NYU-D2 [42], a collection of indoor scenes originally created for depth estimation and semantic segmentation tasks, utilized in the IFCNN method [33]. PASCAL VOC 2012 [43], a benchmark dataset covering a wide variety of objects and commonly employed for tasks like object detection and segmentation, used in methods such as FuseGAN [39]. ILSVRC [36], derived from the large-scale ImageNet challenge, which provides diverse object categories and has been adapted for multi-focus fusion tasks. Additionally, the MFFW dataset [44], specifically designed for multi-focus fusion, contains photographic scenes with varying focal regions to evaluate the generation of fully focused images. For these datasets, the typical approach involves generating pairs of images with different focus areas by applying synthetic blurring to regions of a single source image, while retaining the original image as ground truth. This enables robust supervised training and testing across diverse scenarios, as these datasets [3] provide a blend of controlled synthetic data and real-world complexity.
### 2.1.4. Multi-focus applications

Multi-focus image fusion has numerous potential applications across various fields. One of its primary applications is in photography, where it facilitates the reconstruction of perfectly focused images by fusing images captured at varying focus levels, as demonstrated by the IFCCN method [33]. In the biological sciences, a similar issue arises with microscopes, which may struggle to capture an entire object in focus simultaneously. Multi-focus image fusion methods address this by combining images to produce a single, uniformly focused image [45–48].

These techniques are also applicable in medical imaging, such as the acquisition of Pap smear images [49]. For example, multi-focus image fusion is used to enhance microscopic images of cervical cells [50] and to reconstruct multi-focus images of bacilli in conventional spumum smear microscopy for tuberculosis diagnosis. In industrial contexts, multi-focus image fusion improves quality control imaging of objects [51,52], ensuring that all parts are in focus, which is crucial for detailed inspection tasks such as examining printed circuit boards [53].

Despite their varied applications, these methods consistently address the common challenge of producing fully focused images from sensors with limited depth-of-field capabilities, thereby ensuring comprehensive focus on the subject under study.

### 2.1.5. Multi-focus experiments

The objective of the experimental section is to compare different methods within the same type of category. The goal is not to recommend one architecture over another, but to understand which types of architectures are effective and which approaches could lead to the development of high-performing, multi methods in the future.

The results were obtained by collecting data from experiments on various state-of-the-art methods. To supplement these findings with additional methods, the following protocol was employed: First, the code of an existing method from the results table was obtained, and its performance was tested using the suggested dataset and recommended hyper-parameters. The aim was to achieve similar results to validate the setup of the environment and the dataset. Upon validation, a new method was trained on the same dataset used by the initial method, allowing for comparable and integrable results. The methods were selected based on their relevance in the current state of the art, the suitability of their architecture for this study, and the availability of code associated with the research article.

The Table 4 presents the performance of different multi-focus image fusion methods, evaluated using several quality metrics: Qmi (1), Qncie (2), Qg (3), Qp (4), Qe (5), Qy (6), Qcb (7), and Qcv (8). Each metric provides a different perspective on the quality and effectiveness of the fusion process.

Comparing the two methods, CNN and MADCNN, we observe that the CNN method consistently outperforms MADCNN across most metrics. For instance, CNN achieves a higher Qmi (1) score of 1.1512 compared to MADCNN's 1.1053, indicating better multi-exposure image fusion quality, according to this metric. Similarly, CNN has a higher Qncie (2) (0.8429 vs. 0.8394) and Qg (3) (0.7615 vs. 0.7564), suggesting more effective noise reduction and general image quality. Additionally, the Qp (4) metric, which measures the perceived quality of the fused image, is also higher for CNN (0.8506) compared to MADCNN (0.8428), reflecting a superior visual experience.

When examining the Qe (5) metric, which evaluates the exposure quality, CNN again leads with a score of 0.8808, slightly higher than MADCNN's 0.8818. The Qy (6) metric, reflecting the quality of the fused image in terms of luminance, shows a score of 0.9875 for CNN, indicating very high-quality luminance management, compared to MADCNN's 0.968. This slight edge may translate to better image clarity and brightness consistency in varied lighting conditions.

However, the differences in scores are relatively small, indicating that while CNN shows a marginal advantage over MADCNN, both methods perform well in multi-focus image fusion category. The Qcb

(7) metric, assessing colour balance, and Qcv (8), evaluating colour variance, also show higher scores for CNN (0.8084 and 16.2763, respectively) compared to MADCNN (0.7702 and 16.0603), further supporting the overall trend of CNN's superior performance.

### 2.2. Multi-exposure image fusion

Multi-Exposure Image Fusion (MEF) is an advanced image processing technique that merges multiple photographs of the same scene captured at varying exposure levels. The goal is to produce a single image that captures the optimal details from each source photograph. Thus overcoming the dynamic range limitations of digital cameras. The primary interest in MEF lies in its ability to create high dynamic range (HDR) images without the artefacts commonly associated with traditional HDR methods, such as halos or ghosting effects. By merging the properly exposed regions from each source image, MEF produces a final image that preserves detail in both bright and dark areas, offering a more accurate visual representation of the observed scene. Classical methods of MEF include approaches based on Laplacian pyramids [54], which decompose images into different frequency scales and merge details at each level, and bilateral filtering [55], which uses bilateral filtering to preserve edges while blending exposures. Another method, guided filtering [56], applies a guided filter to retain fine details while seamlessly combining the images, thereby enhancing the clarity and definition of the fused image. Additionally, random walks-based methods [57], utilize a probabilistic approach to determine which parts of the input images should be fused to achieve an optimized final image.

There are numerous traditional methods, but they consistently face issues with design complexity and lack of generalization. This is why the most effective methods now are those based on learning [20].

#### 2.2.1. Multi-exposure methods

Multi-exposure fusion methods have approaches that are quite similar to those used in multi-focus fusion. What connects these two fields is the ability to easily generate datasets by applying specific rules to transform an image into one that is either out of focus or improperly exposed. This is why we can find some methods like End-to-End Exposure Fusion Using CNN [58], which proposes a supervised multi-exposure image fusion method. The goal is to fuse images taken at different exposure levels to produce a single high-quality image, effectively replicating the effect of a well-exposed image from under-exposed and overexposed inputs. The method employs a lightweight CNN that directly learns the mapping between the multi-exposure input images and the desired output image. The network, consisting of convolutional layers followed by nonlinear activation functions (ReLU), extracts and refines features from the input images to reconstruct the fused image. The input images are converted to greyscale, processed by the model, and then combined using weighted fusion to produce the well-exposed image. A Mean Squared Error (MSE) loss is utilized between the generated image and the ground truth to train the system.

Alternatively, unsupervised frameworks address the challenge of ground truth unavailability by learning directly from the input images. DeepFuse [59] is an unsupervised multi-exposure image fusion method. The approach begins by converting the input images to the YCbCr format, after which the different channels are processed independently. The luminance channels of the two images are fused using a convolutional neural network to extract the important features from both images. The Cb and Cr chrominance channels are generated using a weighted sum method. This differential treatment is based on the assumption that the structural details and brightness variations of the image are more apparent in the luminance channel than in the chrominance channels. After processing all three channels, they are combined and converted back to RGB to produce the final fused image. A loss function based on MEF-SSIM [60] is used to compute the cost function by comparing the fused image with the input images.
Another approach is proposed by MEF-Net [61], a method for multi-exposure image designed to produce high-quality fused images from sequences of different exposures. MEF-Net employs a fully convolutional network, specifically a Context Aggregation Network (CAN), which processes a low-resolution version of the input sequence to predict low-resolution weight maps. These weight maps are then up-sampled to high resolution using a guided filter that leverages the high-resolution input sequence as guidance, ensuring smooth transitions and preserving details. The final fused image is obtained through a weighted summation of the high-resolution exposure images using these up-sampled weight maps. The entire MEF-Net is trained end-to-end using the MEF-SSIM [60] index as the loss function, which is perceptually calibrated to account for the human visual quality by optimizing the contrast, structure, and luminance aspects of the fused image.

MEF-GAN [62] is also an unsupervised method for multi-exposure image fusion. This approach employs a GAN architecture by comparing the generated image with a ground truth, in addition to using a discriminator. To create this ground truth, a dataset was developed by capturing three different images of various scenes: a properly exposed image, an underexposed image, and an overexposed image. The adversarial system's generator consists of two branches: a self-attention branch and a local detail branch. The self-attention block comprises a convolutional layer and an attention mechanism, enabling long-range relationships between features. The local detail block consists solely of convolutional layers, focusing on local relationships within the data. A merge block is used to fuse the information and generate the output image. The discriminator, based on convolutional layers, calculates the adversarial loss in addition to the content loss. The content loss compares the output image with the input images using Mean Squared Error (MSE) and includes a gradient-based loss.

GANFuse [63] is a multi-exposure image fusion method also based on GAN. This method aims to fuse images with extreme exposures to produce a well-exposed image. GANFuse employs an unsupervised learning framework that automates feature extraction and fusion from source images. The architecture consists of a generator and two discriminators; the generator synthesizes the fused image, while the discriminators enhance the retention of details from the source images by differentiating between the fused image and the source images. The cost function used in GANFuse includes a content loss, which measures pixel intensity and gradient differences to preserve luminance and texture, and an adversarial loss, which drives the generator to produce fused images that are visually indistinguishable from the source images.

Multi-exposure image fusion methods share similarities with multi-focus fusion techniques, particularly in terms of generating datasets by applying specific transformation rules to images. Among supervised approaches, End-to-End Exposure Fusion Using CNN [58] utilizes a lightweight CNN to directly learn the mapping between multi-exposure input images and a high-quality fused output image. Unsupervised methods, such as DeepFuse [59] and MEF-GAN [62], eliminate the need for ground truth data by using convolutional networks to extract key features from input images and fuse them. Other techniques, like MEF-Net [61], employ convolutional networks to generate weight maps that are used for high-resolution image fusion. Finally, GAN-based approaches, such as MEF-GAN and GANFuse [63], leverage generator and discriminator networks to enhance the visual quality of the fused images. These methods aim to produce well-exposed images by effectively combining information from images taken at different exposure levels.

### 2.2.2. Multi-exposure metrics

In multi-exposure image fusion, various objective metrics are employed to quantitatively assess the quality of fused images when no ground-truth reference image is available. SSIM (Structural Similarity Index) [38] is used to measure the similarity between two images. It considers changes in structural information, luminance, and contrast.

SSIM values range from -1 to 1, where 1 indicates perfect structural similarity. The SSIM index is computed as:

$$SSIM(x, y) = \frac{(2\mu_x\mu_y + C_1)(2\sigma_{xy} + C_2)}{(\mu_x^2 + \mu_y^2 + C_1)(\sigma_x^2 + \sigma_y^2 + C_2)} \quad (9)$$

where x and y are the two images, $\mu_x$ and $\mu_y$ are the average pixel values, $\sigma_x^2$ and $\sigma_y^2$ are the variances, $\sigma_{xy}$ is the covariance, and $C_1$ and $C_2$ are constants to stabilize the division.

QABF (Quality Assessment Based on Feature) [64] evaluates the quality of fused images by comparing feature maps derived from the source images and the fused image. It uses metrics like mutual information and structural similarity to assess how well the features from the source images are preserved in the fused image. The exact computation involves a series of steps that compare edge and texture information across the images.

MEF-SSIM (Multi-Exposure Fusion Structural Similarity Index) [65] is an extension of the Structural Similarity Index (SSIM) (9) specifically designed for MEF evaluation. It measures the structural similarity between the fused image and the input images, taking into account luminance, contrast, and structural information. The MEF-SSIM is calculated similarly to SSIM (9) but adapted to handle multiple exposure images.

Mutual Information (MI) [64] measures the amount of information that one image contains about another. It is useful for assessing how well the fused image retains information from the source images. MI is defined as:

$$MI(X, Y) = \sum_{x \in X} \sum_{y \in Y} P_{XY}(x, y) \log\left(\frac{P_{XY}(x, y)}{P_X(x)P_Y(y)}\right) \quad (10)$$

where X and Y are the source images, $P_X$ and $P_Y$ are the marginal probability distributions, and $P_{XY}$ is the joint probability distribution.

PSNR (Peak Signal-to-Noise Ratio) [66] is a widely used metric that measures the ratio between the maximum possible power of a signal and the power of corrupting noise that affects the fidelity of its representation. The higher the PSNR, the better the quality of the fused image. It is calculated as:

$$PSNR = 10 \cdot \log_{10}\left(\frac{MAX^2}{MSE}\right) \quad (11)$$

where MAX is the maximum possible pixel value of the image and MSE is the mean-squared error between the fused image and a reference image.

NIQE (Natural Image Quality Evaluator) [67] is a no-reference image quality metric that quantifies the naturalness of an image based on its deviation from statistical features derived from natural images. It does not require a reference image for comparison. The NIQE score is derived from a model trained on a collection of natural images, making it a complex but effective measure for assessing image quality.

SD (Standard Deviation) [62] is a statistical measure of the dispersion or variation in pixel intensity values within an image. A higher standard deviation indicates greater contrast and detail. It is calculated as:

$$SD = \sqrt{\frac{1}{M} \sum_{i=1}^{M} \sum_{j=1}^{N} (I_{ij} - \mu)^2} \quad (12)$$

where $I_{ij}$ is the intensity of the pixel at position (i, j), M and N are the dimensions of the image, and upsilon is the mean pixel intensity.

Entropy (EN) [68] measures the amount of information contained in an image. Higher entropy indicates richer details. The entropy H of an image is calculated as:

$$EN = - \sum_{i=1}^{n} p_i \log_2(p_i) \quad (13)$$

where $p_i$ represents the probability of the i-th pixel value.

The Average Gradient (AG) [64] metric measures the sharpness of an image by calculating the average change in intensity between
adjacent pixels. Higher AG values indicate sharper images. The AG is calculated as:

$$AG = \frac{1}{(M-1)(N-1)} \sum_{i=1}^{M-1} \sum_{j=1}^{N-1} \sqrt{\left(\frac{\partial I}{\partial x}\right)^2 + \left(\frac{\partial I}{\partial y}\right)^2} \quad (14)$$

where I is the image, $\frac{\partial I}{\partial x}$ and $\frac{\partial I}{\partial y}$ are the gradients in the horizontal and vertical directions, respectively.

### 2.2.3. Multi-exposure datasets

In multi-exposure image fusion, several datasets are utilized to train and evaluate the performance of various methods. Among them, the MEF-Net [61] dataset offers a wide array of HDR content, spanning diverse scenarios such as indoor and outdoor environments, human subjects, and still-life objects, as well as both day and night settings. This diversity enhances the generalizability of models trained on this dataset to various exposure conditions. Similarly, the Deep-Fuse dataset [59] includes both indoor and outdoor scenes, while also addressing varying lighting conditions, encompassing both natural and artificial light, and supporting linear and non-linear exposure settings. Additionally, the Large-Scale MEF Dataset, introduced by Cai et al. [69], focuses specifically on outdoor scenes, providing a rich source of data to enhance contrast in complex natural environments. Other state-of-the-art datasets, such as MEFB for outdoor scenes, Traffic Sign for road scenarios, and Perceptual MEF, which captures various types of scenes, further contribute to the evaluation and development of multi-exposure fusion methods by offering a range of exposure diversity and visual complexity. These datasets 5 are generally constructed by capturing the same scene under different exposure settings, facilitating robust training and testing of fusion algorithms.

### 2.2.4. Multi-exposure applications

Multi-exposure image fusion, which combines images taken at different exposure levels to generate a more information-rich representation, has diverse applications across numerous scientific and technical fields. For instance, in Optical Coherence Tomography (OCT) [70], it helps overcome the limitations of suboptimal sampling data by enhancing image reconstruction quality using neural networks. This approach is particularly valuable for obtaining high-quality images from incomplete or noisy data, which is critical in medical imaging for early disease detection. Similarly, in microscopy [71], the multi-exposure fusion combines with unsupervised transformations, preserving the content while enhancing image details. Beyond these domains, multi-exposure fusion is widely used in computational photography to create high dynamic range (HDR) images, which more accurately represent scenes containing both very bright and very dark regions. In surveillance, especially under challenging lighting conditions (e.g., strong backlighting or night scenes), combining exposures improves visibility and ensures important scene elements are not lost. In satellite and aerial imaging, multi-exposure fusion can improve detail retrieval in regions affected by varying illumination or atmospheric conditions. Additionally, in autonomous driving, this technique enhances scene understanding in high-contrast environments such as tunnels or sun-glare situations. Finally, in cultural heritage preservation, multi-exposure fusion assists in capturing fine details from artworks or inscriptions that may not be visible under standard lighting conditions.

### 2.2.5. Multi-exposure experiments

Based on the results presented in Table 6, an analysis of multi-exposure image fusion performance across different methods can be conducted using several key metrics. MEF-GAN [62] achieves the highest PSNR (11) (68.3721) and CC (16) (0.942), indicating superior image quality reconstruction and strong correlation with the reference image compared to DeepFuse [59] and FLER. While DeepFuse shows slightly lower PSNR (11) (67.0045) and CC (16) (0.9389), it still maintains competitive performance, particularly in terms of MSSIM (0.9813), suggesting it preserves structural similarity effectively. FLER, however,

exhibits the lowest performance across all metrics, particularly in CC (16) (0.7304), highlighting a weaker ability to correlate reference images. In terms of standard deviation (SD) (12), MEF-GAN and DeepFuse show higher values (0.2103 and 0.1946, respectively), which might indicate better contrast and detail retention compared to FLER. Overall, MEF-GAN provides the most balanced performance, excelling in both image quality and feature retention.

## 2.3. Multi-modal image fusion

Multi-modal image fusion encompasses techniques for integrating images captured from different modalities. This active area of research primarily includes two domains: multispectral image fusion and sensor-based image fusion. Multispectral image fusion is often exemplified by the fusion of infrared and visible images [10–13]. This can also include other methods that deal with images from cameras operating in different light spectra, known as multispectral or hyper-spectral image fusion [72]. Fusion of images from different sensors is referred to as multi-modal image fusion. Here, the term multi-modal specifically refers to images, unlike its broader use in the literature to describe the fusion of different media types such as text, audio, and video. Multi-modal image fusion is commonly applied in medical imaging, particularly when combining different measurements (e.g., MRI, PET [2]). Methods for multi-modal image fusion often address both categories and are divided into two learning approaches: unsupervised and task-driven [15].

### 2.3.1. Multi-modal methods

Multi-modal image fusion methods can be broadly categorized according to their primary objective into two main paradigms: task-driven methods, which aim to optimize performance for downstream tasks such as classification, detection, or segmentation using multi-modal inputs; and image generation methods, which are typically unsupervised and focus on synthesizing a fused image that retains complementary information from the source modalities. In this section, we adopt a structured presentation of the methods to reflect both their learning paradigm and architectural design. Unsupervised approaches are first grouped into two major families: encoder-decoder-based methods, which rely on feature extraction and reconstruction schemes, and GAN-based methods [73], which leverage adversarial learning to enhance fusion quality. This ordering is chosen to reflect increasing architectural complexity and modelling capability. Task-driven methods are presented subsequently, as they represent a distinct category where fusion is guided by performance on specific application-level objectives.

DenseFuse [74] is an unsupervised encoder-decoder architecture specifically developed for visible-infrared image fusion. Its defining characteristic is the incorporation of DenseBlock into its DenseNet architecture, enabling an efficient feature aggregation. The DenseBlock is composed of convolutional layers where each input is a concatenation of the outputs from all preceding layers, facilitating optimal information transmission. It comprises three convolutional layers, enabling the effective aggregation of information. An initial convolutional layer aggregates information from the different images. Subsequently, an information fusion stage is employed between the encoder and decoder sections. Two fusion strategies are considered: an addition-based strategy and an L1-norm-based strategy. Finally, a decoder consisting of four convolutional layers reconstructs the output image. The loss function combines SSIM (9) and a pixel loss, which calculates the Euclidean distance between the output and the input images, ensuring high-quality image fusion.

Another unsupervised approach for multi-modal image fusion is CentralNet [75]. The architecture of CentralNet consists of three CNN. Two of these networks are used to encode individual images, each specialized for a given modality. Each of the modality-specific CNN encodes its respective input image into feature maps. These CNNs are
trained to capture the salient features unique to their modality. A central network is positioned between these modality-specific networks. The central network takes the encoded feature maps from the two modality-specific networks and performs a weighted sum operation on them. This method allows for the fusion of feature map information from each layer of the neural networks. The loss function in CentralNet is a composite of three components: the loss from each of the modality-specific networks and the loss from the central network. This composite loss ensures that the network maintains accuracy in capturing uni-modal features while effectively merging them in the central network.

Other approaches exist with methods based on architectures that handle multi-scale processing. This is the case with Image Fusion Transformer [2], which is an unsupervised infrared and visible image fusion architecture based on transformers. Each image has its own encoder composed of four blocks that operate at multiple scales, and a decoder similar to a U-Net++ [76] decoder. In U-Net++, nested and dense skip connections involve linking feature maps from the encoder to multiple intermediate layers within the decoder, enhancing information flow and gradient propagation. These connections allow each decoder block to receive inputs from all preceding layers, improving feature reuse and segmentation accuracy. The goal is to create an image that integrates the fused features from the input images. Fusion occurs between the encoder and decoder through a Spatio-Transformer Fusion block for each scale of the multi-scale architecture. This module comprises a transformer branch based on multi-head attention methods to capture global context, and a spatial branch made up of convolutional layers to capture local context. The cost function used combines SSIM (9) and a feature similarity loss that calculates the difference in information content between the output image and the two input images.

RFN-Nest [77] builds on the U-Net model, utilizing convolutional layers to generate multi-scale feature maps at four encoding stages, facilitating detailed information capture. The decoder, inspired by U-Net++, employs nested and dense skip connections to link feature maps from the encoder to intermediate layers within the decoder. The objective is to generate an image that effectively combines the features from the input images. The fusion process takes place between the encoder and decoder through a residual fusion network at each scale of the multi-scale architecture. The RFN uses a convolutional structure with residual connections. The cost function is a combination of SSIM (9) and pixel loss, which measures the Euclidean distance between the input images and the output image.

The multi-scale architecture allows for obtaining information at multiple scales of the images. To improve previous methods, approaches like NestFuse [78] use attention layers. Attention layers have the advantage of establishing long-distance relationships between elements in an image compared to convolutional layers. NestFuse is an architecture for infrared and visible image fusion based on nest connections and spatial/channel attention models. The architecture features multiple encoders with densely connected convolutional layers that extract features from both infrared and visible images. These features are fused using a spatial/channel attention mechanism that enhances relevant information while suppressing noise. The nested connections in the encoder ensure efficient feature reuse and improved gradient flow. The decoder reconstructs the fused image by combining the enhanced features, producing an output that integrates the most informative aspects of both modalities, like UNet++ [76]. The loss function includes SSIM (9) and pixel intensity losses to ensure high visual fidelity and accurate detail preservation in the fused image.

The Self-Supervised Transformer, STFNet [79] also uses transformers for multi-modal image fusion. This method is an unsupervised approach based on a dual-branch architecture. Two CNN-based encoders are used to reduce the dimension of the images and select important features. A key aspect of this method is the use of a fusion module after the concatenation and alignment of features. The alignment of features is accomplished using a combination of convolution

and deformable convolution operations [80]. The feature fusion module consists of Detail Self-Attention (DSA) and Saliency Self-Attention (SCA) modules. These attention modules are specifically designed to extract information from infrared (saliency) and visible (detail) images. This method is therefore specialized for the fusion of multi-spectral visible/infrared information. A CNN-based decoder reconstructs the fused image. The quality of the created image is evaluated using a cost function based on structural similarity (MS-SSIM) [81], frequency analysis, and Fourier transform to ensure the fused image is of high quality.

Another method based on transformers for unsupervised infrared/visible image fusion is CMAFusion [82]. The architecture consists of two main components: a feature extractor and an image reconstructor. The feature extractor includes five convolutional layers for each modality, incorporating fusion blocks from layer C2 onward. Fusion information is added to the outputs of the feature extraction layers via element-wise addition. The fusion module, called the Cross-Modal Feature Fusion (CMFF) module, performs element-wise subtraction. A Global Average Pooling (GAP) operation is then applied, followed by a Fully Connected Model. The infrared information is element-wise added to the visible information, and vice versa. The two channels are finally concatenated for the image reconstruction part, which consists of four convolutional layers. The cost function is a combination of SSIM (9), texture loss, and pixel intensity loss.

Finally, the MDAN architecture [83] is a solution that uses transformers and an architecture based on a multi-scale to perform multi-modal image fusion. The training process involves two stages. First, an encoder-decoder architecture, composed of dense convolutional layers and skip connections similar to U-Net [76], is used to learn image encoding. The aim is to encode an image and decode it to reproduce an identical image, with a separate encoder for each input image. The learned architecture and weights are then reused, incorporating fusion blocks between each dense layer of the encoder. These fusion blocks are based on spatial-channel attention and feature aggregation, utilizing dual-branch fusion mechanisms. The two decoders receive fused information, and their outputs are concatenated via element-wise addition to producing the fused image. The fusion system's cost function comprises a SSIM (9), a pixel loss, and an algebra loss.

To summarize, as shown in Fig. 3, encoder-decoder based architectures use multi-level fusion systems that are characterized differently. Initially, simplistic methods perform fusion at the input level models, such as Dense-FG [84]. Subsequently, methods like MDAN [83], RFN-Nest [77], or Image Fusion Transformer [2] use UNet based architectures [76] to enable information fusion at various scales. Finally, some methods feature architectures with multiple branches, each dedicated to a specific modality, such as CentralNet [75] or NestFuse [78]. All these unsupervised methods employ techniques of varying complexity to achieve the fusion task, which involves reconstructing an image containing information from different modalities. To accomplish this, they all use losses based on pixel loss, texture loss, and image quality metrics like SSIM (9). It is worth noting that the specific use of pixel loss and texture loss metrics is designed for the fusion of infrared and visible images.

Unsupervised multi-modal image fusion methods require the use of a loss based on criteria that do not need ground truth to function. Therefore, the use of an adversarial system can be applied. Adversarial systems use an architecture similar to GAN, with one model serving as the generator and another as the discriminator. The competition between the two models allows for the calculation of an adversarial loss. FusionGAN [85] applies GAN to integrate visible and infrared images. It fuses input image's channel-wise, with a convolutional generator producing the final fused image. Another convolutional neural network is then used to compute the adversarial loss. Additionally, a content loss is included to preserve infrared image information and enhance the gradients of visible images. The content loss ensures that essential
<figure><img src="image_6.png" /><figcaption>Fig. 3. Summary of unsupervised multi-modal encoder-decoder architectures: Unsupervised multi-modal encoder-decoder architectures typically take three common forms. The first involves using a separate encoder for each modality, followed by feature fusion, and a single decoder to reconstruct the image. The second, more advanced architecture, also employs one encoder per input (referred to as branches), but includes interactions at different encoding stages to fuse the information, followed by a shared decoder to generate the output image. Lastly, the third type of architecture is inspired by UNet-like methods, featuring multi-scale encoders, a multi-scale fusion system, and a multi-scale decoder to reconstruct the output image.</figcaption></figure>

<figure><img src="image_7.png" /><figcaption>Fig. 4. Summary of unsupervised multi-modal GAN architectures: GAN-based architectures for multi-modal image fusion can be grouped into three forms, similar to encoder-decoder-based methods. The first type of architecture uses a single generator that concatenates the two input images to produce the fused image, along with a discriminator. The second type employs a branch-based architecture, with a generator for each input image, followed by a fusion operation at the output and one or more discriminators. Finally, the third type utilizes a multi-scale generator and applies a discriminator to the fused image at multiple scales.</figcaption></figure>

features from both modalities are maintained, leading to high-quality fused outputs that benefit from the strengths of each input type.

Dense-FG [84] is an unsupervised infrared and visible image fusion architecture which operates similarly. The architecture comprises a generator and a discriminator, both utilizing Dense blocks similar to those in DenseFuse [74]. These Dense blocks incorporate nested connections, linking certain feature maps to the inputs of other convolutional layers. Two different Dense blocks are employed: one with six convolutional layers for the generator and another with four convolutional layers for the discriminator. Each block has its unique configuration of nested connections, enhancing results and information propagation. The cost function consists of the adversarial loss from the GAN system and a content loss that includes gradient and intensity components.

Certain architectures use a GAN base and add a multi-branch structure within the architecture. This is the case with GANMcC [86]. GANMcC is an unsupervised GAN-based architecture designed for fusing infrared and visible images. The generator employs two distinct paths: a gradient path that encodes the visible image to preserve its texture, and a contrast path that encodes the infrared image to maintain its contrast information. The feature maps from both paths are concatenated channel-wise and processed by convolutional layers to generate the fused image. The discriminator is a CNN that can take as input either a real image (infrared or visible) or a fake (fused) image, and it outputs two probabilities: the probability that the image is infrared and the probability that the image is visible. The cost function comprises an intensity loss for visible images, a gradient loss for infrared contrast,
and an adversarial loss tailored to handle the different cases where the image is visible, infrared, or a fused image.

Another method that uses a dual-branch architecture is DDCGAN [87], an architecture designed for multi-resolution image fusion. The architecture consists of a convolutional encoder with connections blocks and two convolutional decoders. Inspired by Dual Discriminator GAN [88], this method incorporates the handling of multi-resolution images using deconvolution layers and integrates the functionality of Conditional GAN [89]. The cost function includes an adversarial loss that combines the errors from the two discriminators and a content loss that balances gradient variation and pixel intensity.

Is it possible to incorporate a multi-scale approach like GAN-FM [90]. The encoder is a multi-scale convolutional network based on a U-Net architecture, known as Full-Scale Skip Connection. These connections are utilized between the encoder and decoder parts of the generator. The discriminator consists of two Markovian networks [91] that analyses the image in fixed-size patches to determine whether the information originates from the infrared or visible spectrum. The output is a matrix representing the probability that a patch is either infrared or visible. The adversarial loss is adapted to use the results from the Markovian discriminator models. Additionally, a content loss is included, consisting of a gradient loss and an intensity loss.

More recent approaches than GAN methods can also work for multi-modal image fusion. For example, denoising diffusion methods like DDFM [92]. DDFM is an unsupervised image fusion method designed to integrate multiple modalities, such as infrared and visible images. The core architecture leverages the principles of denoising diffusion models [93], which iteratively refine the fused image through a series of denoising steps. The model consists of two main components: a diffusion process that gradually adds noise to the images, and a denoising process that removes this noise while preserving and enhancing critical features from each modality. Two additional modules are integrated into the DDPM framework: the Unconditional Diffusion Sampling (UDS) module, which aims to recreate a natural-looking image, and the Expectation Maximization (EM) module, which strives to retain the maximum amount of information in the image.

Similar to encoder-decoder-based methods, GAN-based architectures have various forms: classical architectures like FusionGAN [85], multi-branch architectures like GANMcC [86] or DDCGAN [87], and architectures including multi-scale elements like GAN-FM [90]. It is notable that multi-branch methods have modality-specific models for both the generator and the discriminator. The cost functions used are based on adversarial loss in combination with more traditional losses for unsupervised image fusion. Adding these components to the loss function helps improve model convergence, which is a complex aspect for GAN-based methods. The Fig. 4 summarizes this category of method.

The advantage of unsupervised methods is their ability to generate images without needing a labelled dataset. However, training can be challenging, particularly for methods such as GAN, and it is often difficult to define the target objective. For instance, in visible/infrared image fusion, one might assume the format and information present in the image, but this is not always straightforward, especially with more exotic or significantly different modalities. Furthermore, generating an image is a fusion task but not necessarily the end goal. Other tasks, such as classification, detection, segmentation, etc., can be performed subsequently. These tasks can provide a more reliable cost function with ground truth. This is known as task-driven learning. RD-ML [94] is a task-driven architecture based on a dual-branch system for each modality. Each branch functions as an encoder, similar to the CentralNet [75] architecture. However, RD-ML is a task-driven method designed for detection tasks, aiming to produce bounding box as the output. RD-ML also includes a third, central branch dedicated to fusion, which aggregates information on each layer from the lateral models. This fusion process is facilitated by a Gated Information Fusion cell, GIF, inspired by Long Short-Term Memory cell [95]. GIF cells

perform operations such as concatenation, sigmoid activation, convolution, and element-wise product. This architecture allows for efficient and effective integration of multi-modal information, enhancing the performance of detection tasks.

Another approach is the MF-Net [96] architecture. MF-Net is also a task-driven encoder-decoder architecture designed for the fusion of infrared and visible images to enhance object segmentation capabilities. Each image modality is processed by a separate encoder composed of convolutional layers and mini-inception blocks. The outputs from these encoders are concatenated and fed into the decoder. The decoder, which features unpooling layers and shortcut connections similar to those in a U-Net, outputs an image with *N* channels, where *N* is the number of classes. Each channel indicates whether a pixel belongs to a given class, thereby improving segmentation accuracy by leveraging the fused features from multiple modalities.

GMFNet [1] uses is a combination of several other methods. It is a task-driven architecture designed for semantic segmentation of visible and thermal images. The model consists of three branches, similar in functionality to CentralNet [75]. Two branches are dedicated to encoding the input images, each encoder is specific to one modality. A central network fuses the information from each feature map. The fusion system employs a cell similar to GIF [33] to aggregate information in the central network. Each encoder has its own decoder to reconstruct a segmented image using a U-Net architecture [76], incorporating skip connections between the encoder and decoder. The loss function is based solely on segmentation loss, since no temporary fused image is created to perform the task.

Some of these task-driven image fusion methods use multi-branch methods, such as RTFNet [97]. It is a task-driven method for fusing RGB and thermal images. The architecture employs separate encoders for each modality, both based on ResNet [98], and an asymmetric decoder to reconstruct the image. Fusion via element-wise addition occurs after the first layer and after each residual block of the encoders. The decoder consists of Upception A and B blocks. Block A maintains the number of channels and resolution, while Block B increases resolution and decreases the number of channels. Block A includes convolution, ReLU, and Batch Normalization, whereas Block B features a shortcut connection and transposed convolutions. The goal is to complete scene segmentation to compute the loss against ground truth, thereby refining the entire system.

Another approach, OAFA [99], allows using transformer architectures. This method comprises two components: the Cross Modality Spatial Offset Modelling (CSOM) module and the Offset Deformable Alignment and Fusion (ODAF) module. Initially, a feature extractor specific to each modality is used. The first module, CSOM, includes decoupled multi-modal learning and spatial offset modelling. CSOM separates information into two spaces: modality-invariant and modality-specific features, thanks to the decoupled multi-modal learning component. Meanwhile, spatial offset modelling produces a superior spatial offset in a common subspace. The second module, ODAF, consists of offset-guided deformable alignment and decoupled feature fusion. The offset-guided deformable alignment, using deformable convolutions [80], provides adaptive alignment as the images are not initially aligned. Decoupled feature fusion is employed to handle redundant information and enhance relevant information. The CSOM and ODAF modules are interwoven. Finally, a detection network is used for object detection, and the error is calculated based on the ground truth.

A generative task-driven approach for object detection with visible and infrared image is proposed by TarDal [100]. The system employs cooperative learning, where the fusion module has its own loss functions, including SSIM (9), pixel intensity loss, and adversarial loss, while also incorporating the detector's loss. A YoloV5 [101] model is used to detect objects in the fused image and compute a cost function based on ground truth. Both the generator and discriminator are convolutional networks, with the generator featuring connections block. The architecture includes two discriminators: a target discriminator focused
<figure><img src="image_8.png" /><figcaption>Fig. 5. Task-Driven multi-modal architectures: This figure, inspired by previous work on the definition of Task-Driven architectures [15], illustrates how unsupervised and task-driven architectures differ. While the underlying architectures may be similar, the objectives vary. In unsupervised methods, the loss function relies on custom loss terms. Task-driven methods, on the other hand, employ ground truth based on the specific fusion task, aligning the learning process with the desired fusion task.</figcaption></figure>

on infrared images and a detail discriminator focused on visible images. This dual-discriminator setup helps in preserving crucial details from both modalities, enhancing the detection performance.

Finally, for the task-driven approach, SCGRFuse [102] employs separate feature extractors for each modality using convolutional blocks, and integrates these features through a Visual Transformer block. This architecture, inspired by SeAFusion [103], utilizes convolutional feature extraction composed of residual and dense residual blocks, serving as encoders for each image. The encoders transmit the information to a fusion layer, which consists of two attention systems: a channel attention system and a spatial attention system. These attention systems feed below the feature map to a Pooling Fusion Block for segmentation. The task-driven segmentation network then decodes the information to solve the segmentation task, with segmentation loss calculating the system error. This loss includes a texture loss to enhance texture in the image and an intensity loss to measure pixel-level differences between source and fused images. The absence of the spatial channel attention module degrades visual fidelity and information retention, highlighting the importance of attention mechanisms in preserving image details and focusing on its structure and edges to retain relevant information.

The task-driven approach is characterized by the use of specific tasks to guide the learning process across the entire system. Techniques such as contrastive learning can be considered in this category. CocoNet [104] employs contrastive learning techniques for the fusion of visible and infrared images. This approach utilizes pairs of images (both positive and negative) to guide the learning process. Positive image pairs, which are similar, encourage the model to produce similar outputs, whereas negative pairs, which are dissimilar, should result in distinct outputs. CocoNet addresses the fusion challenges by focusing on two main aspects: target and detail. To facilitate this, manual masks are generated to delineate foreground and background contours in both image modalities. The architecture of CocoNet includes three models: a central fusion model and two lateral models, featuring a modified U-Net [76] structure with shared skip connections. Fusion is achieved through a Multi-level Attention Module (MAM), yielding a single fused output. The network operates in a task-driven manner, with contrastive learning as its core objective. By dividing the fused image into patches and applying contrastive constraints, CocoNet ensures that relevant features from both visible and infrared images are preserved while redundant information is discarded.

In summary, task-driven multi-modal architectures typically feature an encoder coupled with additional structures to accomplish the desired task. The encoder can be a CNN with multiple branches, as proposed in MF-Net [96], or a transformer-based fusion system, such as in OAFA [99]. The previous figure of encoder-decoder based and GAN-based illustrates various types of encoder structures, including multi-branch architectures like TarDAL and multi-scale architectures like the Image Fusion Transformer method [2]. Task-driven methods often reuse the architectural designs of unsupervised methods, with the key difference being a modification of the output to align with the specific task as shown in Fig. 5. The loss functions for these methods are task-specific, meaning that evaluation metrics are based not on the generated images but on the performance of the task itself.

### 2.3.2. Multi-modal metrics

Multi-modal image fusion employs many metrics similar to those used in multi-exposure and multi-focus image fusion. These include Entropy (13), Average Gradient [105], Standard Deviation (12), PSNR (11), Mutual Information (10), SSIM (9), and Qabf, which are image analysis metrics applicable to the context of multi-modal image fusion.

More specific metrics are also used. The SCD (Sum of Correlations of Differences) [106] metric measures the correlation between the differences of the source images and the fused image. A higher SCD value indicates that the fused image retains the distinguishing features of the source images more effectively. The SCD is calculated by computing the sum of the correlation coefficients of the differences between each pair of source images and the fused image.

MSSSIM [60] is an extension of the Structural Similarity Index (SSIM) (9) that evaluates image quality at multiple scales. It is particularly useful for multi-modal fusion, as it can assess structural similarity across different levels of detail. MSSSIM is calculated by combining SSIM measurements at various scales.

SF (Spatial Frequency) [107] is a measure of the level of detail in an image. Higher spatial frequencies indicate more detailed and sharper images. The SF of an image is calculated by analysing the variations in intensity in both the horizontal and vertical directions, using the following formula:

$$SF = \sqrt{(SF_H)^2 + (SF_V)^2} \qquad (15)$$

where $SF_H$ and $SF_V$ are the spatial frequencies in the horizontal and vertical directions, respectively.

VIF (Visual Information Fidelity) [108] measures the fidelity of the visual information in the fused image compared to the source images. It quantifies how much information from the source images is preserved in the fused image, considering the human visual system's sensitivity to different spatial frequencies. VIF is calculated using complex models of natural scene statistics and the human visual system.

The CC (Correlation Coefficient) [87] metric measures the linear correlation between the pixel intensities of the source images and the fused image. A higher CC indicates that the fused image is more similar to the source images in terms of pixel intensity values. The correlation coefficient is computed as:

$$CC = \frac{\sum(A_i - \bar{A})(B_i - \bar{B})}{\sqrt{\sum(A_i - \bar{A})^2 \sum(B_i - \bar{B})^2}} \qquad (16)$$

where $A_i$ and $B_i$ are the pixel values of the source and fused images, and $\bar{A}$ and $\bar{B}$, are their mean values.

FMIx [109] evaluates the mutual information between the source images and the fused image based on specific features such as edges and textures. This metric helps to determine how well the fused image retains important feature details from the source images. FMIdet [109] focuses on the mutual information of detailed features. This variant specifically assesses how well the fine details and intricate structures from the source images are preserved in the fused image. The mutual information is calculated as the previous equation for MI.
These metrics are used for unsupervised methods and are based on image quality and information quantity measurements. However, task-driven methods can also rely on metrics specific to their tasks, such as accuracy, recall, or precision for classification, and metrics like mAP50 and mAP75 for detection. These metrics are specific to the fusion tasks performed within the framework of task-driven learning.

### 2.3.3. Multi-modal datasets

The datasets used in state-of-the-art comparisons for multi-modal image fusion are listed in the table below. These datasets predominantly focus on visible and infrared modalities, with applications often centred on autonomous driving. Each dataset presents unique characteristics, such as the number of image pairs, the capture methodologies, and the suitability for specific learning paradigms. For example, the M3FD [100] and MFNet datasets [96] are tailored for task-driven learning, as they include ground truth annotations for object detection tasks. Other datasets, such as TNO [110], capture multispectral data in military scenarios, while RoadScene, MSRS [111], and LLVIP [112] provide infrared-visible imagery specifically for road scenes, supporting fusion methods relevant to driving environments. The KAIST dataset [113] is notable for its visible-thermal data aimed at pedestrian detection, and the Harvard dataset [114] includes MRI-PET imagery for medical applications, offering a broader perspective beyond autonomous driving. These datasets [7] collectively enable the evaluation and development of fusion methods across diverse modalities and application domains.

### 2.3.4. Multi-modal applications

While multi-modal image fusion has diverse applications, the integration of visible and infrared images remains the most extensively researched area in recent literature. This type of multi-modal fusion is particularly useful for autonomous vehicle navigation [1]. It enables vehicles to detect elements emitting infrared radiation, regardless of lighting conditions or weather. Such methods are especially beneficial for pedestrian detection [100,102]. Another application related to object detection involves using multi-modal data for defect recognition in industrial production. The use of such data provides additional information across various spectral bands or modalities about potential defects. For example, this approach is employed in the detection of defects in wind turbine blades. Finally, when discussing modalities, it is not limited to multispectral fusion but also includes multi-sensor image fusion. This is particularly relevant in the medical field, such as the fusion of MRI and PET data [2]. These methods can, for instance, enhance the detection of cancerous cells across different modalities. In addition to these domains, multi-modal fusion also finds applications in remote sensing, where combining data from optical, SAR (Synthetic Aperture Radar), and hyperspectral sensors improves land cover classification and disaster monitoring. In security and surveillance, the fusion of low-light visible and thermal imagery improves intruder detection in low-visibility environments. Moreover, in augmented and virtual reality, fusing RGB and depth images enhances scene understanding and user interaction, enabling more immersive and accurate spatial experiences.

### 2.3.5. Multi-modal experiments

The provided Tables 8 and 9 or Fig. 6, evaluate the performance of different multi-modal fusion methods on two different datasets: M3FD and TNO. Various performance metrics are used, including Standard Deviation (SD) (12), Mutual Information (MI), Visual Information Fidelity (VIF), Average Gradient (AG), Correlation Coefficient (CC), Spatial Consistency Degree (SCD), Entropy (EN) (13), Quality Assessment Based on Features (Qabf), and Spatial Frequency (SF) (15). These metrics collectively assess aspects such as image contrast, information content, sharpness, and quality.

On the M3FD dataset, the MDAN method generally outperforms others across several metrics, showing the highest Standard Deviation (SD) (12) of 50.023, which indicates superior contrast. It also has

the highest Mutual Information (MI) score of 2.935, suggesting better information retention. MDAN achieves the highest Visual Information Fidelity (VIF) of 0.759 and the highest Average Gradient (AG) (14) of 5.049, highlighting its effectiveness in preserving image detail and sharpness.

Tardal also performs well, particularly with a high SD (12) of 42.916 and MI (10) of 3.191. It shows a high VIF score of 0.606 and AG (14) of 4.128, making it another strong performer in terms of detail preservation. SeAFusion achieves the highest entropy (EN) (13) of 7.431, indicating the highest information richness among the methods, and a high Qabf score of 0.564, reflecting good feature-based quality.

DenseFuse, FusionGAN, and DDcGAN exhibit lower performance in comparison. While DDcGAN shows relatively high SD (12) and AG (14) values (40.730 and 4.539, respectively), suggesting good contrast and sharpness, it does not perform as well in other metrics. FusionGAN scores the lowest in most metrics, indicating that it might not be as effective in multi-modal fusion tasks as the other methods.

For the TNO dataset, MDAN also demonstrates strong performance with the highest SD (12) of 49.462 and a high AG (14) of 4.971, suggesting effective contrast and detail enhancement. Its VIF score of 0.662 and Qabf of 0.424 indicate good image quality and feature retention.

Tardal also shows robust performance with an SD (12) of 46.001 and MI (10) of 2.739, suggesting good contrast and information retention. It has a relatively high VIF of 0.632 and AG (14) of 4.172, indicating good detail preservation.

SeAFusion performs well in terms of VIF and Qabf, with scores of 0.687 and 0.509, respectively, showing its ability to maintain image quality. However, it has lower SD (12) and AG (14) values compared to MDAN and Tardal, which suggests it might not enhance contrast and sharpness as effectively.

DenseFuse and FusionGAN show lower performance across the board, with FusionGAN having the lowest scores in most metrics, similar to its performance in the M3FD dataset. This suggests that FusionGAN may not be as suitable for high-quality multi-modal image fusion as the other methods.

Across both datasets, MDAN consistently shows superior performance in terms of contrast (SD) (12), information retention (MI), and image sharpness (AG) (14). Tardal and SeAFusion also perform well, with SeAFusion excelling in maintaining image quality, as indicated by its high VIF and Qabf scores. DenseFuse, FusionGAN, and DDcGAN perform moderately, with FusionGAN generally showing the weakest performance. These results suggest that MDAN is a particularly strong method for multi-modal image fusion, with Tardal and SeAFusion also being viable options depending on specific application needs.

## 2.4. Multi-view image fusion

Multi-view image fusion represents a distinct category often treated independently of traditional types like multi-exposure, multi-focus, and multi-modal fusion. Numerous surveys [115–119], address this type of fusion independently of other classical categories. The reason for this separation is that classical image fusion methods focus on matching features that appear differently in each image but are in relatively close positions. In contrast, multi-view image fusion matches information that may have different appearances depending on the viewing angle and, more importantly, can be in extremely different positions due to the nature of multi-view images. However, some similarities can be observed in these methods, as they all aim to fuse the images. Notably, they employ similar paradigms (supervised, unsupervised, and task-driven) and fusion approaches that can also be applied to classical methods, such as feature matching [116]. Two categories of multi-view image fusion methods will be discussed: those inspired by LiDAR sensors to simulate depth from images [120] and those that do not use them.
<figure><img src="image_9.png" /><figcaption>Fig. 6. Quantitative evaluation of image fusion methods across multiple quality metrics on the M3FD dataset [100].</figcaption></figure>

### 2.4.1. Multi-view methods

Various multi-view image fusion techniques can be systematically organized according to their learning paradigm and architectural complexity. We first distinguish between task-driven methods, where fusion is explicitly optimized for downstream objectives such as pose estimation, object recognition, or scene understanding, and unsupervised reconstruction approaches, which aim to infer a consistent 3D representation from multiple 2D views without direct supervision. Within the unsupervised category, we further group techniques into two families: neural radiance field (NeRF)-based models [119], which learn a continuous volumetric scene representation, and alternative unsupervised frameworks, such as voxel- or mesh-based methods, that rely on projection error minimization. This progression from NeRF to more classical volumetric methods reflects an increasing sophistication in scene modelling. Task-driven multi-view fusion methods are then presented, highlighting how they integrate multi-view inputs into end-to-end pipelines tailored for specific applications. This structured ordering clarifies the methodological landscape and helps guide the reader through the evolution of multi-view fusion approaches.

Another unsupervised approach is ROMA [121]. ROMA is an unsupervised feature matching method for multi-view images. ROMA is capable of robust feature matching under real-world conditions, such as changes in scale, lighting, viewpoint, and texture. The architecture includes two encoders for images: a coarse feature encoder based on DINOv2 [122] with its original weights and a specialized ConvNet fine encoder. Following this, a Transformer-based decoder is used as a feature matcher to enhance the matching process. The Transformer consists of five Vision Transformer [123] blocks with eight heads each. The model employs two distinct loss functions: a regression-by-classification loss for the coarse stage and a robust regression loss for the fine stage. The coarse loss function is designed to model multi-modal distributions during coarse matching using a regression-by-classification approach. On the other hand, the robust regression loss for the fine stage is conditioned by the previously estimated deformation. This robust regression loss is applied during the refinement phase to improve the deformation accuracy.

Regarding task-driven approaches, some methods are inspired by the functioning of image/LiDAR fusion techniques by simulating LiDAR data as End-to-End Pseudo-LiDAR [120]. The method consists of a novel framework based on differentiable Change of Representation (CoR). Initially, a model takes the two stereo views as input and computes a depth map. This depth map is then converted into a point cloud or voxel representation by the CoR module. The paper introduces two types of CoR: quantization (voxelization) and subsampling (LiDAR simulation with point clouds). This representation enables a 3D object detection model to operate and output 3D bounding boxes. The cost function comprises a component that accounts for depth estimation error and another based on ground truth for the object detection task.

The method CG-Stereo (Confidence Guided Stereo) [124] is also task-driven multi-view image fusion technique for 3D object detection which use a Pseudo-LiDAR approach. The input consists of stereo images. The first image is processed with a segmentation network to separate the foreground from the background. The segmented image and the second stereo image are then used in a stereo matching network called Split Depth Estimation. This network comprises three components: an encoder that takes the segmented image and the second stereo image, and two decoders that output a disparity map converted into a point cloud for the first, and a Confidence map for the second. A 3D object detection model based on point clouds is then used to compute the error relative to the ground truth. This method is categorized as a Pseudo-LiDAR [125] approach because it simulates the functionality of a LIDAR within its architecture.

Another approach is proposed by SIDE [3]. The method estimates the depth information of the centre point of each object. Specifically, it works on the centres of each bounding box representing the same object. From these points and images, a local cost volume based on the regions of interest in the images is reconstructed. Using this information, a structure-aware attention mechanism converts these data into a bird's-eye view to establish the 3D positions of the bounding boxes. Inspired by LiDAR-based methods [126,127], this system aggregates information and reduces spatial noise caused by information sparsity. A 3D CNN is employed to predict the bounding boxes, and this prediction is used to calculate the loss against the ground truth.
Finally, IDA-3D [128] proposes a task-driven approach for 3D object detection using stereo vision, focusing on depth estimation without relying on LiDAR data. The method utilizes stereo pairs of RGB images and corresponding annotated 3D bounding boxes to detect objects. The key innovation is the Instance-Depth-Aware (IDA) module, which estimates the depth of the centre of each 3D bounding box. This module builds a 4D cost volume and employs a 3D CNN to predict the depth information, which is critical for accurate 3D object localization. The system starts with a Stereo Region Proposal Network to generate left-right proposal pairs, extracting regions of interest for each object in the stereo images. By leveraging a structure-aware attention mechanism, the method converts this depth information into a bird's-eye-view representation to establish the 3D positions of bounding boxes. This approach effectively reduces space noise caused by information sparsity and aggregates information more efficiently than previous methods. The IDA module allows the system to directly predict the depth variation of the 3D bounding box centre, significantly improving detection accuracy and performance.

Among the non-LiDAR-based approaches, there are methods that utilize 2D point generators, such as 3DPPE [5]. Initially, the various images are encoded using a Swin Transformer [129]. The obtained features are then utilized in a 3D Points Generator. This block operates with a "hybrid-depth module" system inspired by BEVdepth [130], which calculates depth maps from image features and camera parameters. The depth maps are subsequently transformed to generate a point cloud. These 3D points are fed into the 3D Point Positional Encoder, which converts them into 3D positional encoding (3D PE) applied to the image features and finally mixed with the image features in a decoder. The decoder enables the prediction of objects to be detected. The method employs a supervised loss at the depth estimation level based on projected depth of the point cloud, smooth L1 loss [131,132], and distribution focal loss [133]. Additionally, the system uses a loss at the ground truth level of the objects to be detected.

Similar approaches using 3D volume generators as DSGN [4]. The proposed Deep Stereo Geometry Network (DSGN), start by taking a pair of stereo images as input and extracting features using a Siamese network. These features are used to construct a plane-sweep volume (PSV), capturing pixel-level stereo correspondences. This PSV is then transformed into a 3D geometric volume (3DGV) using differentiable warping techniques, which ensures that the 3D spatial information is accurately represented in the 3D space. Subsequently, a 3D CNN processes the 3DGV to learn the structural information necessary for 3D object detection. The feature extraction network is adapted from PSM-Net [134], optimized to handle high-level semantic features and context information crucial for precise stereo matching. The loss includes a smooth L1 loss for depth regression to minimize the difference between the predicted and ground-truth depths, a focal loss for classification to address class imbalance, and a binary cross-entropy loss for centerness.

Disp R-CNN [135] uses a rather different system based on disparity maps. The pipeline begins with the detection of 2D bounding boxes and instance masks using a stereo variant of Mask R-CNN, extending the Stereo R-CNN framework by incorporating instance segmentation. This is followed by the Instance Disparity Estimation Network (iDispNet), which estimates disparity maps specifically for object pixels, leveraging category-specific shape priors for enhanced accuracy. The disparity maps are then converted into instance point clouds, which are fed into a 3D detector to predict 3D bounding boxes. The loss function in Disp R-CNN includes a smooth L1 loss for instance disparity estimation, measuring the difference between predicted and ground-truth disparities for object pixels. It also incorporates dimensional regularization to penalize voxels outside the 3D bounding box and shape regularization to maintain the shape within reasonable variations.

The method "3D Object Proposals Using Stereo Imagery for Accurate Object Class Detection" [136] involves minimizing an energy function that incorporates object size priors, the ground plane context, and depth-informed features. Energy-Based Learning [137] is a

learning approach where models learn to associate an "energy" to each input-output configuration. Correct or desired configurations have lower energy, while incorrect or undesirable configurations have higher energy. Learning involves adjusting the model's parameters so that desired configurations have minimal energy, making it easier to distinguish between good and bad predictions. The model includes a system of 3D object proposals, which are then processed by a CNN that uses both appearance and depth information to predict 3D bounding box coordinates and object orientation. The depth calculation is based on the method presented by Yamaguchi et al. [138].

Finally, TEMPO [139] uses a method divided into three blocks: object detection, object tracking, and pose estimation with motion prediction. In the first part, object detection is performed using a 3D object detection model that outputs 3D bounding boxes. This model takes as input the unprojected features from each image, encoded by a pose backbone. At this stage, a loss function calculates the object detection error by measuring the distances between 2D and 1D heat-maps and applying a regression on the bounding boxes. This detection step is executed for each time instant. The tracking across different time instants is carried out using a SORT [140] tracker and a Kalman filter without a learned Re-ID mechanism. Finally, pose encoders provide the tracking information to a Spatial GRU [141], and a pose decoder outputs the pose estimation and forecasting. A loss function calculates the error in pose estimation.

Finally, the latest approach in multi-view image fusion is based on the concept of feature matching. SuperGlue [142] is a feature matching method for multi-view images designed to establish correspondences between two sets of local features by solving an assignment optimization problem through a differentiable approach. Its architecture comprises two primary components: an attentional GNN [143] and an optimal matching layer. The GNN utilizes a keypoint encoder to map keypoint positions and visual descriptors into a unified vector representation. It employs alternating self- and cross-attention layers to enhance these representations, facilitating the aggregation of contextual information both within and across images. The optimal matching layer constructs a score matrix, which is augmented with dustbins to handle unmatchable keypoints, and employs the Sinkhorn algorithm [144] to determine the optimal partial assignment. This process elegantly manages occlusions and non-repeatable keypoints by learning priors directly from data. The cost function for the optimization problem is predicted by the GNN, which leverages attention mechanisms inspired by the Transformer model, enabling the network to reason about the 3D scene and feature assignments jointly.

OmniGlue [145] introduces a novel approach to image feature matching for multi-view images, emphasizing generalization to unseen image domains during training. OmniGlue's model design employs contextual reasoning blocks incorporating both intra-image self-attention and inter-image cross-attention layers, with a total of 18 attention layers. The model leverages local descriptors and positional features to compute attention weights, but updates the attention values based solely on local descriptors, thereby accounting for spatial correlation between keypoints without overly relying on it. For similarity computation, a pairwise similarity matrix is created, where each element is the dot product of descriptors of corresponding keypoints. The Sinkhorn algorithm [144] is then utilized to refine similarities, producing a matching matrix that represents the probability of keypoint correspondences. Training involves minimizing the negative log-likelihood of the matching matrix relative to ground truth matches. Unlike SuperGlue [142], which combines local and positional descriptors, OmniGlue and LightGlue [146] avoid dependency on positional features for updated descriptors, though LightGlue uses specific encoding for self-attention.

In multi-view image fusion, various methods have been developed, spanning both task-driven and unsupervised approaches. Task-driven techniques, such as those inspired by image/LiDAR fusion, focus on generating high-quality 3D reconstructions or detecting objects
<figure><img src="image_10.png" /><figcaption>Fig. 7. Summary of multi-view architectures: There are three main groups of methods addressing multi-view image fusion. First, the most common and widely studied are 3D-based methods. These approaches incorporate 3D modules that can either simulate LiDAR data or estimate the depth from multiple views to perform a fusion task, typically focused on 3D object detection. Second, there are unsupervised 3D reconstruction methods, such as those based on neural radiance fields. Lastly, a different approach involves feature matching, which is used to perform various fusion tasks. This method extracts information from each image and employs specialized architectures to establish correspondences and determine the location of features across different images.</figcaption></figure>

by simulating LiDAR data through methods like End-to-End Pseudo-LiDAR [120] and CG-Stereo [124]. These methods often use stereo images to estimate depth and construct 3D bounding boxes, incorporating sophisticated networks and loss functions to enhance accuracy. On the other hand, unsupervised approaches leverage feature matching and neural radiance fields to predict and fuse images from multiple viewpoints without requiring ground truth. Techniques like ROMA [121] utilize advanced feature encoders and transformer-based decoders to handle real-world variations in scale, lighting, and viewpoint. The Fig. 7 summarizes the diverse methods with its unique architecture and objectives, ranging from 3D object detection to robust feature matching.

### 2.4.2. Multi-view metrics

The metrics specified for multi-view image fusion are mostly task-specific metrics, such as 3D object detection. Two metrics are widely used in this field: AP3D and APBEV.

AP3D (Average Precision in 3D) and APBEV (Average Precision in Bird's Eye View) are critical evaluation metrics for 3D object detection, widely used in autonomous driving and robotics. AP3D assesses the accuracy of detecting objects within a 3D space by comparing the predicted 3D bounding boxes to the ground truth, considering their position, size, and orientation. It relies on the Intersection over Union (IoU) metric calculated in 3D space, defined as:

$$IoU_{3D} = \frac{3DboxesIntersectionVolumes}{3DBoxesUnionVolumes} \qquad (17)$$

A detection is considered a true positive if the IoU 3D exceeds a predefined threshold. The AP3D metric is then computed as the area under the precision-recall curve, which captures the trade-off between precision and recall for various IoU thresholds:

$$AP_{3D} = \int_{1}^{0} Precision(R) dR \qquad (18)$$

where R represents the recall.

On the other hand, APBEV focuses on object detection accuracy from a top-down perspective, evaluating the x and y coordinates of the objects while ignoring the height dimension. It uses a similar IoU metric but in the 2D Bird's Eye View (BEV) plane:

$$IoU_{BEV} = \frac{BEV\ Boxes\ Intersection\ Area}{BEV\ Boxes\ Union\ Area} \qquad (19)$$

The APBEV metric also utilizes a precision-recall curve specific to the BEV perspective, computing the area under this curve to quantify performance:

$$AP_{BEV} = \int_{1}^{0} Precision_{BEV}(R) dR \qquad (20)$$

where R represents the recall.

Some other metrics described by Shu et al. [5] are used to evaluate the performance of 3D object detection and are defined as follows: mean Average Translation Error (mATE) measures the average error in the translation of detected objects relative to their actual position in 3D space, assessing the accuracy of object localization.

Mean Average Scale Error (mASE) evaluates the average error in the scale of detected objects, indicating the difference between the actual size of the objects and their detected size, which is crucial for accurate object dimension detection.

Mean Average Orientation Error (mAOE) quantifies the average error in the orientation of detected objects, highlighting discrepancies between the actual and detected orientation, which is vital for applications requiring precise orientation determination, such as autonomous navigation.

Mean Average Velocity Error (mAVE) calculates the average error in the velocity of moving objects, comparing estimated velocities with actual velocities, essential for systems that track object dynamics, such as autonomous vehicles.

Finally, mean Average Attribute Error (mAAE) measures the average error concerning the attributes of detected objects, such as object class or other specific characteristics, thus assessing the accuracy of object classification and identification.

### 2.4.3. Multi-view datasets

As highlighted in state-of-the-art literature, two primary approaches dominate the field of multi-view image fusion: task-driven methods and those focused on 3D reconstruction as the fusion objective. For multi-view object detection, the most common applications are vehicle and object recognition in autonomous driving scenarios. Datasets such as nuScenes [147] and KITTI [148] are widely used for this purpose, offering multi-view imagery and annotations tailored for object detection tasks in driving environments. On the other hand, 3D reconstruction represents another critical use case for multi-view image fusion.
<table><caption>**Table 2**
Summary of all image fusion methods by category.</caption><thead><tr><th>Fusion category</th><th>Methods</th><th>Classification and description</th><th>Year</th><th>Code</th></tr></thead><tbody><tr><td rowspan="5">Multi-Focus</td><td>Liu et al. [32]</td><td>Supervised, CNN-based method</td><td>2017</td><td>-</td></tr><tr><td>IFCNN [33]</td><td>Supervised, convolutional network with feature extraction and fusion</td><td>2020</td><td>https://github.com/uzeful/IFCNN</td></tr><tr><td>MADCNN [36]</td><td>Supervised, CNN and attention-based multi-scale</td><td>2019</td><td>https://github.com/jtguan/MADCNN</td></tr><tr><td>FuseGAN [39]</td><td>Unsupervised, GAN-based framework</td><td>2019</td><td>-</td></tr><tr><td>DRPL [37]</td><td>Unsupervised, pixel-wise regression model</td><td>2020</td><td>https://github.com/Redaimao/DRPL</td></tr><tr><td rowspan="5">Multi-Exposure</td><td>End-to-End exposure Fusion [58]</td><td>Supervised, lightweight CNN for direct multi-exposure fusion</td><td>2018</td><td>-</td></tr><tr><td>MEF-Net [61]</td><td>Supervised, guided fusion using context aggregation network</td><td>2019</td><td>https://github.com/makedede/MEFNet</td></tr><tr><td>DeepFuse [59]</td><td>Unsupervised, CNN-based luminance and chrominance fusion</td><td>2017</td><td>https://github.com/KRamPrabhakar/DeepFuse</td></tr><tr><td>MEF-GAN [62]</td><td>Unsupervised, GAN with self-attention mechanism</td><td>2020</td><td>https://github.com/hanna-xu/MEF-GAN</td></tr><tr><td>GANFuse [63]</td><td>Unsupervised, adversarial loss for extreme exposure fusion</td><td>2021</td><td>-</td></tr><tr><td rowspan="20">Multi-Modal</td><td>CentralNet [75]</td><td>Unsupervised, Bi-branch CNN with feature-level fusion</td><td>2018</td><td>-</td></tr><tr><td>DenseFuse [74]</td><td>Unsupervised, DenseNet with encoder-decoder structure</td><td>2019</td><td>https://github.com/hli1221/imagefusion_densefuse</td></tr><tr><td>NestFuse [78]</td><td>Unsupervised, nested encoder-decoder with attention layers</td><td>2020</td><td>https://github.com/hli1221/imagefusion-nestfuse</td></tr><tr><td>RFN-Nest [77]</td><td>Unsupervised, nested residual network with dense skip connections</td><td>2021</td><td>https://github.com/hli1221/imagefusion-rfn-nest</td></tr><tr><td>Image Fusion Transformer [2]</td><td>Unsupervised, transformer-based multi-scale fusion</td><td>2022</td><td>https://github.com/Vibashan/Image-Fusion-Transformer</td></tr><tr><td>MDAN [83]</td><td>Unsupervised, attention-based multi-scale encoder-decoder</td><td>2022</td><td>-</td></tr><tr><td>CMAFusion [82]</td><td>Unsupervised, cross-modal attention for IR/VIS fusion</td><td>2023</td><td>-</td></tr><tr><td>STFNet [79]</td><td>Unsupervised, self-attention and deformable convolution</td><td>2024</td><td>-</td></tr><tr><td>FusionGAN [85]</td><td>Unsupervised, adversarial learning for cross-modal fusion</td><td>2019</td><td>https://github.com/jiayi-ma/FusionGAN</td></tr><tr><td>GANMcC [86]</td><td>Unsupervised, GAN with multi-path encoding</td><td>2020</td><td>https://github.com/HaoZhang1018/GANMcC</td></tr><tr><td>DDcGAN [87]</td><td>Unsupervised, dual decoder for multi-resolution fusion</td><td>2020</td><td>https://github.com/hanna-xu/DDcGAN</td></tr><tr><td>GAN-FM [90]</td><td>Unsupervised, GAN with full-scale feature mapping</td><td>2021</td><td>https://github.com/yuanjiteng/GAN-FM</td></tr><tr><td>Dense-FG [84]</td><td>Unsupervised, GAN architecture with dense feature connections</td><td>2023</td><td>-</td></tr><tr><td>DDFM [92]</td><td>Unsupervised, diffusion model for multi-modal fusion</td><td>2023</td><td>https://github.com/Zhaozixiang1228/MMIF-DDFM</td></tr><tr><td>SeAFusion [103]</td><td>Unsupervised, attention-based encoder-decoder</td><td>2022</td><td>https://github.com/Linfeng-Tang/SeAFusion</td></tr><tr><td>MF-Net [96]</td><td>Task-Driven, encoder-decoder with segmentation-focused loss</td><td>2017</td><td>https://github.com/haqishen/MFNet-pytorch</td></tr><tr><td>RD-ML [94]</td><td>Task-Driven, dual-branch architecture with gated fusion</td><td>2018</td><td>-</td></tr><tr><td>RTFNet [97]</td><td>Task-Driven, ResNet encoders with specialized decoders</td><td>2019</td><td>https://github.com/yuxiangsun/RTFNet</td></tr><tr><td>GMFNet [1]</td><td>Task-Driven, segmentation-driven multi-branch fusion</td><td>2020</td><td>-</td></tr><tr><td>TarDal [100]</td><td>Task-Driven, cooperative loss for detection tasks</td><td>2022</td><td>https://github.com/JinyuanLiu-CV/TarDAL</td></tr><tr><td>OAFA [99]</td><td>Task-Driven, visual transformers with deformable convolutions</td><td>2024</td><td>-</td></tr><tr><td>SCGRFuse [102]</td><td>Task-Driven, visual transformer with spatial-channel attention</td><td>2024</td><td>-</td></tr><tr><td>CocoNet [104]</td><td>Task-Driven, contrastive learning with multi-attention modules</td><td>2024</td><td>https://github.com/runjia0124/CoCoNet</td></tr><tr><td rowspan="12">Multi-View</td><td>SuperGlue [142]</td><td>Unsupervised, GNN for robust image matching</td><td>2020</td><td>https://github.com/magicleap/SuperGluePretrainedNetwork</td></tr><tr><td>LightGlue [146]</td><td>Unsupervised, efficient feature matching</td><td>2023</td><td>https://github.com/cvg/LightGlue</td></tr><tr><td>OmniGlue [145]</td><td>Unsupervised, graph-based feature matching</td><td>2024</td><td>https://github.com/google-research/omniglue</td></tr><tr><td>ROMA [121]</td><td>Unsupervised, Transformer-based feature matching</td><td>2024</td><td>https://github.com/Parskatt/RoMa</td></tr><tr><td>3D Object Proposal [136]</td><td>Task-Driven, proposal generation for 3D detection</td><td>2017</td><td>-</td></tr><tr><td>DSGN [4]</td><td>Task-Driven, voxel-based stereo depth estimation</td><td>2020</td><td>https://github.com/dvlab-research/DSGN</td></tr><tr><td>IDA-3D [128]</td><td>Task-Driven, stereo depth estimation with instance-level fusion</td><td>2020</td><td>https://github.com/swords123/IDA-3D</td></tr><tr><td>End-to-End Pseudo-LiDAR [120]</td><td>Task-Driven, simulated LiDAR with object detection</td><td>2020</td><td>https://github.com/mileyan/pseudo-LiDAR_e2e</td></tr><tr><td>CG-Stereo [124]</td><td>Task-Driven, stereo depth with confidence mapping</td><td>2020</td><td>-</td></tr><tr><td>DISP R-CNN [135]</td><td>Task-Driven, stereo refinement with object detection</td><td>2020</td><td>https://github.com/zju3dv/disprcnn</td></tr><tr><td>SIDE [3]</td><td>Task-Driven, bird's-eye view localization with structured attention</td><td>2022</td><td>-</td></tr><tr><td>3DPPE [5]</td><td>Task-Driven, point-cloud generation from depth</td><td>2023</td><td>https://github.com/drilistbox/3DPPE</td></tr><tr><td>TEMPO [139]</td><td>Task-Driven, temporal multi-view fusion</td><td>2023</td><td>-</td></tr><tr><td rowspan="3">Multi-Category</td><td>FusionDN [152]</td><td>Unsupervised, unified approach for cross-category fusion</td><td>2020</td><td>https://github.com/hanna-xu/FusionDN</td></tr><tr><td>MVMM Net [17]</td><td>Task-Driven, multi-modal and multi-view fusion</td><td>2021</td><td>-</td></tr><tr><td>U2 Fusion [16]</td><td>Task-Driven, U-Net based fusion for multi-category tasks</td><td>2022</td><td>https://github.com/hanna-xu/U2Fusion</td></tr></tbody></table>

Datasets like ScanNet [149] support work on indoor scene reconstruction, while MegaDepth [150] and WxBS [151] enable the development and evaluation of methods focused on outdoor and monument scenes. These datasets [10] provide diverse benchmarks to advance both task-driven and 3D reconstruction-focused methodologies in multi-view image fusion.

### 2.4.4. Multi-view applications

Multi-view image fusion has numerous applications across various fields. These methods can be applied to any domain where it is necessary to fuse multiple images captured from different viewpoints to solve a specific problem. One of the most popular applications is in autonomous vehicle navigation [120,135]. Autonomous vehicles
<figure><img src="image_11.png" /><figcaption>Fig. 8. Comparison of APBEV and AP3D performances (IoU = 0.7) for different methods according to three difficulty levels (Easy, Mode, Hard) on the KITTI Dataset [148].</figcaption></figure>

<table><caption>**Table 3**
Datasets used in the field of multi-focus image fusion.</caption><thead><tr><td>Dataset</td><td>Subject</td><td>Year</td></tr></thead><tbody><tr><td>NYU-D2 [42]</td><td>Indoor Scene</td><td>2012</td></tr><tr><td>PASCAL VOC [43]</td><td>Various Object</td><td>2012</td></tr><tr><td>ILSVRC [36]</td><td>Various Object</td><td>2015</td></tr><tr><td>MFFW [44]</td><td>Photography</td><td>2020</td></tr></tbody></table>

require multiple cameras to make effective driving decisions, initially to provide a 360-degree view around the vehicle and also to perform depth estimation using only cameras. Another interesting application of multi-view image fusion is in 3D reconstruction, which has a wide range of uses. For example, ROMA [121] enables the 3D reconstruction of monuments from a series of images taken from various viewpoints of these structures. On the other hand, methods like NeRF [155] are primarily used for 3D object reconstruction. Finally, this paper also presents pose estimation as an application of multi-view image fusion. With methods such as TEMPO [139], it is possible to estimate the position of a human body in a 3D scene using multiple images that capture the scene from different angles. Beyond these, multi-view fusion is also widely used in robotics, where fusing images from multiple onboard cameras enhances environmental perception and object manipulation in cluttered or dynamic environments. In medical imaging, multi-view X-ray fusion (e.g., biplanar radiographs) helps reconstruct 3D anatomical structures with low radiation exposure. In remote sensing, aerial or satellite images taken from different angles or orbits can be fused to improve topographic modelling and change detection. Furthermore, in virtual and augmented reality, multi-view fusion contributes to realistic scene rendering and accurate spatial localization of users and virtual objects.

### 2.4.5. Multi-view experiments

The Table 11 or Fig. 8, showcase the performance of various multi-view image fusion methods, evaluated using the metrics APBEV (Average Precision in Bird's Eye View) and AP3D (Average Precision in 3D) under different difficulty levels (Easy, Moderate, Hard) and for two IoU (Intersection over Union) thresholds, 0.5 and 0.7. Analysing the results for an IoU of 0.5, the CG-Stereo method achieves exceptionally high scores, reaching up to 97.04 in AP3D under easy conditions, indicating excellent performance for 3D object detection. However, for a stricter IoU of 0.7, although the overall scores decrease, CG-Stereo still maintains relatively high performance compared to other methods, especially under difficult conditions. In contrast, the DSGN method, despite showing reasonable performance at an IoU of 0.7, does not have available results for an IoU of 0.5, which may suggest specific limitations or a partial evaluation of this method. Other methods, such as IDA-3D, SIDE, and E2E Pseudo Lidar, demonstrate intermediate performances, with more pronounced declines in precision under challenging conditions and for stricter IoU thresholds. These results underscore the

importance of selecting appropriate image fusion methods based on the required accuracy and application conditions.

Continuing the analysis for an IoU of 0.7, it is evident that CG-Stereo stands out for its robustness, with scores of 87.31/76.17 in easy conditions and 65.80/54.63 in difficult conditions, indicating an ability to maintain high accuracy even as conditions become more complex. In comparison, the E2E Pseudo Lidar method also shows good performance, particularly under easy conditions (82.7/71.1), but experiences a more significant drop in difficult conditions (58.4/46.7), which could indicate increased sensitivity to scene complexity.

The SIDE method, while offering solid performance, shows a progressive decline in scores with increasing difficulty and IoU thresholds, decreasing from 88.35/87.70 in easy conditions to 46.16/37.15 in difficult conditions at an IoU of 0.7. This reflects moderate adaptation to the challenges of 3D object detection under varying conditions. Similarly, IDA-3D shows a notable decrease in performance, especially for an IoU of 0.7, where scores drop to 42.93/32.23 in difficult conditions, illustrating the limitations in handling more complex environments.

Conversely, the DSGN method, despite having available performance results only for an IoU of 0.7, exhibits competitive scores, suggesting that it may be optimized for higher IoU thresholds, which could be advantageous for certain applications requiring stricter precision.

Finally, Disp-RCNN, while showing competitive scores at an IoU of 0.5 in easy conditions (90.38), suffers a notable decline in performance at an IoU of 0.7, with scores dropping to 63.57 in easy conditions and 39.73 in difficult conditions. This could suggest a challenge in maintaining high accuracy under stricter criteria, or potential difficulties in managing depth and complex 3D perception.

### 2.5. Synthesis

The analysis of fusion methods and experimental results across distinct fusion categories has enabled the identification of the most effective architectures. This process serves two primary objectives. First, it supports the assessment of which architectural or learning paradigms achieve superior performance, thereby informing the development of future multi-category image fusion systems. Second, it reveals architectural features that recur across categories, suggesting potential design principles for unified fusion models. A summary of these methods is provided in Table 2. Task-driven approaches are particularly prevalent in the multi-view fusion domain, with notable examples including SIDE [3] and DISP-RCNN [135]. Comparable architectures have also demonstrated strong performance in multi-modal fusion, as illustrated by TarDAL [100] and SeAFusion [103]. A key strength of task-driven learning lies in its reliance on explicit supervision via task-specific ground truth, which facilitates stable convergence and improved learning efficiency. In contrast, unsupervised approaches depend on surrogate objectives or handcrafted quality metrics, which may not align as directly with downstream performance.

To support the evaluation of these methods, two summary tables have been compiled. Table 13 enumerates the principal datasets used in
<table><caption>**Table 4**
Comparison of various multi-focus image fusion methods on the ILSVRC dataset [36].</caption><thead><tr><th>Methods</th><th>Qmi (1)</th><th>Qncie (2)</th><th>Qg (3)</th><th>Qp (4)</th><th>Qe (5)</th><th>Qy (6)</th><th>Qcb (7)</th><th>Qcv (8)</th></tr></thead><tbody><tr><td>CNN [32]</td><td>1,1512</td><td>0,8429</td><td>0,7615</td><td>0,8506</td><td>0,8808</td><td>0,9875</td><td>0,8084</td><td>16,2763</td></tr><tr><td>MADCNN [36]</td><td>1,1053</td><td>0,8394</td><td>0,7564</td><td>0,8428</td><td>0,8818</td><td>0,968</td><td>0,7702</td><td>16,0603</td></tr></tbody></table>

<table><caption>**Table 5**
Datasets used in the field of multi-exposure image fusion.</caption><thead><tr><th>Dataset</th><th>Subject</th><th>Year</th></tr></thead><tbody><tr><td>MEFB [153]</td><td>Outdoor scene</td><td>2021</td></tr><tr><td>Large-Scale MEF Dataset [69]</td><td>Outdoor scene</td><td>2018</td></tr><tr><td>TrafficSign [68]</td><td>Road scene</td><td>2021</td></tr><tr><td>Perceptual MEF [154]</td><td>Various scene</td><td>2022</td></tr><tr><td>MEF-Net Dataset [61]</td><td>Various scene</td><td>2019</td></tr><tr><td>DeepFuse dataset [59]</td><td>Indoor/Outdoor scene</td><td>2017</td></tr></tbody></table>

<table><caption>**Table 6**
Comparison of different multi-exposure image fusion methods on large-scale MEF dataset [69].</caption><thead><tr><th>Methods</th><th>PSNR (11)</th><th>CC (16)</th><th>MSSIM [65]</th><th>SD (12)</th></tr></thead><tbody><tr><td>DeepFuse [59]</td><td>67,0045</td><td>0,9389</td><td>0,9813</td><td>0,1946</td></tr><tr><td>FLER</td><td>63,4568</td><td>0,7304</td><td>0,9789</td><td>0,1532</td></tr><tr><td>MEF-GAN [62]</td><td>68,3721</td><td>0,942</td><td>0,982</td><td>0,2103</td></tr></tbody></table>

<table><caption>**Table 7**
Datasets used in the field of multi-modal image fusion.</caption><thead><tr><th>Dataset</th><th>Modality</th><th>Subject</th><th>Year</th></tr></thead><tbody><tr><td>TNO [110]</td><td>Multi-Spectral</td><td>Military scenario</td><td>2014</td></tr><tr><td>RoadScene [16]</td><td>Infrared-Visible</td><td>Road scene</td><td>2022</td></tr><tr><td>MSRS [111]</td><td>Infrared-Visible</td><td>Road scene</td><td>2022</td></tr><tr><td>M3FD [100]</td><td>Infrared-Visible</td><td>Road scene</td><td>2022</td></tr><tr><td>MFNet Dataset [96]</td><td>Visible-Thermal</td><td>Road scene</td><td>2017</td></tr><tr><td>KAIST [113]</td><td>Visible-Thermal</td><td>Pedestrian detection</td><td>2015</td></tr><tr><td>Harvard Dataset [114]</td><td>MRI-PET</td><td>Medical</td><td>2022</td></tr><tr><td>LLVIP [112]</td><td>Infrared-Visible</td><td>Road scene</td><td>2021</td></tr></tbody></table>

image fusion, organized by domain, subject matter, and year of release. This compilation provides researchers with a structured overview of selecting relevant training and benchmarking datasets. Table 12 complements this by detailing the core evaluation metrics employed in the literature. Each entry includes the metric's formulation and the specific fusion categories in which it is applicable. These include information-theoretic measures (e.g., Mutual Information, $Q_{MI}$), perceptual quality indices (e.g., SSIM, QY), and task-oriented metrics (e.g., 3D IoU, BEV AP) commonly used in detection settings.

After identifying effective learning paradigms, we examine the architectural trends among high-performing models. In conventional fusion domains, such as multi-modal, multi-exposure, and multi-focus, GAN-based frameworks have shown competitive results, as demonstrated by MEF-GAN [62] and TarDAL [100]. In contrast, multi-view fusion typically incorporates modules that model geometric relationships or 3D structure, such as pseudo-LiDAR [120]. Attention-based architectures, in particular, have emerged as consistently effective across multiple fusion categories. Models such as SeAFusion [103], MDAN [83], and 3DPPE [5] leverage spatial and cross-modal attention mechanisms to enhance fusion performance by dynamic weighting informative regions.

In summary, the integration of task-driven learning and attention-based architectures represents a promising direction for advancing robust and generalizable image fusion systems. Future work should focus on extending these strategies across fusion modalities to support the development of unified, multi-category fusion frameworks.

### 3. Multi-category image fusion methods

Mono-category image fusion addresses challenges associated with a specific type of category, such as multi-modal, multi-view, multi-focus, or multi-exposure fusion. These methods are typically tailored to their specific domain but can sometimes be adapted to other categories. A new taxonomy, Mono-category/ Multi-category, distinguishes between the methods applicable to a single type of category and those that can be applied across multiple types. Multi-category methods are particularly valuable in scenarios where an application requires the fusion of images from various categories. Although the state-of-the-art research on multi-category fusion is limited, this work aims to present pioneering methods and guide future research directions in this area. Given the traditional distinctions in the literature, two types of multi-category approaches exist: methods that combine classical categories (multi-modal, multi-exposure, multi-focus) referred to as Classical Multi-category, and methods that integrate all category types, known as Generalist Multi-category.

#### 3.1. Classical multi-category image fusion methods

Classical Multi-category Image Fusion Methods encompass techniques that are capable of performing well across tasks such as multi-modal, multi-exposure, and multi-focus fusion. The advantage of these architectures is their potential to evolve into systems that can handle any type of image fusion, regardless of the image source. However, a key challenge is ensuring that these methods perform as well as, or even better than, Mono-category methods on their specific tasks. The risk lies in potentially sacrificing specialized performance in pursuit of generalization across fusion tasks. Nonetheless, these methods pave the way for fusion tasks that combine multiple types of categories.

A first approach is proposed by FusionDN [152]. FusionDN is an unsupervised architecture design, initially for multi-modal image fusion. FusionDN primarily utilizes a densely connected network, DenseNet, which comprises convolutional layers and dense connection blocks. Each input image contains varying amounts of information. To address this variability, a feature retention rate is determined for the different data sources. A weighting block generates weights for the different sources, which are then transferred to the cost function of DenseNet. DenseNet is trained to produce an output image that contains the important features according to its loss and the generated weights. The cost function includes an SSIM (9). FusionDN can be extended to other types of categories. The paper includes a specific data-flow to learn across different tasks. The model is trained on multiple tasks sequentially, employing an Elastic Weight Consolidation (EWC) [156] algorithm to mitigate the issue of catastrophic forgetting.

U2Fusion [16] (Unified Unsupervised Image Fusion Network) is also an unsupervised image fusion method applicable to the three types of categories: multi-modal, multi-exposure, and multi-focus. The architecture is similar to FusionDN [152], employing a convolutional model called DenseNet to generate a fused image from the concatenation of two input images. However, U2Fusion integrates a cost function computation branch using a feature extraction model. This model extracts information from the images to calculate Information Measurement and Information Preservation Degree. The cost function utilizes the MSE between the input and output images in addition to an SSIM (9). Furthermore, U2Fusion employs a specific training workflow designed to learn across multiple tasks by conducting sequential training sessions while retaining a portion of the data and information between each train.
<table><caption>**Table 8**

Comparison of various multi-modal image fusion methods on the M3FD dataset [100].</caption><thead><tr><th>Methods</th><th>SD (12)</th><th>MI (10)</th><th>VIF</th><th>AG (14)</th><th>CC (16)</th><th>SCD</th><th>EN (13)</th><th>Qabf</th><th>SF (15)</th></tr></thead><tbody><tr><td>DenseFuse [74]</td><td>26.045</td><td>2.866</td><td>0.593</td><td>2.625</td><td>0.584</td><td>1.544</td><td>6.433</td><td>0.357</td><td>7.546</td></tr><tr><td>FusionGAN [85]</td><td>26.069</td><td>2.707</td><td>0.121</td><td>2.282</td><td>0.396</td><td>0.953</td><td>6.448</td><td>0.116</td><td>6.577</td></tr><tr><td>DDcGAN [87]</td><td>40.730</td><td>2.541</td><td>0.580</td><td>4.539</td><td>0.540</td><td>1.605</td><td>7.150</td><td>0.480</td><td>12.787</td></tr><tr><td>Tardal [100]</td><td>42.916</td><td>3.191</td><td>0.606</td><td>4.128</td><td>0.510</td><td>1.552</td><td>7.151</td><td>0.408</td><td>12.516</td></tr><tr><td>MDAN [83]</td><td>50.023</td><td>2.935</td><td>0.759</td><td>5.049</td><td>0.551</td><td>1.856</td><td>7.431</td><td>0.509</td><td>14.519</td></tr><tr><td>SeAFusion [103]</td><td>33.575</td><td>3.195</td><td>0.663</td><td>4.687</td><td>0.537</td><td>1.613</td><td>6.800</td><td>0.564</td><td>13.123</td></tr></tbody></table>

<table><caption>**Table 9**

Comparison of different multi-modal image fusion methods on the TNO dataset [110].</caption><thead><tr><th>Methods</th><th>SD (12)</th><th>MI (10)</th><th>VIF</th><th>AG (14)</th><th>CC (16)</th><th>SCD</th><th>EN (13)</th><th>Qabf</th><th>SF (15)</th></tr></thead><tbody><tr><td>DenseFuse [74]</td><td>26.033</td><td>2.096</td><td>0.561</td><td>2.498</td><td>0.528</td><td>1.617</td><td>6.281</td><td>0.335</td><td>6.478</td></tr><tr><td>FusionGAN [85]</td><td>30.936</td><td>1.912</td><td>0.168</td><td>2.337</td><td>0.393</td><td>1.232</td><td>6.569</td><td>0.133</td><td>5.580</td></tr><tr><td>DDcGAN [87]</td><td>46.283</td><td>1.845</td><td>0.519</td><td>4.574</td><td>0.485</td><td>1.607</td><td>7.373</td><td>0.344</td><td>11.280</td></tr><tr><td>Tardal [100]</td><td>46.001</td><td>2.739</td><td>0.632</td><td>4.172</td><td>0.474</td><td>1.659</td><td>7.120</td><td>0.415</td><td>11.826</td></tr><tr><td>MDAN [83]</td><td>49.462</td><td>2.252</td><td>0.662</td><td>4.971</td><td>0.507</td><td>1.869</td><td>7.183</td><td>0.424</td><td>13.196</td></tr><tr><td>SeAFusion [103]</td><td>37.589</td><td>3.069</td><td>0.687</td><td>4.088</td><td>0.481</td><td>1.700</td><td>6.871</td><td>0.509</td><td>10.083</td></tr></tbody></table>

<table><caption>**Table 10**

Datasets used in the field of multi-view image fusion.</caption><thead><tr><th>Dataset</th><th>Task</th><th>Subject</th><th>Year</th></tr></thead><tbody><tr><td>nuScenes [147]</td><td>Object detection</td><td>Autonomous driving</td><td>2019</td></tr><tr><td>KITTI [148]</td><td>Object detection</td><td>Autonomous driving</td><td>2012</td></tr><tr><td>ScanNet [149]</td><td>3D reconstruction</td><td>Indoor scene</td><td>2017</td></tr><tr><td>MegaDepth [150]</td><td>3D reconstruction</td><td>Monument scene</td><td>2018</td></tr><tr><td>WxBS [151]</td><td>3D reconstruction</td><td>Outdoor scene</td><td>2015</td></tr></tbody></table>

<table><caption>**Table 11**

Comparison of various multi-view image fusion methods on the KITTI dataset [148].</caption><thead><tr><th rowspan="2">Method</th><th colspan="3">APBEV/AP3D(IoU=0.5) (18)</th></tr><tr><th>Easy</th><th>Mode</th><th>Hard</th></tr></thead><tbody><tr><td>IDA-3D [128]</td><td>88.05/87.08</td><td>76.69/74.57</td><td>67.29/60.01</td></tr><tr><td>SIDE [3]</td><td>88.35/87.70</td><td>76.01/69.13</td><td>67.46/60.05</td></tr><tr><td>CG-Stereo [124]</td><td>90.58/97.04</td><td>87.01/88.58</td><td>79.76/80.34</td></tr><tr><td>E2E Pseudo Lidar [120]</td><td>90.5/90.4</td><td>84.4/79.2</td><td>78.4/75.75,9</td></tr><tr><td>Disp R-CNN [135]</td><td>90,38</td><td>79,77</td><td>69,81</td></tr></tbody><thead><tr><th rowspan="2">Method</th><th colspan="3">APBEV/AP3D(IoU=0.7) (18)</th></tr><tr><th>Easy</th><th>Mode</th><th>Hard</th></tr></thead><tbody><tr><td>DSGN [4]</td><td>83.24/72.31</td><td>63.91/54.27</td><td>57.83/47.71</td></tr><tr><td>IDA-3D [128]</td><td>70.68/54.97</td><td>50.21/37.45</td><td>42.93/32.23</td></tr><tr><td>SIDE [3]</td><td>72.75/61.22</td><td>53.71/44.46</td><td>46.16/37.15</td></tr><tr><td>CG-Stereo [124]</td><td>76.17/87.31</td><td>57.82/68.69</td><td>54.63/65.80</td></tr><tr><td>E2E Pseudo Lidar [120]</td><td>82.7/71,1</td><td>65.7/51,7</td><td>58.4/46,7</td></tr><tr><td>Disp R-CNN [135]</td><td>63,57</td><td>47,15</td><td>39,73</td></tr></tbody></table>

In summary, these methods employ sequential learning to address the challenges of classical multi-category image fusion. These approaches enable the application of previously discussed fusion methods within a multi-category and multitask context. However, it is observed that the algorithmic structures, models, and techniques used for multi-modal, multi-exposure, and multi-focus fusion are quite similar, allowing them to be easily integrated. This is not necessarily the case for multi-view fusion methods.

### 3.1.1. Experiments

The following experiments adhere to the same experimental protocol as the previous ones. The objective here is to compare the classical multi-category methods with other mono-category methods. To ensure a fair and consistent comparison, the selected methods have similar architectures, maintaining coherence in the evaluation.

The analysis of the presented methods reveals differences in performance between multi-modal methods (DenseFuse, FusionGAN, DDcGAN) and multi-category methods (FusionDN, U2Fusion), particularly in terms of image fusion quality.

Among the multi-modal methods, DenseFuse [74] shows the highest CC (16) (0.565), indicating a relatively better correlation with the reference images. However, its PSNR (11) (61.126) is slightly lower than FusionGAN [85], which achieves a PSNR (11) of 61.341 but at the cost of a lower CC (16) (0.494), implying a trade-off between correlation and noise reduction. DDcGAN [87] performs the weakest in terms of both CC (16) (0.506) and PSNR (11) (60.051), indicating poorer image quality reconstruction and a correlation compared to the other methods.

In contrast, the multi-category methods FusionDN [152] and U2Fusion [16] outperform the multi-modal methods across most metrics. U2Fusion achieves the highest PSNR (11) (63.305) and SSIM (9) (1.9909), reflecting its superior image quality and structural similarity preservation. FusionDN, while slightly lower in performance compared to U2Fusion, still outperforms the multi-modal methods with a CC (16) of 0.627 and a PSNR (11) of 61.684. Both FusionDN and U2Fusion also exhibit higher spatial consistency (SCD scores of 1.778 and 1.635, respectively), indicating better preservation of scene detail during fusion.

On another analysis, we compare the performance of multi-exposure methods (DeepFuse, FLER) with multi-category methods (FusionDN, U2Fusion) based on the metrics provided.

Among the multi-exposure methods, DeepFuse [59] shows strongest performance with a high SSIM (9) (1.967), indicating good structural similarity retention, and a PSNR (11) of 58.518, reflecting its ability to reduce noise while maintaining image quality. Its CC (16) value (0.84) is also the highest among all methods, suggesting a strong correlation with reference images. In contrast, FLER performs the weakest in almost every metric, with the lowest CC (16) (0.518) and PSNR (11) (57.77), implying a significant drop in image quality and correlation. However, FLER achieves a higher edge intensity (0.235) compared to DeepFuse, which may indicate that it retains finer details in certain areas, but at the cost of overall quality.

On the other hand, the multi-category methods, FusionDN [152] and U2Fusion [16], generally perform better or on par with the multi-exposure methods. U2Fusion achieves the highest SSIM (9) (1.968) and a PSNR (11) (58.454) close to DeepFuse, indicating comparable image quality and structure preservation. Its CC (16) (0.831) is slightly lower than DeepFuse but still competitive, demonstrating its robustness in correlating with reference images. FusionDN also shows strong performance, with a higher edge intensity (0.277), suggesting that it captures finer image details more effectively than the other methods.

The results shown in Tables 14 and 15, demonstrate that methods with architectures resembling those of each fusion category can outperform the specialized methods within those categories. The methods
<table><caption>**Table 12**
Summary of evaluation metrics in image fusion.</caption><thead><tr><th>Metric name</th><th>Formula</th><th>Application domain</th></tr></thead><tbody><tr><td>Q<sub>MI</sub></td><td>2 · I(F, A) · I(F, B) / (I(F, A) + I(F, B))</td><td>Multi-focus, Multi-modal</td></tr><tr><td>Q<sub>NCIE</sub></td><td>H(F) - H(F|AB)</td><td>Multi-focus</td></tr><tr><td>Q<sub>G</sub></td><td>(Σ<sup>N</sup><sub>i=1</sub> |G<sub>F</sub>(i)| · max(|G<sub>A</sub>(i)|, |G<sub>B</sub>(i)|) / (Σ<sup>N</sup><sub>i=1</sub> (|G<sub>A</sub>(i)| + |G<sub>B</sub>(i)|))</td><td>Multi-focus, Multi-exposure</td></tr><tr><td>Q<sub>P</sub></td><td>(Σ<sup>N</sup><sub>i=1</sub> PC<sub>F</sub>(i) · max(PC<sub>A</sub>(i), PC<sub>B</sub>(i)) / (Σ<sup>N</sup><sub>i=1</sub> (PC<sub>A</sub>(i) + PC<sub>B</sub>(i)))</td><td>Multi-focus</td></tr><tr><td>Q<sub>E</sub></td><td>1/N Σ<sup>N</sup><sub>i=1</sub> SSIM(F(i), A(i)) · SSIM(F(i), B(i))</td><td>Multi-focus</td></tr><tr><td>Q<sub>Y</sub></td><td>1/N Σ<sup>N</sup><sub>i=1</sub> (2 · SSIM(F(i), A(i)) · SSIM(F(i), B(i)) / SSIM(F(i), A(i)) + SSIM(F(i), B(i)))</td><td>Multi-focus</td></tr><tr><td>Q<sub>CB</sub></td><td>1/N Σ<sup>N</sup><sub>i=1</sub> CB(F(i), A(i), B(i))</td><td>Multi-focus, Multi-modal</td></tr><tr><td>Q<sub>CV</sub></td><td>1/N Σ<sup>N</sup><sub>i=1</sub> CV(F(i), A(i), B(i))</td><td>Multi-focus, Multi-modal</td></tr><tr><td>SSIM</td><td>(2μ<sub>x</sub>μ<sub>y</sub> + C<sub>1</sub>)(2σ<sub>xy</sub> + C<sub>2</sub>) / (μ<sub>x</sub><sup>2</sup> + μ<sub>y</sub><sup>2</sup> + C<sub>1</sub>)(σ<sub>x</sub><sup>2</sup> + σ<sub>y</sub><sup>2</sup> + C<sub>2</sub>)</td><td>Multi-focus, Multi-exposure, Multi-modal</td></tr><tr><td>MI</td><td>Σ<sub>x,y</sub> P<sub>XY</sub>(x, y) log<sub>10</sub> (P<sub>XY</sub>(x, y) / P<sub>X</sub>(x)P<sub>Y</sub>(y))</td><td>Multi-modal, Multi-exposure</td></tr><tr><td>PSNR</td><td>10 · log<sub>10</sub> (MAX<sup>2</sup> / MSE)</td><td>Multi-exposure, Multi-modal</td></tr><tr><td>SD</td><td>√(1 / MN) Σ<sup>M</sup><sub>i=1</sub> Σ<sup>N</sup><sub>j=1</sub> (I<sub>ij</sub> - μ)<sup>2</sup></td><td>Multi-exposure, Multi-modal</td></tr><tr><td>EN</td><td>- Σ<sup>n</sup><sub>i=1</sub> p<sub>i</sub> log<sub>2</sub>(p<sub>i</sub>)</td><td>Multi-exposure, Multi-modal, Multi-focus</td></tr><tr><td>AG</td><td>1 / (M - 1)(N - 1) Σ<sup>M - 1</sup><sub>i=1</sub> Σ<sup>N - 1</sup><sub>j=1</sub> √((∂I/∂x)<sup>2</sup> + (∂I/∂y)<sup>2</sup>)</td><td>Multi-focus, Multi-exposure, Multi-modal</td></tr><tr><td>SF</td><td>√((SF<sub>H</sub>)<sup>2</sup> + (SF<sub>V</sub>)<sup>2</sup>)</td><td>Multi-focus, Multi-exposure, Multi-modal</td></tr><tr><td>CC</td><td>(Σ<sub>(A, -A)(B, -B)</sub> / √(Σ<sub>(A, -A)</sub><sup>2</sup> Σ<sub>(B, -B)</sub><sup>2</sup>))</td><td>Multi-exposure, Multi-modal</td></tr><tr><td>IoU<sub>3D</sub></td><td>V<sub>pred</sub> ∩ V<sub>gt</sub> / V<sub>pred</sub> ∪ V<sub>gt</sub></td><td>Multi-view (3D detection)</td></tr><tr><td>AP<sub>3D</sub></td><td>Area under 3D precision-recall curve</td><td>Multi-view (3D detection)</td></tr><tr><td>IoU<sub>BEV</sub></td><td>A<sup>BEV</sup><sub>pred</sub> ∩ A<sup>BEV</sup><sub>gt</sub> / A<sup>BEV</sup><sub>pred</sub> ∪ A<sup>BEV</sup><sub>gt</sub></td><td>Multi-view (Bird's Eye View detection)</td></tr><tr><td>AP<sub>BEV</sub></td><td>Area under BEV precision-recall curve</td><td>Multi-view (Bird's Eye View detection)</td></tr></tbody></table>

<table><caption>**Table 13**
Summary of image fusion datasets.</caption><thead><tr><th>Dataset name</th><th>Application domain</th><th>Subject</th><th>Year</th></tr></thead><tbody><tr><td>NYU-D2</td><td>Multi-Focus</td><td>Indoor scene</td><td>2012</td></tr><tr><td>PASCAL VOC</td><td>Multi-Focus</td><td>Various objects</td><td>2012</td></tr><tr><td>ILSVRC</td><td>Multi-Focus</td><td>Various objects</td><td>2015</td></tr><tr><td>MFFW</td><td>Multi-Focus</td><td>Photography</td><td>2020</td></tr><tr><td>MEFB</td><td>Multi-Exposure</td><td>Outdoor scene</td><td>2021</td></tr><tr><td>Large-Scale MEF</td><td>Multi-Exposure</td><td>Outdoor scene</td><td>2018</td></tr><tr><td>TrafficSign</td><td>Multi-Exposure</td><td>Road scene</td><td>2021</td></tr><tr><td>Perceptual MEF</td><td>Multi-Exposure</td><td>Various scenes</td><td>2022</td></tr><tr><td>MEF-Net</td><td>Multi-Exposure</td><td>Various scenes</td><td>2019</td></tr><tr><td>DeepFuse</td><td>Multi-Exposure</td><td>Indoor/Outdoor scene</td><td>2017</td></tr><tr><td>TNO</td><td>Multi-Modal</td><td>Infrared/Visible pairs</td><td>2014</td></tr><tr><td>RoadScene</td><td>Multi-Modal</td><td>Traffic/Urban</td><td>2016</td></tr><tr><td>LLVIP</td><td>Multi-Modal</td><td>Low-Light Visible/IR</td><td>2021</td></tr><tr><td>Lytro</td><td>Multi-Focus/Multi-View</td><td>Natural scenes</td><td>2016</td></tr><tr><td>Middlebury</td><td>Multi-View</td><td>Indoor/Outdoor scenes</td><td>2003</td></tr><tr><td>Flicker1024</td><td>Multi-View</td><td>Urban Street View</td><td>2020</td></tr></tbody></table>

presented here approach the multi-category problem through a multi-task paradigm, meaning a single architecture is employed to solve multiple tasks without explicitly considering the different types of input images. However, this analysis confirms that a single architecture is indeed capable of generalizing across multiple classical fusion categories.

<table><caption>**Table 14**
Comparison between classical multi-category methods and multi-modal methods on the roadscene [16] dataset.</caption><thead><tr><th>Methods</th><th>CC (16)</th><th>SSIM (9)</th><th>PSNR (11)</th><th>SCD</th></tr></thead><tbody><tr><td>DenseFuse [74]</td><td>0,565</td><td>1,9873</td><td>61,126</td><td>1,31</td></tr><tr><td>FusionGAN [85]</td><td>0,494</td><td>1,985</td><td>61,341</td><td>0,844</td></tr><tr><td>DDcGAN [87]</td><td>0,506</td><td>1,9805</td><td>60,051</td><td>1,187</td></tr><tr><td>FusionDN [152]</td><td>0,627</td><td>1,9866</td><td>61,684</td><td>1,778</td></tr><tr><td>U2Fusion [16]</td><td>0,635</td><td>1,9909</td><td>63,305</td><td>1,635</td></tr></tbody></table>

<table><caption>**Table 15**
Comparison between classical multi-category methods and multi-exposure methods on large-scale MEF dataset [69].</caption><thead><tr><th>Methods</th><th>SSIM (9)</th><th>Edge Intensity</th><th>PSNR (11)</th><th>CC (16)</th></tr></thead><tbody><tr><td>DeepFuse [59]</td><td>1,967</td><td>0,181</td><td>58,518</td><td>0,84</td></tr><tr><td>FLER</td><td>1,961</td><td>0,235</td><td>57,77</td><td>0,518</td></tr><tr><td>FusionDN [152]</td><td>1,965</td><td>0,277</td><td>57,78</td><td>0,801</td></tr><tr><td>U2Fusion [16]</td><td>1,968</td><td>0,24</td><td>58,454</td><td>0,831</td></tr></tbody></table>

## 3.2. Generalist multi-category image fusion methods

Generalist Multi-category Image Fusion methods specifically address the challenge of integrating classical category methods with multi-view image fusion techniques. The goal is to meet the demands of applications that require both multi-view and multi-modal, multi-focus, or multi-exposure fusion, while also proposing universal architectures
Sequential Learning Architecture

<figure><img src="image_12.png" /></figure>

Single Model Architecture

<figure><img src="image_13.png" /><figcaption>**Fig. 9.** Summary of generalist multi-category architectures: Generalist multi-category architectures are divided into two paradigms: multi-stage learning with a single model and single-stage learning with a unified model. The key difference is that the former allows for a single model to handle any fusion category problem but processes only one type of image at a time. In contrast, the latter enables simultaneous processing of all fusion categories, accommodating a diverse range of input images. This can be achieved through architectures based on transformers or feature matching, for example.</figcaption></figure>

for image fusion. Although the current state-of-the-art in this area is limited, this survey aims to pave the way and suggest future research directions. A preliminary approach to this challenge is exemplified by MVMM-NET [17]. The Multiview Multimodal Network is a task-driven architecture for the fusion of multi-view and multi-modal images, specifically designed for tumour classification (benign or malignant). The architecture employs a relatively straightforward information fusion system, utilizing four convolutional neural networks to encode each image. Each encoded image undergoes a flattening operation before being concatenated into a single vector. A multilayer perceptron (MLP) is then applied to predict the class of the object present across the different views and modalities. The cost function uses cross-entropy to measure the discrepancy between the architecture's output and the ground truth, ensuring accurate classification.

Three architectural frameworks have emerged from the state-of-the-art for developing Generalist Multi-category methods. The first approach involves multitask learning methodologies, as proposed by U2Fusion [16] and FusionDN [152]. The aim is to design a more comprehensive architecture capable of handling classical image fusion tasks while integrating generic components to address multi-view aspects. Through iterative learning across defined tasks, a single set of weights could be developed, enabling the architecture to manage different tasks effectively. This type of architecture necessitates mechanisms to control the transfer of information between learning tasks and to prevent forgetting, similar to those proposed in existing methods. However, this approach manages different tasks separately during inference and does not inherently fuse information across the various categories.

Architectures such as MVMM-NET [17] are designed to fully manage image fusion across multiple categories, including multi-view fusion. This second category of methods implements a model capable of encoding images from different categories and utilizing them within a complex fusion system. However, a limitation of the proposed method lies in its lightweight internal architecture, which may reduce its effectiveness. Future architectures for generalist multi-category fusion should employ more robust designs that can accurately represent and fuse information, potentially incorporating more sophisticated techniques, such as deep learning.

The final type of structure is a specialized architecture that enables image fusion through a unified framework, as previously mentioned.

The state-of-the-art demonstrates that feature matching methods can be effective for multi-modal image fusion, as seen in RIFT [157] and RDFM [158]. Additionally, feature matching techniques such as OmniGlue [145] and SuperGlue [142] have proven to be highly effective in handling multi-view data. These findings suggest that feature matching approaches hold significant potential for developing a common architecture applicable to all these categories. This type of single-model architecture presents a promising direction, as do transformer-based architectures that enable attention across multiple images simultaneously [83].

As illustrated in Fig. 9, two types of architectures are proposed as research directions: sequential learning architectures and single-model architectures like feature matching-based or transformer-based architectures. A key challenge lies in the cost functions and metrics, which differ significantly between classical image fusion methods and multi-view techniques. To address this, the Task-Driven paradigm can be employed, aligning these two constraints with the model's application and facilitating both learning and evaluation. In summary, regardless of the chosen approach, the Task-Driven learning paradigm offers a means to unify multi-category image fusion methods.

As observed in the experimental analysis, attention-based methods demonstrate particularly good performance overall and represent a promising approach for achieving high performance in generalist multi-category systems.

Such methods have the potential to revolutionize various fields. Firstly, in the medical domain, certain cases require the use of both multi-modal and multi-view data. For instance, breast cancer detection is one area where multi-modal and multi-view information are crucial for enhancing diagnostic accuracy and cancer cell recognition [17]. Generalist multi-category methods are also of particular interest in the field of autonomous driving. Several methods [1,2] employ multi-modal sensors to improve object recognition performance, while others [128,135] utilize multi-view sensors to perform depth estimation, thus enabling 3D object detection on the road. Methods that can effectively integrate both multi-view and multi-modal approaches could lead to significant advancements in this field.
## 4. Challenges and future research areas

In summary, future generalist multi-category approaches can draw inspiration from existing image fusion methods. Multi-exposure and multi-focus image fusion techniques are particularly notable, as many methods design supervised datasets to address specific issues. In contrast, unsupervised methods often resemble the more rudimentary approaches seen in multi-modal image fusion. Architectures for these methods can include multi-branch systems, each dedicated to a specific modality, fusion systems based on encoder-decoder frameworks, or multi-scale architectures. Learning methods vary widely, ranging from unsupervised techniques that rely on image quality metrics to more advanced approaches, such as those employing adversarial networks. Finally, task-driven methods enable learning from ground truth that is not the fused image itself, but rather the ground truth of a fusion-related task. This approach is particularly favoured in multi-view image fusion, which often employs similar architectures but with an additional component to provide prior knowledge of depth within the image.

Generalist multi-category image fusion methods represent a promising direction for the field, aiming to address the limitations of mon category approaches by seamlessly integrating multiple fusion types (e.g., multi-modal, multi-view, multi-focus, and multi-exposure). While these methods hold significant potential, their implementation introduces several fundamental challenges that span architectural design, dataset availability, performance optimization, and explainability. Addressing these issues is critical for advancing the state-of-the-art and broadening the applicability of image fusion technologies.

**Architectural Design Complexity:** One of the primary obstacles in the development of generalist multi-category methods lies in their architectural complexity. Existing mono-category methods offer diverse architectures, including encoder-decoder frameworks, multi-branch systems dedicated to specific modalities, and multi-scale designs. However, unifying these approaches requires balancing their strengths while ensuring flexibility and generalization across multiple fusion tasks. Specialized branched architectures, where each branch processes a specific input type or modality, must effectively integrate features in a way that is robust to differences in input characteristics. Conversely, common architectures that process all inputs through a shared pathway may struggle to capture modality-specific nuances. Benchmarking these architectural variations on standardized public datasets is essential to identify the most effective designs. Furthermore, the interplay between architectural components, such as attention mechanisms, fusion modules, and task-specific layers, must be carefully optimized to ensure compatibility across different image types and tasks. Given the increasing reliance on advanced techniques like transformers and generative adversarial networks (GANs), researchers must also address the computational trade-offs associated with this resource-intensive architectures.

**Dataset Availability and Standardization:** The shortage of publicly available datasets tailored to generalist multi-category fusion represents another significant challenge. While there are specialized datasets for multi-modal or multi-view fusion (e.g., 3MDAD for driver distraction analysis [159]), datasets that encompass multiple fusion categories within diverse application domains are exceedingly rare. The creation of such datasets would enable more comprehensive experimentation and benchmarking of generalist methods. An ideal dataset for generalist fusion would include a variety of scenarios that span different fusion tasks, such as combining multi-spectral, multi-exposure, and multi-view data. For instance, datasets capturing autonomous driving scenarios could integrate visible, infrared, and depth images from multiple viewpoints under various lighting and environmental conditions. Similarly, medical imaging datasets could include modality data (e.g., MRI, CT, PET) from different anatomical perspectives. Such datasets would not only facilitate the training and evaluation of generalist methods but also encourage cross-domain

applicability, an essential requirement for industrial and scientific adoption.

**Handling Diverse Data Modalities:** Generalist multi-category methods must demonstrate superior performance compared to mono-category approaches by effectively handling diverse data types simultaneously. This requirement demands that these methods excel in feature extraction and integration, regardless of the underlying data modalities. For example, a generalist method must be capable of merging features from visible and infrared images (multi-modal), images captured at different exposures (multi-exposure), and those acquired from distinct viewpoints (multi-view), all while maintaining high-quality outputs. The heterogeneity of data introduces additional challenges, such as managing variations in resolution, scale, alignment, and noise levels between input sources. Techniques such as deformable convolutions, spatial-channel attention mechanisms, and transformer-based global context modelling have shown promise in addressing these issues. However, integrating these techniques into a unified framework without compromising efficiency and scalability remains an open research problem.

**Computational Efficiency and Industrial Applicability:** The computational demands of generalist methods involve a significant barrier to their practical deployment, particularly in real-time or resource-constrained environments. Applications such as autonomous driving, robotics, and industrial inspection require low-latency and high-throughput processing to meet stringent performance requirements. The inference time of these models is often constrained by their size and the number of input images processed in a single batch. Optimizing memory usage and minimizing the computational footprint is therefore critical for enabling real-world applications. Techniques such as model pruning, quantization, and knowledge distillation can reduce the resource requirements of generalist methods without compromising their performance. Additionally, the use of lightweight architectures or hardware accelerators (e.g., GPUs, TPUs) could further enhance their applicability in industrial settings.

**Explainability and Interpretability:** Explainability is a critical consideration for generalist multi-category image fusion methods, particularly in high-stakes domains such as healthcare, autonomous systems, and surveillance. Understanding the contribution of each input modality to the final fused output is essential for building trust in these models and ensuring their reliability. For example, in medical imaging, clinicians must be able to discern how information from MRI and CT scans has influenced a diagnostic decision. Current approaches to explainability in image fusion include saliency maps, attention visualizations, and feature attribution techniques. However, these methods are often designed for specific architectures or mono-category tasks and may not generalize well to unified systems. Developing more robust and intuitive explainability frameworks tailored to multi-category fusion remains an important area of research. These frameworks should be capable of disentangling the contributions of individual modalities while providing actionable insights into the decision-making process.

Future research should focus on developing unified frameworks capable of effectively integrating information across both multi-view and multi-modal data. One promising direction is the design of adaptive architectures that can dynamically align and weight heterogeneous inputs based on their contextual relevance and quality. Furthermore, the exploration of cross-modal self-supervised learning could enhance representation learning in scenarios with limited annotated data. Another important avenue lies in improving spatial and semantic consistency across views and modalities, for instance through graph-based reasoning or attention mechanisms that preserve structural and contextual relationships. Benchmark datasets that jointly reflect multi-view and multi-modal complexities are also critically needed to facilitate fair evaluation and foster progress in this area. Lastly, addressing challenges such as domain gaps, real-time performance, and interpretability will be key to enabling the deployment of robust fusion models in real-world applications, including autonomous systems, remote sensing, and medical imaging.
In conclusion, while generalist multi-category image fusion methods offer significant potential for integrating diverse data modalities, their development faces several challenges. These include architectural complexity, limited datasets, handling heterogeneous data, computational efficiency, and the need for explainability. Overcoming these obstacles through innovative architectures, optimized computational strategies, and enhanced transparency will be essential for realizing the full potential of these methods. Continued research in these areas will enable more robust, scalable, and applicable solutions, fostering advancements across a wide range of industries and applications.

## 5. Conclusion

This survey presents a comprehensive and systematic examination of image fusion methodologies and offers several key contributions to the field. It first categorizes and analyzes single-category approaches, including multi-focus, multi-exposure, multi-modal, and multi-view fusion techniques. These approaches are assessed in terms of their underlying architectures, corresponding datasets, evaluation metrics, and application domains. Through a comparative analysis of state-of-the-art methods, this work identifies effective strategies within each category and provides practical guidance for researchers in selecting appropriate fusion techniques for specific tasks and contexts. A notable contribution of this study is the integration of multi-view fusion into the broader discourse on image fusion, an aspect often underrepresented in existing surveys. This inclusion helps bridge the methodological divide between classical fusion strategies and the spatial challenges intrinsic to multi-view settings. Furthermore, the survey highlights the connections across fusion categories, illustrating how advances in one area can benefit others. The review also evaluates emerging multi-category fusion methods, particularly those based on unified architectures capable of addressing multiple fusion tasks. These generalist approaches constitute a promising direction for future research, and this survey offers insights into their current strengths and limitations. In doing so, it supports the development of more versatile and adaptive image fusion frameworks. Nonetheless, several limitations in the current body of work should be acknowledged. Most notably, few architectures are explicitly designed for multi-category fusion. Additionally, the lack of domain-agnostic, task-independent evaluation metrics complicates the consistent assessment of model performance across categories. Moreover, the field remains fragmented, with limited methodological coherence or cross-pollination among multi-focus, multi-exposure, multi-modal, and multi-view research. This survey seeks to address these gaps and promote a more unified perspective on image fusion. Looking ahead, two key research directions are proposed. First, the creation of a large-scale, multi-category image fusion dataset would provide a critical resource for training and evaluating generalist models. Second, the development of novel architectures capable of effectively handling multiple fusion domains is essential to advancing a new state of the art. Together, these efforts would help establish a more cohesive and generalizable framework for image fusion, with broad applicability across diverse tasks and domains.

## CRediT authorship contribution statement

**Gwendal Bernardi:** Writing – review & editing, Writing – original draft, Conceptualization. **Godefroy Brisebarre:** Writing – review & editing. **Sébastien Roman:** Writing – review & editing. **Mohsen Ardabilian:** Writing – review & editing. **Emmanuel Dellandrea:** Writing – review & editing.

## Declaration of competing interest

The authors declare the following financial interests/personal relationships which may be considered as potential competing interests: Bernardi Gwendal reports financial support was provided by Tiama. If there are other authors, they declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

## Data availability

No data was used for the research described in the article.

## References

[1] E. Balit, A. Chadli, GMFNet: Gated multimodal fusion network for visible-thermal semantic segmentation, in: Proceedings 16th the European Conference on Computer Vision, 2020, pp. 1–4.

[2] V. Vs, J.M.J. Valanarasu, P. Oza, V.M. Patel, Image fusion transformer, in: 2022 IEEE International Conference on Image Processing (ICIP), 2022, pp. 3566–3570.

[3] X. Peng, X. Zhu, T. Wang, Y. Ma, Side: Center-based stereo 3d detector with structure-aware instance depth estimation, in: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, 2022, pp. 119–128.

[4] Y. Chen, S. Liu, X. Shen, J. Jia, Dsgn: Deep stereo geometry network for 3d object detection, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020, pp. 12536–12545.

[5] C. Shu, J. Deng, F. Yu, Y. Liu, 3Dppe: 3d point positional encoding for transformer-based multi-camera 3d object detection, in: Proceedings of the IEEE/CVF International Conference on Computer Vision, 2023, pp. 3580–3589.

[6] Z. Liu, K. Tsukada, K. Hanasaki, Y.-K. Ho, Y. Dai, Image fusion by using steerable pyramid, Pattern Recognit. Lett. 22 (9) (2001) 929–939.

[7] Q. Zhang, Y. Liu, R.S. Blum, J. Han, D. Tao, Sparse representation based multi-sensor image fusion for multi-focus and multi-modality images: A review, Inf. Fusion 40 (2018) 57–75.

[8] N. Paramanandham, K. Rajendiran, Multi sensor image fusion for surveillance applications using hybrid image fusion algorithm, Multimedia Tools Appl. 77 (2018) 12405–12436.

[9] L. Yang, B. Guo, W. Ni, Multifocus image fusion algorithm based on contourlet decomposition and region statistics, in: Fourth International Conference on Image and Graphics (ICIG 2007), IEEE, 2007, pp. 707–712.

[10] T. Baltrušaitis, C. Ahuja, L.-P. Morency, Multimodal machine learning: A survey and taxonomy, IEEE Trans. Pattern Anal. Mach. Intell. 41 (2) (2018) 423–443.

[11] S. Karim, G. Tong, J. Li, A. Qadir, U. Farooq, Y. Yu, Current advances and future perspectives of image fusion: A comprehensive review, Inf. Fusion 90 (2023) 185–217.

[12] S.-C. Huang, A. Pareek, S. Seyyedi, I. Banerjee, M.P. Lungren, Fusion of medical imaging and electronic health records using deep learning: A systematic review and implementation guidelines, NPJ Digit. Med. 3 (1) (2020) 136.

[13] K. Bayoudh, R. Knani, F. Hamdaoui, A. Mtibaa, A survey on deep multimodal learning for computer vision: Advances, trends, applications, and datasets, Vis. Comput. 38 (8) (2022) 2939–2970.

[14] S. Kalamkar, et al., Multimodal image fusion: A systematic review, Decis. Anal. J. (2023) 100327.

[15] B. Gwendal, S. David, B. Godefroy, G. Jean-François, A. Mohsen, D. Emmanuel, Image fusion survey: A novel taxonomy integrating transformer and recent approaches, in: S. Palaiahnakote, S. Schuckers, J.-M. Ogier, P. Bhattacharya, U. Pal, S. Bhattacharya (Eds.), Pattern Recognition. ICPR 2024 International Workshops and Challenges, Springer Nature Switzerland, Cham, 2025, pp. 167–180.

[16] H. Xu, J. Ma, J. Jiang, X. Guo, H. Ling, U2Fusion: A unified unsupervised image fusion network, IEEE Trans. Pattern Anal. Mach. Intell. 44 (1) (2020) 502–518.

[17] J. Song, Y. Zheng, M. Zakir Ullah, J. Wang, Y. Jiang, C. Xu, Z. Zou, G. Ding, Multiview multimodal network for breast cancer diagnosis in contrast-enhanced spectral mammography images, Int. J. Comput. Assist. Radiol. Surg. 16 (6) (2021) 979–988.

[18] H.-F. Yan, Y.-Q. Zhao, J.C.-W. Chan, S.G. Kong, N. El-Bendary, M. Reda, Hyperspectral and multispectral image fusion: When model-driven meet data-driven strategies, Inf. Fusion 116 (2025) 102803, http://dx.doi.org/10.1016/j.inffus.2024.102803, URL https://www.sciencedirect.com/science/article/pii/S1566253524005815.

[19] S. Kalamkar, G.M. A, Multimodal image fusion: A systematic review, Decis. Anal. J. 9 (2023) 100327, http://dx.doi.org/10.1016/j.dajour.2023.100327, URL https://www.sciencedirect.com/science/article/pii/S2772662223001674.

[20] F. Xu, J. Liu, Y. Song, H. Sun, X. Wang, Multi-exposure image fusion techniques: A comprehensive review, Remote. Sens. 14 (3) (2022) 771.

[21] M.A. Azam, K.B. Khan, S. Salahuddin, E. Rehman, S.A. Khan, M.A. Khan, S. Kadry, A.H. Gandomi, A review on multimodal medical image fusion: Comprehensive analysis of medical modalities, multimodal databases, fusion techniques and quality metrics, Comput. Biol. Med. 144 (2022) 105253, http://dx.doi.org/10.1016/j.compbiomed.2022.105253, URL https://www.sciencedirect.com/science/article/pii/S0010482522000452.

[22] H. Zhang, H. Xu, X. Tian, J. Jiang, J. Ma, Image fusion meets deep learning: A survey and perspective, Inf. Fusion 76 (2021) 323–336, http://dx.doi.org/10.1016/j.inffus.2021.06.008, URL https://www.sciencedirect.com/science/article/pii/S1566253521001342.
[23] H. Kaur, D. Koundal, V. Kadyan, Image fusion techniques: A survey, Arch. Comput. Methods Eng. 28 (7) (2021) 4425-4447, http://dx.doi.org/10.1007/s11831-021-09540-7, URL https://doi.org/10.1007/s11831-021-09540-7.

[24] Y. Liu, L. Wang, J. Cheng, C. Li, X. Chen, Multi-focus image fusion: A survey of the state of the art, Inf. Fusion 64 (2020) 71-91.

[25] B. Meher, S. Agrawal, R. Panda, A. Abraham, A survey on region based image fusion methods, Inf. Fusion 48 (2019) 119-132, http://dx.doi.org/10.1016/j.inffus.2018.07.010, URL https://www.sciencedirect.com/science/article/pii/S1566253517307583.

[26] W. Kong, Y. Lei, Multi-focus image fusion using biochemical ion exchange model, Appl. Soft Comput. 51 (2017) 314-327.

[27] Y. Yang, M. Yang, S. Huang, Y. Que, M. Ding, J. Sun, Multifocus image fusion based on extreme learning machine and human visual system, IEEE Access 5 (2017) 6989-7000.

[28] Z. Ji, X. Kang, K. Zhang, P. Duan, Q. Hao, A two-stage multi-focus image fusion framework robust to image mis-registration, IEEE Access 7 (2019) 123231-123243.

[29] A. Baradarani, Q.J. Wu, M. Ahmadi, P. Mendapara, Tunable halfband-pair wavelet filter banks and application to multifocus image fusion, Pattern Recognit. 45 (2) (2012) 657-671.

[30] S. Paul, I.S. Sevcenco, P. Agathoklis, Multi-exposure and multi-focus image fusion in gradient domain, J. Circuits Syst. Comput. 25 (10) (2016) 1650123.

[31] Z. Zhu, H. Yin, Y. Chai, Y. Li, G. Qi, A novel multi-modality image fusion method based on image decomposition and sparse representation, Inform. Sci. 432 (2018) 516-529.

[32] Y. Liu, X. Chen, H. Peng, Z. Wang, Multi-focus image fusion with a deep convolutional neural network, Inf. Fusion 36 (2017) 191-207.

[33] Y. Zhang, Y. Liu, P. Sun, H. Yan, X. Zhao, L. Zhang, IFCNN: A general image fusion framework based on convolutional neural network, Inf. Fusion 54 (2020) 99-118.

[34] J. Johnson, A. Alahi, L. Fei-Fei, Perceptual losses for real-time style transfer and super-resolution, in: Computer Vision-ECCV 2016: 14th European Conference, Amsterdam, the Netherlands, October 11-14, 2016, Proceedings, Part II 14, Springer, 2016, pp. 694-711.

[35] C. Ledig, L. Theis, F. Huszár, J. Caballero, A. Cunningham, A. Acosta, A. Aitken, A. Tejani, J. Totz, Z. Wang, et al., Photo-realistic single image super-resolution using a generative adversarial network, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017, pp. 4681-4690.

[36] R. Lai, Y. Li, J. Guan, A. Xiong, Multi-scale visual attention deep convolutional neural network for multi-focus image fusion, IEEE Access 7 (2019) 114385-114399.

[37] J. Li, X. Guo, G. Lu, B. Zhang, Y. Xu, F. Wu, D. Zhang, DRPL: Deep regression pair learning for multi-focus image fusion, IEEE Trans. Image Process. 29 (2020) 4816-4831.

[38] Z. Wang, A.C. Bovik, H.R. Sheikh, Structural similarity based image quality assessment, in: Digital Video Image Quality and Perceptual Coding, CRC Press, 2017, pp. 225-242.

[39] X. Guo, R. Nie, J. Cao, D. Zhou, L. Mei, K. He, FuseGAN: Learning to fuse multi-focus image via conditional generative adversarial network, IEEE Trans. Multimed. 21 (8) (2019) 1982-1996.

[40] P. Isola, J.-Y. Zhu, T. Zhou, A.A. Efros, Image-to-image translation with conditional adversarial networks, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017, pp. 1125-1134.

[41] X. Mao, Q. Li, H. Xie, R.Y. Lau, Z. Wang, S. Paul Smolley, Least squares generative adversarial networks, in: Proceedings of the IEEE International Conference on Computer Vision, 2017, pp. 2794-2802.

[42] H. Yan, X. Yu, Y. Zhang, S. Zhang, X. Zhao, L. Zhang, Single image depth estimation with normal guided scale invariant deep convolutional fields, IEEE Trans. Circuits Syst. Video Technol. 29 (1) (2017) 80-92.

[43] M. Everingham, S.A. Eslami, L. Van Gool, C.K. Williams, J. Winn, A. Zisserman, The pascal visual object classes challenge: A retrospective, Int. J. Comput. Vis. 111 (2015) 98-136.

[44] S. Xu, X. Wei, C. Zhang, J. Liu, J. Zhang, MFFW: A new dataset for multi-focus image fusion, 2020, arXiv preprint arXiv:2002.04780.

[45] S. Gabarda, G. Cristóbal, On the use of a joint spatial-frequency representation for the fusion of multi-focus images, Pattern Recognit. Lett. 26 (16) (2005) 2572-2578.

[46] R. Redondo, F. Šroubek, S. Fischer, G. Cristóbal, Multifocus image fusion using the log-Gabor transform and a multisize windows technique, Inf. Fusion 10 (2) (2009) 163-171.

[47] M. Liu, X. Wang, H. Zhang, Classification of nematode image stacks by an information fusion based multilinear approach, Pattern Recognit. Lett. 100 (2017) 22-28.

[48] M. Liu, X. Wang, H. Zhang, Taxonomy of multi-focal nematode image stacks by a CNN based image fusion approach, Comput. Methods Programs Biomed. 156 (2018) 209-215.

[49] S. Tello-Mijares, J. Bescós, Region-based multifocus image fusion for the precise acquisition of pap smear images, J. Biomed. Opt. 23 (5) (2018) 056005.

[50] Y. Liang, Y. Mao, Z. Tang, M. Yan, Y. Zhao, J. Liu, Efficient misalignment-robust multi-focus microscopical images fusion, Signal Process. 161 (2019) 111-123.

[51] X. Yan, S.Z. Gilani, H. Qin, A. Mian, Unsupervised deep multi-focus image fusion, 2018, arXiv preprint arXiv:1806.07272.

[52] H.T. Mustafa, J. Yang, M. Zare Moor, Multi-scale convolutional neural network for multi-focus image fusion, Image Vis. Comput. 85 (2019) 26-35.

[53] H. Jung, Y. Kim, H. Jang, N. Ha, K. Sohn, Unsupervised deep image fusion with structure tensor representations, IEEE Trans. Image Process. 29 (2020) 3845-3858.

[54] Y. Liu, D. Zhou, R. Nie, R. Hou, Z. Ding, Construction of high dynamic range image based on gradient information transformation, IET Image Process. 14 (7) (2020) 1327-1338.

[55] S. Raman, S. Chaudhuri, Bilateral filter based compositing for variable exposure photography, in: Eurographics (Short Papers), 2009, pp. 1-4.

[56] H.S.V. Kumar, S. Bhoosan, A Novel Approach for Detail-Enhanced Exposure Fusion Using Guided Filter, Hindawi Publishing Corporation the Scientific World.

[57] R. Shen, I. Cheng, J. Shi, A. Basu, Generalized random walks for fusion of multi-exposure images, IEEE Trans. Image Process. 20 (12) (2011) 3634-3646.

[58] J. Wang, W. Wang, G. Xu, H. Liu, End-to-end exposure fusion using convolutional neural network, IEICE Trans. Inf. Syst. 101 (2) (2018) 560-563.

[59] K. Ram Prabhakar, V. Sai Srikar, R. Venkatesh Babu, Deepfuse: A deep unsupervised approach for exposure fusion with extreme exposure image pairs, in: Proceedings of the IEEE International Conference on Computer Vision, 2017, pp. 4714-4722.

[60] K. Ma, K. Zeng, Z. Wang, Perceptual quality assessment for multi-exposure image fusion, IEEE Trans. Image Process. 24 (11) (2015) 3345-3356.

[61] K. Ma, Z. Duanmu, H. Zhu, Y. Fang, Z. Wang, Deep guided learning for fast multi-exposure image fusion, IEEE Trans. Image Process. 29 (2019) 2808-2819.

[62] H. Xu, J. Ma, X.-P. Zhang, MEF-GAN: Multi-exposure image fusion via generative adversarial networks, IEEE Trans. Image Process. 29 (2020) 7203-7216.

[63] Z. Yang, Y. Chen, Z. Le, Y. Ma, GANFuse: A novel multi-exposure image fusion method based on generative adversarial networks, Neural Comput. Appl. 33 (2021) 6133-6145.

[64] T. Nie, L. Huang, H. Liu, X. Li, Y. Zhao, H. Yuan, X. Song, B. He, Multi-exposure fusion of gray images under low illumination based on low-rank decomposition, Remote. Sens. 13 (2) (2021) 204.

[65] O. Martorell, C. Sbert, A. Buades, Ghosting-free DCT based multi-exposure image fusion, Signal Process., Image Commun. 78 (2019) 409-425.

[66] J.-W. Kim, J.-H. Ryu, J.-O. Kim, Deep gradual flash fusion for low-light enhancement, J. Vis. Commun. Image Represent. 72 (2020) 102903.

[67] L. Huang, Z. Li, C. Xu, B. Feng, Multi-exposure image fusion based on feature evaluation with adaptive factor, IET Image Process. 15 (13) (2021) 3211-3220.

[68] M. Gao, J. Wang, Y. Chen, C. Du, C. Chen, Y. Zeng, An improved multi-exposure image fusion method for intelligent transportation system, Electronics 10 (4) (2021) 383.

[69] J. Cai, S. Gu, L. Zhang, Learning a deep single image contrast enhancer from multi-exposure images, IEEE Trans. Image Process. 27 (4) (2018) 2049-2062.

[70] Y. Zhang, T. Liu, M. Singh, E. Çetintaş, Y. Luo, Y. Rivenson, K.V. Larin, A. Ozcan, Neural network-based image reconstruction in swept-source optical coherence tomography using undersampled spectral data, Light.: Sci. Appl. 10 (1) (2021) 155.

[71] X. Li, G. Zhang, H. Qiao, F. Bao, Y. Deng, J. Wu, Y. He, J. Yun, X. Lin, H. Xie, et al., Unsupervised content-preserving transformation for optical microscopy, Light.: Sci. Appl. 10 (1) (2021) 44.

[72] Q. Xie, M. Zhou, Q. Zhao, D. Meng, W. Zuo, Z. Xu, Multispectral and hyperspectral image fusion by MS/HS fusion net, Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR) (2019).

[73] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, Y. Bengio, Generative adversarial networks, Commun. ACM 63 (11) (2020) 139-144.

[74] H. Li, X.-J. Wu, DenseFuse: A fusion approach to infrared and visible images, IEEE Trans. Image Process. 28 (5) (2018) 2614-2623.

[75] V. Vielzeuf, A. Lechervy, S. Pateux, F. Jurie, Centralnet: A multilayer approach for multimodal fusion, Proc. Eur. Conf. Comput. Vis. (ECCV) Work. (2018) 0-1.

[76] Z. Zhou, M.M. Rahman Siddiquee, N. Tajbakhsh, J. Liang, Unet++: A nested u-net architecture for medical image segmentation, in: Deep Learning in Medical Image Analysis and Multimodal Learning for Clinical Decision Support: 4th International Workshop, DLMIA 2018, and 8th International Workshop, ML-CDS 2018, Held in Conjunction with MICCAI 2018, Granada, Spain, September 20, 2018, Proceedings 4, Springer, 2018, pp. 3-11.

[77] H. Li, X.-J. Wu, J. Kittler, RFN-nest: An end-to-end residual fusion network for infrared and visible images, Inf. Fusion 73 (2021) 72-86.

[78] H. Li, X.-J. Wu, T. Durrani, NestFuse: An infrared and visible image fusion architecture based on nest connection and spatial/channel attention models, IEEE Trans. Instrum. Meas. 69 (12) (2020) 9645-9656.

[79] Q. Liu, J. Pi, P. Gao, D. Yuan, STFNet: Self-supervised transformer for infrared and visible image fusion, IEEE Trans. Emerg. Top. Comput. Intell. (2024).

[80] J. Dai, H. Qi, Y. Xiong, Y. Li, G. Zhang, H. Hu, Y. Wei, Deformable convolutional networks, in: Proceedings of the IEEE International Conference on Computer Vision, 2017, pp. 764-773.
[81] Z. Wang, E.P. Simoncelli, A.C. Bovik, Multiscale structural similarity for image quality assessment, in: The Thrifty-Seventh Asilomar Conference on Signals, Systems & Computers, 2003, Vol. 2, IEEE, 2003, pp. 1398-1402.

[82] Z. Liu, K. Geng, X. Cheng, K. Shen, A. Li, S. Cheng, CMAFusion: Cross modal attention based end-to-end infrared and visible image fusion network, in: 2023 7th CAA International Conference on Vehicular Control and Intelligence, CVCI, IEEE, 2023, pp. 1-6.

[83] L. Xu, Z. Wang, B. Wu, S. Lui, Mdan: Multi-level dependent attention network for visual emotion analysis, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 9479-9488.

[84] X. Xu, Y. Shen, S. Han, Dense-FG: A fusion GAN model by using densely connected blocks to fuse infrared and visible images, Appl. Sci. 13 (8) (2023) 4684.

[85] J. Ma, W. Yu, P. Liang, C. Li, J. Jiang, FusionGAN: A generative adversarial network for infrared and visible image fusion, Inf. Fusion 48 (2019) 11-26.

[86] J. Ma, H. Zhang, Z. Shao, P. Liang, H. Xu, GANMcC: A generative adversarial network with multiclassification constraints for infrared and visible image fusion, IEEE Trans. Instrum. Meas. 70 (2020) 1-14.

[87] J. Ma, H. Xu, J. Jiang, X. Mei, X.-P. Zhang, DDcGAN: A dual-discriminator conditional generative adversarial network for multi-resolution image fusion, IEEE Trans. Image Process. 29 (2020) 4980-4995.

[88] T. Nguyen, T. Le, H. Vu, D. Phung, Dual discriminator generative adversarial nets, Adv. Neural Inf. Process. Syst. 30 (2017).

[89] M. Mirza, S. Osindero, Conditional generative adversarial nets, 2014, arXiv preprint arXiv:1411.1784.

[90] H. Zhang, J. Yuan, X. Tian, J. Ma, GAN-FM: Infrared and visible image fusion using GAN with full-scale skip connection and dual Markovian discriminators, IEEE Trans. Comput. Neural Imaging 7 (2021) 1134-1147.

[91] M. Kovačič, Markovian neural networks, Biol. Cybernet. 64 (4) (1991) 337-342.

[92] Z. Zhao, H. Bai, Y. Zhu, J. Zhang, S. Xu, Y. Zhang, K. Zhang, D. Meng, R. Timofte, L. Van Gool, DDFM: Denoising diffusion model for multi-modality image fusion, in: Proceedings of the IEEE/CVF International Conference on Computer Vision, 2023, pp. 8082-8093.

[93] J. Ho, A. Jain, P. Abbeel, Denoising diffusion probabilistic models, Adv. Neural Inf. Process. Syst. 33 (2020) 6840-6851.

[94] J. Kim, J. Koh, Y. Kim, J. Choi, Y. Hwang, J.W. Choi, Robust deep multi-modal learning based on gated information fusion network, Asian Conf. Comput. Vis. (2018) 90-106.

[95] A. Graves, A. Graves, Long short-term memory, Supervised Seq. Label. Recurr. Neural Netw. (2012) 37-45.

[96] Q. Ha, K. Watanabe, T. Karasawa, Y. Ushiku, T. Harada, MFNet: Towards real-time semantic segmentation for autonomous vehicles with multi-spectral scenes, in: 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), IEEE, 2017, pp. 5108-5115.

[97] Y. Sun, W. Zuo, M. Liu, RTFNet: RGB-thermal fusion network for semantic segmentation of urban scenes, IEEE Robot. Autom. Lett. 4 (3) (2019) 2576-2583.

[98] K. He, X. Zhang, S. Ren, J. Sun, Deep residual learning for image recognition, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2016, pp. 770-778.

[99] C. Chen, J. Qi, X. Liu, K. Bin, R. Fu, X. Hu, P. Zhong, Weakly misalignment-free adaptive feature alignment for UAVs-based multimodal object detection, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024, pp. 26836-26845.

[100] J. Liu, X. Fan, Z. Huang, G. Wu, R. Liu, W. Zhong, Z. Luo, Target-aware dual adversarial learning and a multi-scenario multi-modality benchmark to fuse infrared and visible for object detection, IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (2022) 5802-5811.

[101] G. Jocher, A. Chaurasia, A. Stoken, J. Borovec, Y. Kwon, K. Michael, J. Fang, Z. Yifu, C. Wong, D. Montes, et al., Ultralytics/yolov5: V7. 0-yolov5 sota realtime instance segmentation, Zenodo (2022).

[102] Y. Wang, J. Pu, D. Miao, L. Zhang, L. Zhang, X. Du, SCGRFuse: An infrared and visible image fusion network based on spatial/channel attention mechanism and gradient aggregation residual dense blocks, Eng. Appl. Artif. Intell. 132 (2024) 107898.

[103] J.M. Linfeng Tang, Image fusion in the loop of high-level vision tasks: A semantic-aware real-time infrared and visible image fusion network, Inf. Fusion 82 (2022) 28-42.

[104] J. Liu, R. Lin, G. Wu, R. Liu, Z. Luo, X. Fan, Coconet: Coupled contrastive learning network with multi-level feature ensemble for multi-modality image fusion, Int. J. Comput. Vis. 132 (5) (2024) 1748-1775.

[105] Z. Wang, A.C. Bovik, H.R. Sheikh, E.P. Simoncelli, Image quality assessment: From error visibility to structural similarity, IEEE Trans. Image Process. 13 (4) (2004) 600-612.

[106] V. Aslantas, E. Bendes, A new image quality metric for image fusion: The sum of the correlations of differences, Aeu-Int. J. Electron. Commun. 69 (12) (2015) 1890-1896.

[107] A.M. Eskicioglu, P.S. Fisher, Image quality measures and their performance, IEEE Trans. Commun. 43 (12) (1995) 2959-2965.

[108] Y. Han, Y. Cai, Y. Cao, X. Xu, A new image fusion performance metric based on visual information fidelity, Inf. Fusion 14 (2) (2013) 127-135.

[109] M. Haghighat, M.A. Razian, Fast-FMI: Non-reference image fusion metric, in: 2014 IEEE 8th International Conference on Application of Information and Communication Technologies, AICT, IEEE, 2014, pp. 1-3.

[110] A. Toet, TNO image fusion dataset, 2014, http://dx.doi.org/10.6084/m9.fishshare.1008029.v2.

[111] L. Tang, J. Yuan, H. Zhang, X. Jiang, J. Ma, Piafusion: A progressive infrared and visible image fusion network based on illumination aware, Inf. Fusion (2022).

[112] X. Jia, C. Zhu, M. Li, W. Tang, W. Zhou, LLVIP: A visible-infrared paired dataset for low-light vision, in: Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021, pp. 3496-3504.

[113] S. Hwang, J. Park, N. Kim, Y. Choi, I.S. Kweon, Multispectral pedestrian detection: Benchmark dataset and baselines, Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR) (2015).

[114] M.D. Keith A. Johnson, J.A. Becker, MRI PET dataset, 2022.

[115] X. Wu, K. Fu, Z. Liu, W. Chen, A brief survey of feature based image matching, in: 2022 IEEE 17th Conference on Industrial Electronics and Applications, ICIEA, IEEE, 2022, pp. 1634-1639.

[116] Q. Huang, X. Guo, Y. Wang, H. Sun, L. Yang, A survey of feature matching methods, IET Image Process. (2024).

[117] S. Xu, S. Chen, R. Xu, C. Wang, P. Lu, L. Guo, Local feature matching using deep learning: A survey, 2024, arXiv preprint arXiv:2401.17592.

[118] M. Contreras, A. Jain, N.P. Bhatt, A. Banerjee, E. Hashemi, A survey on 3D object detection in real time for autonomous driving, Front. Robot. AI 11 (2024) 1212070.

[119] A. Rabby, C. Zhang, Beyondpixels: A comprehensive review of the evolution of neural radiance fields, 2023, arXiv preprint arXiv:2306.03000.

[120] R. Qian, D. Garg, Y. Wang, Y. You, S. Belongie, B. Hariharan, M. Campbell, K.Q. Weinberger, W.-L. Chao, End-to-end pseudo-lidar for image-based 3d object detection, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020, pp. 5881-5890.

[121] J. Edstedt, Q. Sun, G. Bökman, M. Wadenbäck, M. Felsberg, RoMa: Robust dense feature matching, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024, pp. 19790-19800.

[122] M. Oquab, T. Darcet, T. Moutakanni, H. Vo, M. Szafraniec, V. Khalidov, P. Fernandez, D. Haziza, F. Massa, A. El-Nouby, et al., Dinov2: Learning robust visual features without supervision, 2023, arXiv preprint arXiv:2304.07193.

[123] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, et al., An image is worth 16x16 words: Transformers for image recognition at scale, 2020, arXiv preprint arXiv:2010.11929.

[124] C. Li, J. Ku, S.L. Waslander, Confidence guided stereo 3D object detection with split depth estimation, in: 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems, IROS, IEEE, 2020, pp. 5776-5783.

[125] Y. Wang, W.-L. Chao, D. Garg, B. Hariharan, M. Campbell, K.Q. Weinberger, Pseudo-lidar from visual depth estimation: Bridging the gap in 3d object detection for autonomous driving, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019, pp. 8445-8453.

[126] X. Chen, H. Ma, J. Wan, B. Li, T. Xia, Multi-view 3d object detection network for autonomous driving, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017, pp. 1907-1915.

[127] M. Liang, B. Yang, S. Wang, R. Urtasun, Deep continuous fusion for multi-sensor 3d object detection, in: Proceedings of the European Conference on Computer Vision, ECCV, 2018, pp. 641-656.

[128] W. Peng, H. Pan, H. Liu, Y. Sun, Ida-3d: Instance-depth-aware 3d object detection from stereo vision for autonomous driving, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020, pp. 13015-13024.

[129] Z. Liu, Y. Lin, Y. Cao, H. Hu, Y. Wei, Z. Zhang, S. Lin, B. Guo, Swin transformer: Hierarchical vision transformer using shifted windows, in: Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021, pp. 10012-10022.

[130] Y. Li, Z. Ge, G. Yu, J. Yang, Z. Wang, Y. Shi, J. Sun, Z. Li, Bevdepth: Acquisition of reliable depth for multi-view 3d object detection, in: Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 37, (2) 2023, pp. 1477-1485.

[131] T. Wang, X. Zhu, J. Pang, D. Lin, Fcos3d: Fully convolutional one-stage monocular 3d object detection, in: Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021, pp. 913-922.

[132] T. Wang, Z. Xinge, J. Pang, D. Lin, Probabilistic and geometric depth: Detecting objects in perspective, in: Conference on Robot Learning, PMLR, 2022, pp. 1475-1485.

[133] X. Li, W. Wang, L. Wu, S. Chen, X. Hu, J. Li, J. Tang, J. Yang, Generalized focal loss: Learning qualified and distributed bounding boxes for dense object detection, Adv. Neural Inf. Process. Syst. 33 (2020) 21002-21012.

[134] H. Wang, W. Hutchcroft, Y. Li, Z. Wan, I. Boyadzhiiev, Y. Tian, S.B. Kang, Psmnet: Position-aware stereo merging network for room layout estimation, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 8616-8625.