###### Abstract  
In this paper, a detail-injection method based on a coupled convolutional neural network (CNN) is proposed for hyperspectral (HS) and multispectral (MS) image fusion with the goal of enhancing the spatial resolution of HS images. Owing to the excellent performance in spectral fidelity of the detail-injection model and the image spatial-spectral feature exploration ability of CNN, the proposed method utilizes a couple of CNN networks as the feature extraction method and learns details from the HS and MS images individually. By appending an additional convolutional layer, both the extracted features of two images are concatenated to predict the missing details of the anticipated HS image. Experiments on simulated and real HS and MS data show that compared with some state-of-the-art HS and MS image fusion methods, our proposed method achieves better fusion results, provides excellent spectrum preservation ability, and is easy to implement. convolutional neural network; hyper-sharpening; hyperspectral; image fusion; multispectral +
Footnote †: journal: Medical  
HS sharpening and super-resolution problems. Sparse representation [11; 12] is also used to redefine the fusion problem for HS and MS images and shows excellent enhancement, especially in spatial aspects. The matrix factorization-based hyper-sharpening techniques draw worldwide interest, due to the report of the coupled nonnegative matrix factorization (CNMF) algorithm [13], which alternately unmixes HS and MS images into endmember and abundance matrices based on a linear spectral mixture model. Inspired by CNMF, the dictionary-pair learning method (DPLM) [14] and many other matrix factorization methods are proposed successively. For instance, two approaches are proposed in [15], i.e., gradient-based joint-criterion nonnegative matrix factorization (JNMF) and multiplicative JNMF method. [PERSON]. [16] proposes the spectral modulation hyper-sharpening methods, aiming at minimizing the spectral distortion of real MS and HS images acquired by different sensors or platforms. Considering the spectral variability of remote sensing scenes acquired at different times, [17] combines the unmixing-based formulation with an explicit parametric model to implement HS and MS image fusion. Compared with those conventional methods, matrix factorization-based methods show excellent spectral fidelity and capability of anti-noise. In addition, tensor-based approaches have also been explored to further improve the performance over matrix-based techniques, since they generally regard HS image as a 3D tensor rather than multiple 2D matrices and jointly exploit the spatial-spectral structure information. Refs. [18; 19; 20; 21; 22] have made comprehensive analyses and comparisons about those classical hyper-sharpening algorithms. In recent years, with the development and increased application of deep learning techniques, convolutional neural networks (CNNs) [23] were widely applied to those image-related tasks owing to their outstanding adaptability and practicability in exploring and extracting local spatial structure characteristics. Accordingly, several CNN models have been proposed to deal with tasks related to image fusion or super-resolution: Refs. [24; 25; 26; 27] implement the fusion work by utilizing a 3-D convolution neural network, with the dimension of HS image reduced beforehand. In [28], a deep HS sharpening method, abbreviated to DHSIS, is presented to learn the image priors via deep CNN-based residual learning. Ref. [29] proposes a two-branches CNN fusion method (abbreviated to TCNNF for convenience in this paper), which explores the features from the spectrum of each pixel in low-resolution HS images and its corresponding spatial neighborhood in MS images by 1-D and 2-D CNN branches, respectively. The extracted features are subsequently concatenated and fed to fully connected (FC) layers. In this way, the spatial and spectral information of HS and MS images could be fully fused.