HighRes-net: Multi-Frame Super-Resolution by Recursive Fusion

1 INTRODUCTION . Multiple low-resolution images collectively contain more information than any individual lowresolution image , due to minor geometric displacements , e.g . shifts , rotations , atmospheric turbulence , and instrument noise . Multi-Frame Super-Resolution ( MFSR ) ( Tsai , 1984 ) aims to reconstruct hidden high-resolution details from multiple low-resolution views of the same scene . Single Image Super-Resolution ( SISR ) , as a special case of MFSR , has attracted much attention in the computer vision , machine learning and deep learning communities in the last 5 years , with neural networks learning complex image priors to upsample and interpolate images ( Xu et al. , 2014 ; Srivastava et al. , 2015 ; He et al. , 2016 ) . However , in the meantime not much work has explored the learning of representations for the more general problem of MFSR to address the additional challenges of co-registration and fusion of multiple low-resolution images . This paper explores how Multi-Frame Super-Resolution ( MFSR ) can benefit from recent advances in learning representations with neural networks . To the best of our knowledge , this work is the first to introduce a deep-learning approach that solves the co-registration , fusion and registration-at-theloss problems in an end-to-end learning framework . Prompting this line of research is the increasing drive towards planetary-scale Earth observation to monitor the environment and human rights violations . Such observation can be used to inform policy , achieve accountability and direct on-the-ground action , e.g . within the framework of the Sustainable Development Goals ( Jensen & Campbell , 2019 ) . Nomenclature Registration is the problem of estimating the relative geometric differences between two images ( e.g . due to shifts , rotations , deformations ) . Fusion , in the MFSR context , is the problem of mapping multiple low-res representations into a single representation . By coregistration , we mean the problem of registering all low-resolution views to improve their fusion . By registration-at-the-loss , we mean the problem of registering the super-resolved reconstruction to the high-resolution ground-truth prior to computing the loss . This gives rise to the notion of a registered loss . Co-registration of multiple images is required for longitudinal studies of land change and environmental degradation . The fusion of multiple images is key to exploiting cheap , high-revisit-frequency satellite imagery , but of low-resolution , moving away from the analysis of infrequent and expensive high-resolution images . Finally , beyond fusion itself , super-resolved generation is required throughout the technical stack : both for labeling , but also for human oversight ( Drexler , 2019 ) demanded by legal context ( Harris et al. , 2018 ) . Summary of contributions • HighRes-net : We propose a deep architecture that learns to fuse an arbitrary number of lowresolution frames with implicit co-registration through a reference-frame channel . • ShiftNet : Inspired by HomographyNet ( DeTone et al. , 2016 ) , we define a model that learns to register and align the super-resolved output of HighRes-net , using ground-truth high-resolution frames as supervision . This registration-at-the-loss mechanism enables more accurate feedback from the loss function into the fusion model , when comparing a super-resolved output to a ground truth high resolution image . Otherwise , a MFSR model would naturally yield blurry outputs to compensate for the lack of registration , to correct for sub-pixel shifts and account for misalignments in the loss . • By combining the two components above , we contribute the first architecture to learn fusion and registration end-to-end . • We test and compare our approach to several baselines on real-world imagery from the PROBA-V satellite of ESA . Our performance has topped the Kelvins competition on MFSR , organized by the Advanced Concepts Team of ESA ( Märtens et al. , 2019 ) ( see section 5 ) . The rest of the paper is divided as follows : in Section 2 , we discuss related work on SISR and MFSR ; Section 3 outlines HighRes-net and section 4 presents ShiftNet , a differentiable registration component that drives our registered loss mechanism during end-to-end training . We present our results in section 5 , and in Section 6 we discuss some opportunities for and limitations and risks of super-resolution . 2 BACKGROUND . 2.1 MULTI-FRAME SUPER-RESOLUTION . How much detail can we resolve in the digital sample of some natural phenomenon ? Nyquist ( 1928 ) observed that it depends on the instrument ’ s sampling rate and the oscillation frequency of the underlying natural signal . Shannon ( 1949 ) built a sampling theory that explained Nyquist ’ s observations when the sampling rate is constant ( uniform sampling ) and determined the conditions of aliasing in a sample . Figure 2 illustrates this phenomenon . Under review as a conference paper at ICLR 2020 0.0 0.2 0.4 0.6 0.8 1.0t ( sec ) chirp signal sin ( 2πω ( t ) t ) 0.0 0.2 0.4 0.6 0.8 1.0t ( sec ) 0 250 ( Hz ) frequency ω ( t ) 128 t sufficient ( Nyquist ) sampling rate 0.0 0.2 0.4 0.6 0.8 1.0t ( sec ) high-resolution ( HR ) sampling frequency 1 kHzfHR −400 −200 0 200 400 frequency ( Hz ) DFT magnitudeF fHR 0.0 0.2 0.4 0.6 0.8 1.0t ( sec ) chirp signal sin ( ω ( t ) t ) 0.0 0.2 0.4 0.6 0.8 1.0t ( sec ) 0 250 ( Hz ) frequency ω ( t ) /2π128 t sufficient ( Nyquist ) sampling rate 0.0 0.2 0.4 0.6 0.8 1.0t ( sec ) high-resolution ( HR ) sampling frequency 1 kHzfHR −400 −200 0 200 400 frequency ( Hz ) DFT magnitudeF fHR 0.0 0.2 0.4 0.6 0.8 1.0t ( sec ) chirp signal sin ( ω ( t ) t ) 0.0 0.2 0.4 0.6 0.8 1.0t ( sec ) 0 250 ( Hz ) frequency ω ( t ) /2π 128 t sufficient ( Nyquist ) sampling rate fLR sampling rate 0.0 0.2 0.4 0.6 0.8 1.0t ( sec ) ALIASINGlow-resolution ( LR ) sampling frequency 250 HzfLR −100 −50 0 50 100 frequency ( Hz ) DFT magnitudeF fLRFigure 2 : Top : A chirp harmonic oscillator sin ( 2πω ( t ) t ) , with instantaneous frequency ω ( t ) . Left : The shape of the high-resolution sample resembles the underlying chirp signal . Right : Close to t = 1 , the apparent frequency of the low-resolution sample does not match that of the chirp . This is an example of aliasing ( shown with red at its most extreme ) , and it happens when the sampling rate falls below the Nyquist rate , sN = 2 · sB , where sB is the highest non-zero frequency of the signal . Sampling at high-resolution ( left ) maintains the frequency of the chirp signal ( top ) . Sampling at a lower resolution ( right ) , this apparent chirped frequency is lost due to aliasing , which means that the lower-resolution sample has a fundamentally smaller capacity for resolving the information of the natural signal , and a higher sampling rate can resolve more information . Shannon ’ s sampling theory has since been generalized for multiple interleaved sampling frames ( Papoulis , 1977 ; Marks , 2012 ) . One result of the generalized sampling theory is that we can go beyond the Nyquist limit of any individual uniform sample by interleaving several uniform samples taken concurrently . When an image is down-sampled to a lower resolution , its high-frequency details are lost permanently and can not be recovered from any image in isolation . However , by combining multiple low-resolution images , it becomes possible to recover the original scene at a higher resolution . Moreover , different low-resolution samples may be sampled at different phases , such that the same high-resolution frequency information will be packed with a phase shift . As a consequence , when multiple low-resolution samples are available , the fundamental challenge of MFSR is de-aliasing , i.e . disentangling the high-frequency components ( Tsai , 1984 ) . The first work on MSFR ( Tsai , 1984 ) considered the reconstruction of a high-resolution image as a fusion of co-registered low-resolution images in the Fourier domain . With proper registration and fusion ( Irani & Peleg , 1991 ; Fitzpatrick et al. , 2000 ; Capel & Zisserman , 2001 ) , a composite super-resolved image can reveal some of the original high-frequency detail that would not have been accessible from single low-resolution image . In this work , we introduce HighRes-Net , which aims to provide an end-to-end deep learning framework for MFSR settings . Relation to Video and Stereo Super-Resolution While there are obvious similarities to Video SR ( Tao et al. , 2017 ; Sajjadi et al. , 2018 ; Yan et al. , 2019 ; Wang et al. , 2019b ) and Stereo SR ( Wang et al. , 2019d ) ( Wang et al. , 2019a ) , the setting of this work differs in several ways : HighResnet learns to super-resolve sets and not sequences of low-res views . Video SR relies on motion estimation from a sequence of observations . Also , prediction at time t = T relies on predictions at t < T ( autoregressive approach ) . Whereas in our case , we predict a single image from an unordered set of low-res inputs . Also , the low-res views are multi-temporal ( taken at different times ) . Video SR methods assume that the input is a temporal sequence of frames . Motion or optical flow can be estimated to super-resolve the sequences of frames . In this work , we do not assume low-res inputs to be ordered in time . Our training input is a set of low-res views with unknown timestamps and our target output is a single image — not another sequence . 2.2 A PROBABILISTIC APPROACH . In addition to aliasing , MFSR deals with random processes like noise , blur , geometric distortions – all contributing to random low-resolution images . Traditionally , MFSR methods assume a-priori knowledge of the data generating motion model , blur kernel , noise level and degradation process ; see for example , Pickup et al . ( 2006 ) . Given multiple low-resolution images , the challenge of MFSR is to reconstruct a plausible image of higher-resolution that could have generated the observed lowresolution images . Optimization methods aim to improve an initial guess by minimizing an error between simulated and observed low-resolution images . These methods traditionally model the additive noise and prior knowledge about natural images explicitly , to constrain the parameter search space and derive objective functions , using e.g . Total Variation ( Chan & Wong , 1998 ; Farsiu et al. , 2004 ) , Tikhonov regularization ( Nguyen et al. , 2001 ) or Huber potential ( Pickup et al. , 2006 ) to define appropriate constraints on images . In some situations , the image degradation process is complex or not available , motivating the development of nonparametric strategies . Patch-based methods learn to form high-resolution images directly from low-resolution patches , e.g . with k-nearest neighbor search ( Freeman et al. , 2002 ; Chang et al. , 2004 ) , sparse coding and sparse dictionary methods ( Yang et al. , 2010 ; Zeyde et al. , 2010 ; Kim & Kwon , 2010 ) ) . The latter represents images in an over-complete basis and allows for sharing a prior across multiple sites . In this work , we are particularly interested in super-resolving satellite imagery . Much of the recent work in Super-Resolution has focused on SISR for natural images . For instance , Dong et al . ( 2014 ) showed that training a CNN for super-resolution is equivalent to sparse coding and dictionary based approaches . Kim et al . ( 2016 ) proposed an approach to SISR using recursion to increase the receptive field of a model while maintaining capacity by sharing weights . Many more networks and learning strategies have recently been introduced for SISR and image deblurring . Benchmarks for SISR ( Timofte et al. , 2018 ) , differ mainly in their upscaling method , network design , learning strategies , etc . We refer the reader to ( Wang et al. , 2019d ) for a more comprehensive review . Few deep-learning approaches have considered the more general MFSR setting and attempted to address it in an end-to-end learning framework . Reecently , Kawulok et al . ( 2019 ) proposed a shiftand-add method and suggested “ including image registration ” in the learning process as future work . In the following sections , we describe our approach to solving both aspects of the registration problem – co-registration and registration-at-the-loss – in a memory-efficient manner .

Summary: This paper proposes an end-to-end multi-frame super-resolution algorithm, that relies on a pair-wise co-registrations and fusing blocks (convolutional residual blocks), embedded in a encoder-decoder network 'HighRes-net' that estimates the super-resolution image. Because the ground truth SR image is typically misaligned with the estimation SR image, the authors proposed to learn the shift with a neural network 'ShiftNet' in a cooperative setting with HighRes-net. The experiments were performed on the ESA challenge on satellite images, showing good results.