HighRes-net: Multi-Frame Super-Resolution by Recursive Fusion

1 INTRODUCTION . Multiple low-resolution images collectively contain more information than any individual lowresolution image , due to minor geometric displacements , e.g . shifts , rotations , atmospheric turbulence , and instrument noise . Multi-Frame Super-Resolution ( MFSR ) ( Tsai , 1984 ) aims to reconstruct hidden high-resolution details from multiple low-resolution views of the same scene . Single Image Super-Resolution ( SISR ) , as a special case of MFSR , has attracted much attention in the computer vision , machine learning and deep learning communities in the last 5 years , with neural networks learning complex image priors to upsample and interpolate images ( Xu et al. , 2014 ; Srivastava et al. , 2015 ; He et al. , 2016 ) . However , in the meantime not much work has explored the learning of representations for the more general problem of MFSR to address the additional challenges of co-registration and fusion of multiple low-resolution images . This paper explores how Multi-Frame Super-Resolution ( MFSR ) can benefit from recent advances in learning representations with neural networks . To the best of our knowledge , this work is the first to introduce a deep-learning approach that solves the co-registration , fusion and registration-at-theloss problems in an end-to-end learning framework . Prompting this line of research is the increasing drive towards planetary-scale Earth observation to monitor the environment and human rights violations . Such observation can be used to inform policy , achieve accountability and direct on-the-ground action , e.g . within the framework of the Sustainable Development Goals ( Jensen & Campbell , 2019 ) . Nomenclature Registration is the problem of estimating the relative geometric differences between two images ( e.g . due to shifts , rotations , deformations ) . Fusion , in the MFSR context , is the problem of mapping multiple low-res representations into a single representation . By coregistration , we mean the problem of registering all low-resolution views to improve their fusion . By registration-at-the-loss , we mean the problem of registering the super-resolved reconstruction to the high-resolution ground-truth prior to computing the loss . This gives rise to the notion of a registered loss . Co-registration of multiple images is required for longitudinal studies of land change and environmental degradation . The fusion of multiple images is key to exploiting cheap , high-revisit-frequency satellite imagery , but of low-resolution , moving away from the analysis of infrequent and expensive high-resolution images . Finally , beyond fusion itself , super-resolved generation is required throughout the technical stack : both for labeling , but also for human oversight ( Drexler , 2019 ) demanded by legal context ( Harris et al. , 2018 ) . Summary of contributions • HighRes-net : We propose a deep architecture that learns to fuse an arbitrary number of lowresolution frames with implicit co-registration through a reference-frame channel . • ShiftNet : Inspired by HomographyNet ( DeTone et al. , 2016 ) , we define a model that learns to register and align the super-resolved output of HighRes-net , using ground-truth high-resolution frames as supervision . This registration-at-the-loss mechanism enables more accurate feedback from the loss function into the fusion model , when comparing a super-resolved output to a ground truth high resolution image . Otherwise , a MFSR model would naturally yield blurry outputs to compensate for the lack of registration , to correct for sub-pixel shifts and account for misalignments in the loss . • By combining the two components above , we contribute the first architecture to learn fusion and registration end-to-end . • We test and compare our approach to several baselines on real-world imagery from the PROBA-V satellite of ESA . Our performance has topped the Kelvins competition on MFSR , organized by the Advanced Concepts Team of ESA ( Märtens et al. , 2019 ) ( see section 5 ) . The rest of the paper is divided as follows : in Section 2 , we discuss related work on SISR and MFSR ; Section 3 outlines HighRes-net and section 4 presents ShiftNet , a differentiable registration component that drives our registered loss mechanism during end-to-end training . We present our results in section 5 , and in Section 6 we discuss some opportunities for and limitations and risks of super-resolution . 2 BACKGROUND . 2.1 MULTI-FRAME SUPER-RESOLUTION . How much detail can we resolve in the digital sample of some natural phenomenon ? Nyquist ( 1928 ) observed that it depends on the instrument ’ s sampling rate and the oscillation frequency of the underlying natural signal . Shannon ( 1949 ) built a sampling theory that explained Nyquist ’ s observations when the sampling rate is constant ( uniform sampling ) and determined the conditions of aliasing in a sample . Figure 2 illustrates this phenomenon . Under review as a conference paper at ICLR 2020 0.0 0.2 0.4 0.6 0.8 1.0t ( sec ) chirp signal sin ( 2πω ( t ) t ) 0.0 0.2 0.4 0.6 0.8 1.0t ( sec ) 0 250 ( Hz ) frequency ω ( t ) 128 t sufficient ( Nyquist ) sampling rate 0.0 0.2 0.4 0.6 0.8 1.0t ( sec ) high-resolution ( HR ) sampling frequency 1 kHzfHR −400 −200 0 200 400 frequency ( Hz ) DFT magnitudeF fHR 0.0 0.2 0.4 0.6 0.8 1.0t ( sec ) chirp signal sin ( ω ( t ) t ) 0.0 0.2 0.4 0.6 0.8 1.0t ( sec ) 0 250 ( Hz ) frequency ω ( t ) /2π128 t sufficient ( Nyquist ) sampling rate 0.0 0.2 0.4 0.6 0.8 1.0t ( sec ) high-resolution ( HR ) sampling frequency 1 kHzfHR −400 −200 0 200 400 frequency ( Hz ) DFT magnitudeF fHR 0.0 0.2 0.4 0.6 0.8 1.0t ( sec ) chirp signal sin ( ω ( t ) t ) 0.0 0.2 0.4 0.6 0.8 1.0t ( sec ) 0 250 ( Hz ) frequency ω ( t ) /2π 128 t sufficient ( Nyquist ) sampling rate fLR sampling rate 0.0 0.2 0.4 0.6 0.8 1.0t ( sec ) ALIASINGlow-resolution ( LR ) sampling frequency 250 HzfLR −100 −50 0 50 100 frequency ( Hz ) DFT magnitudeF fLRFigure 2 : Top : A chirp harmonic oscillator sin ( 2πω ( t ) t ) , with instantaneous frequency ω ( t ) . Left : The shape of the high-resolution sample resembles the underlying chirp signal . Right : Close to t = 1 , the apparent frequency of the low-resolution sample does not match that of the chirp . This is an example of aliasing ( shown with red at its most extreme ) , and it happens when the sampling rate falls below the Nyquist rate , sN = 2 · sB , where sB is the highest non-zero frequency of the signal . Sampling at high-resolution ( left ) maintains the frequency of the chirp signal ( top ) . Sampling at a lower resolution ( right ) , this apparent chirped frequency is lost due to aliasing , which means that the lower-resolution sample has a fundamentally smaller capacity for resolving the information of the natural signal , and a higher sampling rate can resolve more information . Shannon ’ s sampling theory has since been generalized for multiple interleaved sampling frames ( Papoulis , 1977 ; Marks , 2012 ) . One result of the generalized sampling theory is that we can go beyond the Nyquist limit of any individual uniform sample by interleaving several uniform samples taken concurrently . When an image is down-sampled to a lower resolution , its high-frequency details are lost permanently and can not be recovered from any image in isolation . However , by combining multiple low-resolution images , it becomes possible to recover the original scene at a higher resolution . Moreover , different low-resolution samples may be sampled at different phases , such that the same high-resolution frequency information will be packed with a phase shift . As a consequence , when multiple low-resolution samples are available , the fundamental challenge of MFSR is de-aliasing , i.e . disentangling the high-frequency components ( Tsai , 1984 ) . The first work on MSFR ( Tsai , 1984 ) considered the reconstruction of a high-resolution image as a fusion of co-registered low-resolution images in the Fourier domain . With proper registration and fusion ( Irani & Peleg , 1991 ; Fitzpatrick et al. , 2000 ; Capel & Zisserman , 2001 ) , a composite super-resolved image can reveal some of the original high-frequency detail that would not have been accessible from single low-resolution image . In this work , we introduce HighRes-Net , which aims to provide an end-to-end deep learning framework for MFSR settings . Relation to Video and Stereo Super-Resolution While there are obvious similarities to Video SR ( Tao et al. , 2017 ; Sajjadi et al. , 2018 ; Yan et al. , 2019 ; Wang et al. , 2019b ) and Stereo SR ( Wang et al. , 2019d ) ( Wang et al. , 2019a ) , the setting of this work differs in several ways : HighResnet learns to super-resolve sets and not sequences of low-res views . Video SR relies on motion estimation from a sequence of observations . Also , prediction at time t = T relies on predictions at t < T ( autoregressive approach ) . Whereas in our case , we predict a single image from an unordered set of low-res inputs . Also , the low-res views are multi-temporal ( taken at different times ) . Video SR methods assume that the input is a temporal sequence of frames . Motion or optical flow can be estimated to super-resolve the sequences of frames . In this work , we do not assume low-res inputs to be ordered in time . Our training input is a set of low-res views with unknown timestamps and our target output is a single image — not another sequence . 2.2 A PROBABILISTIC APPROACH . In addition to aliasing , MFSR deals with random processes like noise , blur , geometric distortions – all contributing to random low-resolution images . Traditionally , MFSR methods assume a-priori knowledge of the data generating motion model , blur kernel , noise level and degradation process ; see for example , Pickup et al . ( 2006 ) . Given multiple low-resolution images , the challenge of MFSR is to reconstruct a plausible image of higher-resolution that could have generated the observed lowresolution images . Optimization methods aim to improve an initial guess by minimizing an error between simulated and observed low-resolution images . These methods traditionally model the additive noise and prior knowledge about natural images explicitly , to constrain the parameter search space and derive objective functions , using e.g . Total Variation ( Chan & Wong , 1998 ; Farsiu et al. , 2004 ) , Tikhonov regularization ( Nguyen et al. , 2001 ) or Huber potential ( Pickup et al. , 2006 ) to define appropriate constraints on images . In some situations , the image degradation process is complex or not available , motivating the development of nonparametric strategies . Patch-based methods learn to form high-resolution images directly from low-resolution patches , e.g . with k-nearest neighbor search ( Freeman et al. , 2002 ; Chang et al. , 2004 ) , sparse coding and sparse dictionary methods ( Yang et al. , 2010 ; Zeyde et al. , 2010 ; Kim & Kwon , 2010 ) ) . The latter represents images in an over-complete basis and allows for sharing a prior across multiple sites . In this work , we are particularly interested in super-resolving satellite imagery . Much of the recent work in Super-Resolution has focused on SISR for natural images . For instance , Dong et al . ( 2014 ) showed that training a CNN for super-resolution is equivalent to sparse coding and dictionary based approaches . Kim et al . ( 2016 ) proposed an approach to SISR using recursion to increase the receptive field of a model while maintaining capacity by sharing weights . Many more networks and learning strategies have recently been introduced for SISR and image deblurring . Benchmarks for SISR ( Timofte et al. , 2018 ) , differ mainly in their upscaling method , network design , learning strategies , etc . We refer the reader to ( Wang et al. , 2019d ) for a more comprehensive review . Few deep-learning approaches have considered the more general MFSR setting and attempted to address it in an end-to-end learning framework . Reecently , Kawulok et al . ( 2019 ) proposed a shiftand-add method and suggested “ including image registration ” in the learning process as future work . In the following sections , we describe our approach to solving both aspects of the registration problem – co-registration and registration-at-the-loss – in a memory-efficient manner .

Summary: This paper presents a multi-frame super-resolution method applied to satellite imagery. It first estimates a reference image for the multiple input LR images by median filtering. Then it pairwise encodes the reference image and each of the multiple images in a recursive fashion then fuses the corresponding feature maps with residual blocks and bottleneck layers until only one feature maps for the entire multiple images obtained. In other words, LR images are fused into a single global encoding. Then, it applies a standard upsampling network to obtain the super-resolved image this image is fed into a network that estimates only the translational shift, and the shifted image with the estimated translation parameters finally resampled.