HighRes-net: Multi-Frame Super-Resolution by Recursive Fusion
Authors: Michel Deudon
Alfredo Kalaitzis
Md Rifat Arefin
Israel Goytom
Zhichao Lin
Kris Sankaran
Vincent Michalski
Samira E Kahou
Julien Cornebise
Yoshua Bengio
Venue:  (2020)
Keywords: multi-frame super-resolution
super-resolution
remote sensing
fusion
de-aliasing
deep learning
registration

Abstract: Generative deep learning has sparked a new wave of Super-Resolution (SR) algorithms that enhance single images with impressive aesthetic results, albeit with imaginary details. Multi-frame Super-Resolution (MFSR) offers a more grounded approach to the ill-posed problem, by conditioning on multiple low-resolution views. This is important for satellite monitoring of human impact on the planet -- from deforestation, to human rights violations -- that depend on reliable imagery. To this end, we present HighRes-net, the first deep learning approach to MFSR that learns its sub-tasks in an end-to-end fashion: (i) co-registration, (ii) fusion, (iii) up-sampling, and (iv) registration-at-the-loss. Co-registration of low-res views is learned implicitly through a reference-frame channel, with no explicit registration mechanism. We learn a global fusion operator that is applied recursively on an arbitrary number of low-res pairs. We introduce a registered loss, by learning to align the SR output to a ground-truth through ShiftNet. We show that by learning deep representations of multiple views, we can super-resolve low-resolution signals and enhance Earth observation data at scale. Our approach recently topped the European Space Agency's MFSR competition on real-world satellite imagery.

Under review as a conference paper at ICLR 2020
HIGHRES-NET: MULTI-FRAME SUPER-RESOLUTION BY RECURSIVE FUSION
Anonymous authors Paper under double-blind review
ABSTRACT
Generative deep learning has sparked a new wave of Super-Resolution (SR) al- gorithms that enhance single images with impressive aesthetic results, albeit with imaginary details. Multi-frame Super-Resolution (MFSR) offers a more grounded approach to the ill-posed problem, by conditioning on multiple low-resolution views. This is important for satellite monitoring of human impact on the planet from deforestation, to human rights violations that depend reliable im- agery. To this end, we present HighRes-net, the first deep learning approach to MFSR that learns its sub-tasks in an end-to-end fashion: (i) co-registration, (ii) fusion, (iii) up-sampling, and (iv) registration-at-the--ss low-res views is learned implicitly through a reference-framt channel, with explicit registration mechanism. We learn a global fusion operator that is applied recursively on an arbitrary number of low-res pairs. We introduce registered loss, by learning to align the SR output to a ground-truth through ShiftNet. We show that by learning deep representations of multiple views, we can super-resolve low-resolution signals and enhance Earth observation data at scale. Our approach recently topped the European Space Agency's MFSR competition on real-world satellite imagery.
Co-registration
INTRODUCTION
Multiple low-resolution images collectively contain more information than any individual low- resolution image. due to minor geometric displacements, e.g. shifts, rotations, atmospheric tur- bulence, and instrument noise. Multi-Frame Super-Rcsolution (MFSR) (Tsai, 1984) aims to recon- struct hidden high-resolution details from multiple low-resolution views of the same scene. Single Image Super- Reppe-Resoottion (SISR), as a special case of MFSR, has attracted much attention in the computer vision, machine learning and deep learning communities in the last 5 years, with neu- ral networks learning complex image priors to upsample and interpolate images (Xu et al., 2014; Srivastava et al., 2015; He al., 2016). However, the meantime not much work has explored the learning of representations for the more general problem of MFSR to address the additional challenges of co-registration and fusion of multiple low-resolution images.
This paper explores how Multi-Frame Super-Resolution (MFSR) can benefit from recent advances in learning representations with neural networks. To the best of knowledge, this work is the first to introduce deep-learning approach that solves the co-registration, fusion and registration-at-tee loss problems in an end-to-end learning framework.
Prompting this line of research is the increasing drive towards planetary-scale Earth observation monitor the environment and human rights violations. Such observation can be used inform policy, achieve accountability and direct on-the-ground action, e.g. within the framework of the Sustainable Development Goals (Jensen & Campbell, 2019).
Nomenclature Registration is the problem of estimating the relative geometric differences be- tween two images (e.g. due to shifts, rotations, deformations). Fusion, the MFSR context, is the problem of mapping multiple low-res representations into single representation. By registration, we mean the problem of registering all Jow-resolution views to improve their fusion. mean the problem of registering the super-resolved reconstruction
By angistration-a-thh--lee

---

Under review as a conference paper at ICLR 2020
to the high-resolution ground-trun truuth-trthhhh prior computing the loss. This gives rise to the notion of registered loss.
Co-registration of multiple images is required for longitudinal studies of land change and environ- mental degradation. The fusion multiple images is key to exploiting cheap, high-revisit-frequeee satellite imagery, but of owwressoution, moving away from the analysis of infrequent and expensive high-resolution images. Finally, beyond fusion itself, super-resolved generation is required through- the technical stack: both for labeling, but also for human oversight (Drexler, 2019) demanded by legal context (Harris et al., 2018).
Summary of contributions
We propose deep architecture that learns to fuse arbitrary number of low- resolution frames with implicit co-registration through a reference-frame channel.
High
ShiftNet: Inspired by HomographyNet (De'Tone et al., 2016), define model that learns to register and align the super-resolved output of HighRes-ned net, using ground-truth high-resolution frames supervision. This registration-at-th--boo mechanism enables more accurate feedback from the loss function into the fusion model, when comparing a super-resolved output to a ground truth high resolution image. Otherwise, MFSR model would naturally yield blurry outputs to compensate for the lack of registration, to correct for sub-pixel shifts and account for misalign- ments in the loss.
By combining the two components above, we contribute the first architecture to learn fusion and registration end-to-end.
We test and compare our approach to several baselines on real-world imagery from the PROBA-V satellite of ESA. Our performance has topped the Kelvins competition MFSR, organized by the Advanced Concepts Team of ESA (Märtens et al., 2019) (see section 5).
Supervres
Super-res
High-ss
Figure 1: HighRes-net combines many low-resolution images (300 meters/pixel) into one image of superior resolution. The same site shot in high-resolution (1 (100m/pix) also shown for reference. Source of low-res and high-res: imgset1087 and ingset 0285 of PROBA-V dataset, section 5.
The rest of the paper is divided as follows: in Section 2, we discuss related work on SISR and MESR; Section 3 outlines HighRes-net and section 4 presents ShiftNet, a differentiable registration component that drives our registered loss mechanism during end-to-end training. We present our results in section 5, and in Section we discuss some opportunities for and limitations and risks of super-resolution.
2 BACKGROUND
2.1 MULTI-FRAME SUPER-RESOLUTEE
How much detail can we resolve in the digital sample of some natural phenomenon? Nyquist (1928) observed that it depends on the instrument's ' sampling rate and the oscillation frequency the under- lying natural signal. Shannon (1949) built sampling theory that explained Nyquist's observations when the sampling rate is constant (uniform sampling) and determined the conditions aliasing in sample. Figure 2 illustrates this phenomenon.
2

---

Under review as a conference paper at ICLR 2020
chipp signal sin(2me(()r)
0.0
Lo
highvresolaatoon (HR) sampling frequeccy
fux aesolution (LR) sampling froquency 250 ALMSING
((K)
Figure 2:
Top: A chirp harmonic oscillator sin (ttw())t). with instantaneous frequency wt)). Left: The shape of the high-resolution sample resembles the underlying chirp signal. Right: Close to t = 1. the apparent frequency of the low- resolustttinn sample does not match that of the chirp. This is an example of aliasing (shown with red at its most extreme). and it happens when the sampling rate falls below the Nyquist rate, AN 2 sp. where ag the highest non-zero frequeney of the signal.
Sampling at high-resolution (left) maintains the frequency of the chirp signal (top). Sampling at lower resolution (right), this apparent chirped frequency is lost due to aliasing, which means that the lower-resolution sample has fundamentally smaller capacity for resolving the information of the natural signal, and a higher sampling rate can resolve more information.
Shannon's is sampling theory has since been generalized for multiple interleaved sampling frames (Pa- poulis, 1977; Marks, 2012). One result of the generalized sampling theory that we can go beyond the Nyquist limit of any individual uniform sample by interleaving several uniform samples taken concurrently. When an image is down-sampled to a lower resolution, its high-frequency details lost permanently and cannot be recovered from any image in isolation. However, by combin- ing multiple low-resolution images, it becomes possible to recover the original scene at a higher resolution.
Moreover, different low-resolution samples may be sampled at different phases, such that the same high-resolution frequency information will be packed with phase shift. As consequence, when multiple low-resolution samples are available, the fundamental challenge of MFSR is de-aliasing, i.c. disentangling the freuency components (Tsai, 1984).
The first work on MSFR (Tsai, 1984) considered the reconstruction of high-resolution image as fusion of co-registered low-resolution images in the Fourier domain. With proper registration and fusion (Irani & Peleg, 1991; Fitzpatrick et al., 2000; Capel & Zisserman, 2001), composite super-resolved image can reveal some of the original high-frequency detail that would not have been accessible from single low-resolution image. In this work, we introduce HighRes-Net, which aims to provide an end-to-end deep learning framework for MFSR settings.
Relation Video and Stereo Super-Resolution While there are obvious similarities to Video SR (Tao et al., 2017; Sajjadi et al., 2018; Yan et al., 2019; Wang et al., 2019b) and Stereo SR (Wang et al., 2019d) (Wang et al., 2019a), the setting of this work differs in several ways: HighRes- net learns to super-resolve sets and not sequences of low-res views. Vidco SR relies motion estimation from a sequence of observations. Also, prediction at time = T relies on predictions at t < T (autoregressive approach). Whereas in case, we predict single image from an unordered set of low-res inputs. Also, the low-res views are multi-temporal (taken at different times).
Video SR methods assume that the input is a temporal sequence of frames. Motion optical flow can be estimated to super-resolve the sequences of frames. In this work, we do not assume low-res inputs be ordered in time. Our training input is of low-res views with unknown timestamps and our target output is a single image- - not another sequence.
2.2 A PROBABILISTIC APPROACH
In addition to aliasing, MFSR deals with random processes like noise, blur, geometric distortions all contributing to random low-resolution images. Traditionally, MFSR methods assume a-priori knowledge of the data generating motion model, blur kernel, noise level and degradation process; see for example, Pickup et al. (2006). Given multiple low-resolution images, the challenge of MFSR is to reconstruct plausible image of higherrresolution that could have generated the observed low- resolution images. Optimization methods aim to improve an initial guess by minimizing an error between simulated and observed low-resolution images. These methods traditionally model the
3

---

Under review as a conference paper at ICLR 2020
HighRes-net
LR
Encode (recursive fusion)
Decode (upsample)
Super resolve
ShiftNet
Lanezos resampling
(megeee)
integpllacc)
(Ax, Ay)
HR
SR
shifted SR
reference L.R encoding
encoding
(a) HighRes-net
(b) Registered loss
Figure 3: Schematic of the full processing pipeline, trained end-to-end. At test time, only HighRes-net is used, (a) HighRes-net: In the Encode stage, an arbitrary number of LR views paired with the reference low-res image (the median low-res in this work). Each LR view-reference pair is encoded into view-specific latent representation. The LR encodings are fused recursively into a single global encoding. In the Decode stage, the global representation is upsampled by certain zoom factor (x3 in this work). Finally, the super- resolved image is reconstructed by combining all channels of the upsampled global encoding. (b) Registered loss: Generally, the reconstructed SR will be shifted with respect the ground truth HR. ShiftNet learns estimate the (Az, Ay) shift that improves loss. Lanezos resampling: (Aæ, Ay) define two 1D shifting Lanezos kernels that translate the SR by separable convolution.
additive noise and prior knowledge about natural images explicitly, to constrain the parameter search space and derive objective functions, using e.g. Total Variation (Chan & Wong, 1998; Farsiu et al., 2004), Tikhonov regularization (Nguyen et al., 2001) or Huber potential (Pickup et al., 2006) to define appropriate constraints images.
In some situations, the image degradation process is complex or not available, motivating the de- velopment of nonparametric strategies. Patch-based methods learn form high-resolution images directly from low-resolution patches, e.g. with k-nearest neighbor search (Freeman et al., 2002; Chang et al., 2004), sparse coding and sparse dictionary methods (Yang et al.. 2010; Zeyde al., 2010; Kim & Kwon, 2010)). The latter represents images in an over-complete basis and allows for sharing prior across multiple sites.
In this work, we are particularly interested in super-resolving satellite imagery. Much of the re- cent work (2014) showed that training CNN for super-resolution is equivalent to sparse coding and dictio- nary based approaches. Kim et (2016) proposed an approach SISR using recursion to increase the receptive field of model while maintaining capacity by sharing weights. Many more networks and learning strategies have recently been introduced for SISR and image deblurring, Benchmarks for SISR (Timofte al., 2018), differ mainly in their upscaling method, network design, learning strategies, etc. We refer the reader to (Wang et al., 2019d) for a more comprehensive review.
Resolution has focused on SISR for natural images. For instance, Dong et al.
Few deep-learning approaches have considered the more general MFSR setting and attempted to address it in an end-to-end learning framework. Reecently, Kawulok et al. (2019) proposed a shift- and-add method and suggested "including image registration" in the learning process future work.
In the following sections, we describe our approach to solving both aspects of the registration prob- lem co-registration and registration-at-the--
in memory-effciiet manner.
HIGHRES-NET: MFSR BY RECURSIVE FUSION
In this section, we present HighRes-net, a neural network for multi-frame super-resolution inside a single spectral band (greyscale images), using joint co-registration and fusion of multiple low- resolution views in end-to-end learning framework. From high-level, HighRes-net consists of an encoder-decoder architecture and can be trained by stochastic gradient descent using high- resolution ground truth supervision, shown in Figure

---

Under review as a conference paper at ICLR 2020
Notation We denote by 0 the parameters of HighRes-net trained for a given upscaling factor y LR E R Rx is one of set of K low-resolution views from the same site v, where C, W and H are the number of imput channels, width and height of LR respectively. We denote by SR. F (LRu,11, ..... k) the output of HighRes-net and by HR, E RCxxWxxf ground truth high-resolution image. We denote by [Ti, T2] the concatenation of two images channel- wise. In the following we supress the index v over sites for clarity.
HighRes-Ner consists of three main steps: (1) encoding, which learns relevant features associated with each low-resolution view, (2) fusion, which merges relevant information from views within the same scene, and (3) decoding, which proposes a high-resolution reconstruction from the fused summary.
3.1 ENCODE, FUSE, DECODE
Embed, Encode The core assumption of MFSR is that the low-resolution image set contains col- lectively more information than any single low-resolution image alone, due to differences in pho- tometric spatial coverage for instance. However, the redundant low frequency information in multiple views can hinder the training and test performance of a MFSR model. We thus compute a reference image ref as a shared representation for multiple low-resolution views (LR)) and embed each image jointly with ref. This highlights differences across the multiple views (Sanchez et al., 2019), and potentially allows HighRes-net to focus on difficult high-frequency features such as crop boundaries and rivers during super-resolution. The shared representation or reference image intuitively serves an anchor for implicitly aligning and denoising multiple views in deeper layers. We refer to this mechanism as implicit co-registration.
HighRes-net'ss embedding layer emb, consists of convolutional layer and two residual blocks with PReLu activations (He et al., 2015) and is shared across all views. The embedded hidden states s? are computed in parallel as follows:
ref (c, i, median (LRi (,,,)), LRx(ci,j)), such that ref E CxWH
5 = emba (LLR,, ref)) E R.OxxxH
where Cj denotes the channels of the hidden state.
The imageset is padded if the number of low-res views K' is not a power of 2: we pad the set with dummy valued views, such that the new size of the imageset K is the next power of 2. See Algorithm 1, line 1.
Fuse The embedded hidden states 8f are then fused recursively, halving by two the number of low-resolution states at each fusion step t, shown in Figure 4. Given pair of hidden states s s HighRes-nel computes a new representation:
5% 5y] = |s) sy} + go (15), sJ]) E 20xxXWH ++ = ayfo (5) 85) CRCWWWH
(3)
(4)
where 5/, 5 are intermediate representations; go is a shared-representatie block (equation 3); fo is fusion block, and a, is 0 if the j-th low-resolution view is part of the padding, and I otherwise. fo squashes 2C,, input channels into C₁ channels and consists of a (cony))+PreLu). Intuitively, go aligns the representations and it consists of two (conv2d PreLU) layers.
within an inner residual
The blocks (Jo, go) are shared across all pairs and depths, giving it the flexibility to deal with variable size inputs and significantly reduce the number of parameters to learn.
Upscale and Decode After = loga K fusion layers, the final low-resolution encoded state sf contains information from all K input views. Any information of spatial location that was initially missing from LR,, is now encoded implicitly in sf" T called the depth of HighRes-net. Only then, s!" is upsampled with a deconvolutional layer (Xu et al., 2014) to higher-resolutin space f (Cxx>WXn1t The hidden high-resolution encoded state s eventually convolved with 1 x 1 2D kernel produce final seperede image SR² C
5

---

Under review as a conference paper at ICLR 2020
8
S
ge (5/: 8}}
fo (8f, 45)
8 1+11
8)
S
Figure 4:
HighRes net's global fusion operator consists of co-registration gs and a fusion fo block which aligns and combines two representations into a single representation.
The overall architecture of HighRes-ne net is summarized in Figure 3(a) and the pseudocode for the forward pass is given in Algorithm 1.
Algorithm 1: HighRes-net forward pass
Input: low-res views LR1 LRK
..RRK..RKK aq...ak) + pad [LRI... LRxe)
11 pad inputs to next power of
2 sf + encode (LR.)
// parallelized across 1...
3 T < log2 K
// Susion depth
k +--- K
5 for = 1...T do
for
=1...k/2 do
sj t fuse (5)-1 k-s. ax-i)
// encoded
k =k/2
9 SR. & decode (s?)
Output: super-resolved view SR
REGISTRATION MATTERS
Co-registration matters for fusion. HighRes-net learns to implicity co-register multiple low- resolution views LR; and fuse them into single super resolved image SRg.
more explicit registration-attthe-lss can also be used for measuring similarity metrics and dis- tances between SRa and H. Indeed, training HighRes-Net alone, by minimizing reconstruction еиrоr such as the mean-squared eтror between SRg and R,, leads to blurry outputs, since the neural network has to compensate for pixel and sub-pixel misalignments between output SRg and R.
Here, we present ShiftNet-Lanczos, neural network that can be paired with HighRes-net to account for pixel and sub-pixel shifts in the loss, depicted in Figure 3(b). Our ablation study A.2 and qualitative visual analysis suggest that this strategy helps HighRes-net learn to super-resolve and leads to clearly improved results.
4.1 SHIFTNETTLANCZOS
ShiftNet learns to align pair of images with sub-pixel translations. ShiftNet registers pairs of images by predicting two parameters defining a global translation. Once a sub-pixel translation is found for a given pair of images, it is applied through Lanezos shift kernel align the images.
ShiftNet
The architecture of ShiftNet is adapted from HomographyNet (DeTone et al., 2016). Translations are special case of homographies. In this sense, ShiftNet is simply special case of HomographyNet, predicting shift parameters instead of homography parameters. See Appendix A.3, for details on the architecture of ShiftNet.

---

Under review as a conference paper at ICLR 2020
Algorithm 2: Sub-pixel registered loss through ShiftNet-Lanczos
Input: S HR
// super-resolved view, high-resclutien greund-truth
(Ax, Ay) + ShiftNet (SRg, HR)
// register SR HR
KA < LanczosShfttKenell (A) SRa,a + SRg * RAr * BAy 4 la.A f- loss (SRo,a, HR)
// in sanezas kernels for y sub-pixel shifta
11 20 sub-pixel shift by separable 10 convolutions
1/ mbb-iiall registered loss
Output: E0,A
One major difference from NomographyNet is the way we train ShiftNet: In (DeTone et al., 2016), HomographyNet is trained on synthetically transformed data, supervised with ground-trute truth homog- raphy matrices. In our setting, ShiftNet is trained to cooperate with HighRes-net, towards the com- mon goal of MFSR (see section Objective function below).
Lanezos shift & interpolation kernel To shift and align an image by a sub-pixel amount, it must be convolved with filter that shifts for the integer parts and interpolates for the fractional parts of the translation. Standard options for interpolation include the nearest-neighhor, sinc, bilinear, bicubic, and Lanczos filters (Turkowski, 1990). The sine filter has an infinite support as opposed to any digital signal, so in practice it produces ringing or ripple artifacts phenomenon. The nearest-neighhor and bilinear filters do not induce ringing. but strongly attenuate the higher-frequency components (over-smoothing), and can filter reduces the ringing significantly by using only a finite part of the since (up to few lobes from the origin). Experimentally, we found the Lanczos filter perform the best.
example of the Gibbs
alias the image. The Lanezos
Objective function In our setting. registration benefits super-resolution. HighRes-net receives more informative gradient signals when its output is aligned with the ground truth high-resolution image. Conversely, super-resolution benefits registration, since good features are key to align images (Clement et al., 2018). We thus trained HighRes- Net and ShiftNet-Lanczer in a cooperative setting, where both neural networks work together to minimize objective function, as opposed to an adversarial setting where a generator tries to fool a discriminator. HighRes-net infers a latent super- resolved variable and ShiftNet maximises its similarity to ground truth high-resolution image with sub-pixel shifts.
By predicting and applying sub-pixel translations in a differentiable way, approach for registra- tion and super-resolution can be combined in an end-to-end learning framework. Shift-Net predicts sub-pixel shift A from a pair of high-resolution images. The predicted transformation is applied with Lanczos interpolation to align the two images at pixel level. ShiftNet and HighRes-Ner are trained jointly to minimize a common loss function, using backpropagation and stochastic gradient descent. Our objective function is composed of registered reconstruction loss computed as in Al- gorithm In our case, we used the corrected clear PSNR metric (cPSNR), chosen by ESA, which variant of the mean squared error, designed to correct for brightness and clouds in satellite images (Märtens et al., 2019), but the proposed architecture is decoupled from the choice of loss.
See Algorithm+ 4 for the pseudo-code for computing the alignment loss la We further regularize the L2 norm ShiftNet's ouput with a hyperparameter and our final joint objective given by:
Lo.a (S HR) l0. + A|1A/12
(5)
EXPERIMENTS AND RESULTS
Prior SR work has focused on super-resolving low-res images that are artificially generated by sim- ple bilinear down-sampling, (Bulat et al., 2018) The PROBA-V satellite has separate cameras on- board for capturing high-res/ low-res pairs. As far as we know, the PROBA-V dataset is the first publicly available dataset for MFSR that contains naturally occurring low-res and high-res pairs. This is in contrast to most of the work in SR (SISR, MFSR, Video SR, Stereo SR) that synthetically down-sample high-res images and frames (Wang et al., 2019e; Nah et al., 2019). Methods that
7

---

Under review as a conference paper at ICLR 2020
trained on artificially downscaled datasets fail to produce good results when applied to real-world low-resolution, low quality images (Shocher et al., 2018). For this reason, we experiment only on PROBA-V. dataset that does not suffer from biases induced by artificial down-sampling.
5.1 PROBA-V KELVIN DATASET
The performance of our method is illustrated with satellite imagery from the Kelvin competition, organized by ESA's Advanced Concept Team (ACT).
The Proba-V V Kelvin dataset (Märtens et al., 2019) contains 1450 scenes (RED and NIR spectral bands) from 74 hand-selected Earth regions around the globe at different points in time. The scenes are split into 1 116 scenes for training and 290 scenes for testing. Each data-point consists of exactly one 100m resolution image as 384 x 384 grey-scale pixel images (HR) and several 300m resolution images from the same scene 128 x 128 grey-scale pixel images (LR), spaced days apart. We refer the reader to the Proba-V V manual (Wolters et al., 2014) for further details on image acquisition.
Each scene comes with least 9 low-res views, and an average of 19. Each view comes with noisy quality map. The quality map is a binary map, that indicates concealed pixels due to volatile features, such as clouds, cloud shadows, ice, water and snow. The sum of clear pixels (Is in the binary mask) is defined as the clearance of a low-res view. These incidental and noisy features change fundamental aspects of the image, such the contrast, brightness, illumination and landscape features. We use the clearance scores to randomly sample from the imageset of low-res views, such that views with higher clearance are more likely to be selected. This strategy helps to prevent overfitting. See Appendix A.4 for more details.
Working with missing or noisy values A quality map can be used as a binary mask to indicate noisy occluded pixels, due to clouds, snow, or other volatile objects. Such mask can be fed as an additional input channel in the respective low-res view, the same fashion as thereference frame. available, neural networks
When missing value masks anomalous, noisy, or missing, when provided with such binary masks (see e.g. Che et al. (2018)). In satellite applications where clouds masks are available, other segmentation methods would be in order to infer such masks as a preprocessing step (e.g. Long et al. (2015)). In the case of the PROBA-V V dataset, we get improved results when we make no use of the masks provided. Instead we use the masks only to inform the sampling scheme within the low-res imageset to prevent overfitting.
learn which parts of the input
5.2 EXPERIMENTS
Across all experiments, we used the same hyperparameters, reported in Appendix A. By default, each imageset is padded to 32 views for training and testing, unless specified otherwise. Our pytorch implementation requires less than 9h of training on a single NVIDIA V100 GPU. test time, super- resolving imageset of size 128x128 by factor of 3, takes less than 0.2 seconds. Our code is made available on github'.
We evaluated different models on ESA's Kelvin competition. Our best model, HighRes-Ne trained jointly with shiftNet-L.anzza scored consistently at the top of the public and final leaderboard, see Table 1. In the following, we discuss several baselines and report our experiments.
5.3
COMPARISONS
ESA bascline upsamples each low-resolution view separately with bicubic up-sampling and av- erages those of maximum clearance.
SRResNet (Ledig et al., 2017) is a deep learning SISR baseline.
SRResNet-1 + shiftNet was trained with ShiftNet.
SRResNet-6 + shiftNet differs from the previous model during test time only. It independently
upsamples 6 low-resolution views with SRRevNet-J, co-registers the super-resolved images using shiftNet, and averages the 6 aligned super-resolved images into final prediction.
ACT baseline (Märtens et al., 2019) is Convolutional Neural Network with five fixed channels for the five clearest low-resolution views.
httpp::/ hitpss:/aanoynnenn http::///mooyyoous.pppaaaaa
44 bbeb- F84 F44
8

---

Under review as a conference paper at ICLR 2020
DeepSUM baseline (Molini et al., 2019) can be variant of SRResNer-- shiftNet. Multi- ple low-res views are independently upsampled, then co-registered and fused into a single image.
HighRes-net + shiftNet are described in sections and 4. Upsampling is done in the last step as opposed (Molini al., 2019).
Ensemble ensemble of two trained (HighRes-net shiftNet) models, one with K=l6 and one with K=32 imput views, whose outputs are averaged.
5.4
ESA KELVIN LEADERBOARD
The Kelvin competition used the corrected clear PSNR (ePSNR) quality metric as the standardized measure of performance. The cPSNR is variant of the Peak Signal to Noise Ratio (PSNR) used to compensate for pixel-shifts and brightness bias. We refer the reader to (Märtens al., 2019) for the motivation and derivation of this quality metric. The cPSNR metric is normalized by the score of the ESA baseline algorithm so that score smaller than means "better than the baseline" and lower is better. We also use it training objective with sub-pixel registration (see also section 3(b) on ShiftNet).
Table 1: Public and final leaderboard scores for the ESA's Kelvin competition. Lower score means a better reconstruction.
Method
Public score
Final score
SRResNet (Ledig et al., 2017)
1,0095
1.0084
ESA baseline
1.0000
1.0000
SRResNet-1 + shiftNet
1.0002
0,9995
ACT baseline (Märtens al., 2019)
0.9874
0.9879
SRResNet-6 + shiftNet
0.9808
0.9794
HighRes-net shiftNet (ours)
0.9496
0.9488
HighRes-net + shiftNet Ensemble (ours)
0.94738
0.94774
DeepSUM (Molini al., 2019)
0.94884
0.94744
5.4.1 ABLATION STUDY
We further ran an ablation study on the available labeled data (1450 image sets), split in 90% / 10% for training and testing. Our results suggest that more low-resolution views benefit the reconstruction еггоr, plateuing after 16 views, see Appendix A.2. Another finding is that registration matters for MESR, both co-registering low-res views, and registering-at lossrlllss see Appendix A.3. Finally, selecting the h clearest views for fusion can lead to ovefitting. One remedy is to randomly sample the views with a bias for clearance, see A.4.
DISCUSSION
6.1 THE IMPORTANCE GROUNDED DETAILS
The PROBA-V satellite (Dierckx et al., 2014) was launched by ESA to monitor Earth's vegetation growth, water resources and agriculture. As form of data fusion and enrichment, multi-frame super-resolution could enhance the vision of such satellites for scientific and monitoring applications (Carlson & Ripley, 1997: Pettorelli et al., 2005). More broadly, satellite imagery can help NGOs and non-profits monitor the environment and human rights (Cornebise et al., 2018; Helber et al., 2018; Rudner et al., 2019; Rolnick et al., 2019) at scale, from space, ultimately contributing to UN sustainable development goals. Low-resolution imagery is cheap sometimes free,, and it is frequently updated. However, with the addition fake or imaginary details, such enhancement wouldn't be valuable scientific, legal, or forensic evidence.
6.2 FUTURE WORK
Registration matters at the loss stage but also at the fusion stage. The latter not explicit in our model and the reason why and how it works is less understood. Learning sample reference frame

---

Under review as a conference paper at ICLR 2020
and learning to fuse multiple representations with attention could also be promising approach to extend HighRes-net. net.
Ensuring authenticity of detail is major challenge and quantifying uncertainty of super-resolved images is an important line of future work for real world applications. Along this line research, the question of how evaluate super-resolved image is important for downstream tasks and, more generally, similarity metrics remain an open question for many computer visions tasks (Bruna et al., 2015; Johnson et al., 2016; Isola et al., 2017; Ledig al., 2017).
6.3 CONCLUSION
In this paper, we presented HighRes-net the first deep learning approach to multi-frame super- resolution that learns typical sub-tasks of MFSR in an end-to-end fashion: (i) co-registration, (ii) fusion, (iii) up-sampling, and (iv) peiistration-t-the-loo
recursively fuses variable number low-resolution views by learning global fusion operator: The overall fusion also aligns all low-resolution views with implicit co-registration mechanism through the reference channel. We also introduced ShiftNet-Lanczos, a network that learns to regis- ter and align the super-resolved output of HighRes-net with high-resolution ground-truth.
Registration is important both to align multiple low-resolution inputs (co-registration) and to com- pute similarity metrics between shifted signals. Our experiments suggest that end-to-end co- operative setting (HighRes-net + ShiftNet-Lanczos helps with training and test performance. By design, our approach is fast train, faster test, and low in terms of memory-footprint by doing the bulk of the computational work (co-registration + fusion) on multiple images while maintaining their low-resolution height & width.
There is an ongoing proliferation of low-resolution yet high-revisit low-cost satellite imagery, but they often lack the detailed information of expensive high-resolution imagery. We believe MFSR can raise its potential to NGOs and non non-profits that contribute to the UN Sunstainable Development Goals.
10

---

Under review as a conference paper at ICLR 2020
REFERENCES
Joan Bruna, Pablo Sprechmann, and Yann LeCun. Super-resolution with deep convolutional sufficient statisties. arXiv preprint urXiv::511.000 1.06666, 2015.
Adrian Bulat, Jing Yang. and Georgios Tzimiropoulos. To learn image super-resolution, a gan to learn how to do image degradation first. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 185-200, 2018.
David Capel and Andrew Zisserman. Super-resolution from multiple views using learnt image models. In Proceedings of the 2001 IEEF Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001, volume 2. pp. 11-11. IEEE, 2001.
Toby N Carlson and David A Ripley. On the relation between ndvi, fractional vegetation cover, and leaf area index. Remote sensing of Environment, 62(3):241-22 222,, 1997.
Tony F Chan and cessing, 7(3):370-775, 1998.
Wong. Total variation blind deconvolution. IEEE transactions on Image Pro-
Hong Chang. Yan Yeung. and Yimin Xiong. Super-resolution through neighbor embedding. In Proceedings of 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2004. CVPR 2004., volume 1, pp. I-I. IRRF, 2004.
Zhengping Che, Sanjay Purushotham, Kyunghyun Cho, David Sontag, and Yan Liu. Recurrent neural networks multivariate time series with missing values. Scientific reports, 8(1):6085, 2018.
Colin B Clement, Matthew Bierbaum, and James Sethna. Image registration and super resolution from first principles. arXiv preprint arXiv: 1809.05583 2018
Julien Cornehise, Daniel Worrall, Micah Farfour, and Milena Marin. Witnessing atrocities: Quantifying villages destruction in darfur with crowdsourcing and transfer learning. In Al for Social Good NIPS2018 Workshop. Montréal, Canada. 2018
Daniel DeTooee, Tomasz Malisiewiez, and Andrew Rabinovich. Deep image homography estimation. arXiv preprint arXiv:1606.00 03798, 2016.
Wouter Dierckx, Sindy Sterckx, Iskander Benhadj, Stefan Livens, Geert Duhoux, Tanja Van Achteren, Michael Francois, Karim Mellab, and Gilbert Saint. Proba-v mission for global vegetation monitoring: standard products and image quality. International Journal of Remote Sensing, 35(7):259--2114 2014.
Chao Dong, Chen Change Loy, Kaiming Hc, and Xiaoou Tang. Learning deep convolutional network for image super-rcsolution. In European conference on computer vision, pp. 199.199. Springer, 2014.
K Eric Drexler. Reframing superintelliggnce: Comprehensive ai services general intelligence, 2019.
Sina Farsiu, M Dirk Robinson, Michael Elad, and Peyman Milanfar. Fast and robust multiframe super resolu- tion. IEEE transactions on image processing, (3110):327-13: 1344, 2004.
J Michael Fitzpatrick, Derek LG Hill, Calvin R Maurer, al. Image registration. Handbook of medical imaging, 2:447-513, 2000.
William Freeman, Thouis R Jones, and Egon Pasztor. Example-base super-resolution. IEEE Computer graphics and Applications, 22(2):56-66 65, 2002.
Theresa Harris, Jonathan Drake, Jessica M. Wyndham, Susan R. Wolfinbarger, Stephen D. Lott, and Michael Lemner. Geospatial evidence international human rights litigation: Technical and legal considerations. Technical report, AAAS Scientific Responsibility, Human Rights and Law Program, 2018
Kaiming He. Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human- level performance on imagenet classification. In Proceedings of the 2015 IEEE International Conference on Computer Vision (ICCV), ICCV '15, pp. 1036-1034, Washington, DC, USA, 2015. IEEE Computer So- cicty. ISBN 1888--4673-8391 -2. ICCV.2015.123.
doi: 109 223.5.123. URL http: httt::://cx..o..ogg/1......
Kaiming He. Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedingy of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Patrick Helber, Bradley Gram- Hansen. Indhu Varatharajan, Faiza Azam, Alejandro Coca-Castre Veronika Kopackova, and Piotr Bilinski.. Mapping informal settlements in developing countries with multi-resolution, multi-spectral data. arXiv preprint arXiv: 1812.00812. 2018.

---

Under review as a conference paper at ICLR 2020
Michal Irani and Shmuel Peleg. Improving resolution by image registration. CVGIP: Gruphical models and image processing, 53(3):31-225 1991.
Phillip Isola, Jun- Yan Zhu. Tinghui Zhou, and Alexei A Efros. Image-to-image translation with conditional adversarial networks. In Proceedings of the IEFE conference on computer vision and pattern recognition, pp. 1125-1134, 2017.
David Jensen Jillian Campbell.
The promise and peril of a digital ecosystem for the planet, Septem- 2019. URL https:/ htt///dduummmmm um.com/ /edevvdddeddeeeeeene 19936/bbu h/b.//diiiiignggg.............. a digitall ecosystem- sar-the-plancee
[Online; posted 1 11-September-2019].
Justin Johnson, Alexandre Alahi, and Li Fei-Fci. Perceptual losses for real-time style transfer and super- resolution. In European conference on computer vision, pp. 694-711. Springer, 2016.
Michal Kawulok, Pawel Benecki, Krzysztof Hrynczenko, Daniel Kostrzewa, Szymon Piechaczek, Jakub Nalepa, and Bogdan Smolka. Deep learning for fast super-resolution reconstruction from multiple images. In Real- Time Image Processing and Deep Learning 2019, volume 10996, pp. 109960B. International Society for Optics and Photonies, 2019.
Jiwon Kim, Jung Kwon Lee, and Kyoung Mu Lee. Deeply-recursive convolutional network for image super- resolution. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 1637- 1645, 2016.
Kwang In Kim and Younghee Kwon, Single-image super-resolution using sparse regression and natural image prior. IEEE transactions on pattern analysis and machine intelligence, 32(6):117--113 2010.
Diederik P Kingma and Jimmy Ba.
Adam:
A method for stochastic optimization. arXiv preprint
arXiv: 1412.6980, 2014.
Christian Ledig. Lucas Theis, Ferene Huszár, Jose Caballero, Andrew Cunningham, Alejandro Acosta, Andrew Aitken, Alykhan Tejani, Johannes Totz, Zehan Wang, Photo-realistir single image super-resolution using generative adversarial network. Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4681-4690, 2017.
Jonathan Long, Evan Shelhamer, and Trevor Darrell, Fully convolutional networks for semantic segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3431-3440, 2015.
Robert II Marks. Introduttoon to Shannon sampling and interpolation theory. Springer Science & Business Media, 2012.
Marcus Märtens, Dario Izzo, Andrej Krzic, Daniël Cox. Super-resolution of proba-v images using convo- lutional neural networks. Astrodynamics, 3(4):387-40,, Dee 2019.
Andrea Bordone Molini, Diego Valsesia, Giulia Fracastoro, and Enrico Magli. Deepsum: Deep neural network for super-resolution of unregistered multitemporal images. arXiv preprini arXiv:1907.06490 2019.
Seungjun Nah, Radu Timofte, Sungyong Baik, Seokil Hong, Gyeongsik Moon, Sanghyun Son, and Kyoung Mu Lee. Ntire 2019 challenge on video deblurring: Methods and results. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition Workshops, pp. 0-0, 2019.
Nhat Nguyen, Peyman Milanfar, and Gene Golub. A computationally efficient superresolution image recon- struction algorithm. IEEE transactions on image processing. 10(4):573-555 2001.
Harry Nyquist. Certain topics in telegraph transmission theory. Transactions of the American Institute of Electrical Engineers, 47(2):117-644, 1928.
Athanasios Papoulis. Generalized sampling expansion. IEEE transactions on circuits and systems, 24(11): 652 654, 1977.
Nathalic Pettorelli Jon Olav Vik, Atle Mysterud, Jean-Michel Gaillard, Compton J Tucker, and Nils Chr Stenseth, Using the satellite-derieed ndvi to assess ecological responses to environmental change. Trends in ecology evolution, 20(9):503 510, 2005.
Lyndsey C Pickup, Stephen J Roberts, and Andrew Zisserman. Optimizing and learning for super-resolution. In BMVC, volume 9, pp. 4-7, Citeseer, 2006.
David Rolnick, Priya L Donti, Lynn H Kaack, Kelly Kochanski. Alexandre Lacoste, Kris Sankaran. An- drew Slavin Ross, Nikola Milojevie-Dupont, Natasha Jaques, Anna Waldman-Brown, et al. Tackling climate change with machine learning. arXiv preprint arXiv: N0v.:9996000033 2019.
12

---

Under review as a conference paper at ICLR 2020
Tim GJ Rudner, Mare Rußwurm, Jakub Fil, Ramona Pelich, Benjamin Bischke, Veronika Kopačková, and Piotr Biliñíski. Multišnet: Segmenting flooded buildings via fusion of multiresolution, multisensor, and multitemporal satellite imagery. In Proceedings the AAAI Conference on Artificial Intelligence, volume 33, pp. 709, 2019.
Mehdi SM Sajjadi, Raviteja Vemulapalli, and Matthew Brown. Frame-recurrent video super-resolution. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 6626-6634, 2018.
Eduardo Sanchez, Mathieu Serturier, and Mathias Ortner. Learning disentangled representations of satellite image time series. arXiv preprint 19Xi::903.00863 2019.
Claude Elwood Shannon. Communication in the presence of noisc. Proceedings of the IRE, 37(1))10-21, 1949.
Assaf Shocher, Nadav Cohen, and Michal Irani. zero-shot super-resolution using deep internal learning. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3118-3122 3126, 2018.
Rupesh Kumar Srivastava, Klaus Greff, and Jürgen Schmidhuber. Highway networks. arXiv:1505 150.00877 2015.
arXiv preprint
Xin Tao, Hongyun Gao, Renjie Liao, Jue Wang, Jiaya Jia. Detail-revealing deep video super-resolution, In Proceedings of the IEEE International Conference on Computer Vision, pp. 4472-4480, 2017.
Radu Timofte, Shuhang Gu, Jiqing Wu, Lue Van Gool, Lei Zhang, Ming-Hsuan Yang. Muhammad Haris, et al. Ntire 2018 challenge on single image super- superresolution: Methods and results. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, June 2018.
RY Tsai. Multiple frame image restoration and registration, Advances Computer Vision and Image Process- ing. 1:1715- 1984, 1984.
Ken Turkowski. Filters for common resampling tasks. In Graphics gems, pp. 147-165. Academic Press Professional, Inc., 1990.
Oriol Vinyals, Samy Bengio, and Manjunath Kudlur. Order matters: Sequence to sequence for sets. arXiv preprint arXiv:1511.066 06399, 2015.
Longguang Wang, Yingqian Wang, Zhengfa Liang, Zaiping Lin, Jungang Yang. Wei An, and Yulan Guo. Learning parallax attention for stereo image super-resolution. In Proceedings of the IEEE Conference on Computer Vision and Patlern Recognition, pp. 12250-12259, 2019a.
Xintao Wang, Kelvin CK Chan, Ke Yu, Chao Dong, and Chen Change Loy. Edvr: Video restoration with enhanced deformable convolutional networks. In Proceedings of the IEEE Conference Computer Vision and Puttern Recognition Workshops, pp. 0-0, 2019b.
Yingqian Wang, Longguang Wang, Jungang Yang. Wei An, and Yulan Guo. Flickr1024: A large-scale dataset for stereo image super-resolue resolutiso Proceedings of the IEEE International Conference on Computer Vision Workshops, pp. 0-0, 2019c.
Zhihao Wang, Jian Chen, and Steven CH Hoi. Deep learning for image super-resolution: A survey. arXiv preprint /902 06068, 2019d.
Erwin Wolters, Wouter Dierekx. Dries, and Else Swinnen. Proba-v products user manual. VITO. http://proba V. vgl. vito. routessttfffllllfllee/ruutt.... User Manual. pdf. 2014.
Xu, Jimmy SJ Ren, Ce Liu, and Jiaya Deep convolutional neural network for image deconvolution. In Advances in Neural Information Processing Systems, pp. 1790-1798, 2014.
Bo Yan, Chuming Lin, and Weimin Tan. Frame and feature-context video super-resolution. arXiv preprint arXiv:1999.1355,, 2019.
Jianchao Yang. John Wright, Thomas Huang, and Ma. Image super-resolution via sparse representation. IEEE transactions on image processing. 19(11):2861 28711 2010.
Roman Zeyde, Michael Elad, and Matan Protter. single image scale-up using sparse-representations.. In International conference on curves and surfaces, pp. 711-730. Springer, 2010.
13

---

Under review as a conference paper at ICLR 2020
A APPENDIX
A.1 EXPERIMENTAL DETAILS
We trained our models on low-resolution patches of size 64x64. HighRes-net's architecture is re- ported in Table 3. We denote by Conv2d(in, out, k, S, p) conv2I) layer with in and oul input/output channels, kernel of size (k,k), stride (s,s) and padding p. We used the ADAM optimizer (Kingma & Ba, 2014) with default hyperparameters and trained our models batches of size 32, for 400 epochs, using 90% of the data for training and 10% for validation. Our learning rate is initialized to 0.0007, decayed by factor of 0.97 if the validation loss plateaus for more than 2 epochs. For the regularization shiftNet, we employed 0.000001.
Table ResidualBlock(h) architecture
layerO
Conv2d(in=h, out=h, k3, sl, p!)
layerl
PReLU
layer2 layer3
Conv2d(in=h, out=h, k3, sl, pl)
PReLU
Table 3: HRNet architecture
Step
Layers
Number of params
encode
Conv2d out=64, k3, sl, pl)
1216
PReLU
ResidualBlock(64)
73858
ResidualBlock(64)
73858
Conv2d(in=64, out=64, k3, sl, pl)
36928
fuse
ResidualBBook(1225
295170
Conv2d(in=128, out=64, k3, sl, pl)
73792
PReLU
decode
Conv'l Con/Trransoseea((nnz6. out=64, k3, sl) PreLU
36928
Conv2d(in=64, out=1, kl, sl)
65
residual (optional)
Upsample(scale factorr33, mideibicubic)
0
591818 (total)
Thanks to weight sharing. HighRes-net super-resolves scenes with 32 views in recursive steps, while requiring less than 600K parameters. ShiftNet has more than 34M parameters (34187648) but is dropped during test time. We report GPU memory requirements in table 4 for reproducibility purposes.
Table GPU memory requirements train HighRes-net ShiftNet on patches of size 64x64 with batches of size 32, and variable number of low-resolution frames.
# views
32
16
4
GPU memory (GB)
27
A.2 HOw MANY FRAMES DO YOU NEED?
We trained and tested HighRes-net net with ShiftNet using 1 to 32 frames. With a single image, our approach performs worse than the ESA baseline. Doubling the number of frames significantly im- proves both our training and validation scores. After 16 frames, our model's performance stops increasing show in Figure 5.
14

---

Under review as a conference paper at ICLR 2020
1.00
test score
train_scorf
0.98
Scores
0.94
0.92
0.90
2
4
16
32
Figure 5: Public leaderboard scores vs. nviews for HighRes-net + -ShiftNet. Lower better.
A.3 REGISTRATION MATTERS
Registered loss The only explicit registration that we perform is at the loss stage, to allow the model partial credit for a solution. This solution can be enhanced but otherwise mis-registered wrt to the ground truth. We trained our base model HighRes-net without ShiftNett-anczor and observed a drop in performance as shown Table 5. Registration matters and aligning outputs with targets helps HighRes-net generate sharper outputs and achieve competitive results.
Table 5: Registration matters:
Train and validation scores for HighResNet trained with and without
ShiftNet-Lanczan Lower is better.
HighRes-net
train score
test score
w/o registration
0.9616
0.9671
ShiftNet Lanczos
0.9501
0.9532
Implicit co-registration The traditional practice in MFSR explicitly co-register the LR views prior to super-resolution (Tsai, 1984; Molini et al., 2019). The knowledge of sub-pixel miss- alignments tells an algorithm what pieces of information fuse from each LR image for any pixel in the SR output. Contrary to the conventional practice in MFSR, we propose implicit co-registeration by pairing LR views with reference frame a.k.a. anchor. In this sense, we never explicitly compute the relative shifts between any LR pair. Instead, we simply stack each view with chosen reference frame as an additional channel to the input. We call this strategy implicit co-re co-registration, We found this strategy to be effective in the following ablation study which addresses the impact of the choice of a reference frame aka anchor.
We observe the median reference is the most effective in terms of train and test score. We suspect the median performs better than the mean because the median is more robust to outliers and can help denoise the LR views. Interestingly, training and testing without shared reference performed worse than the ESA baseline. This shows that co-registration (implicit or explicit) matters. This can be due to the fact that the model lacks information align and fuse the multiple views.
ReLU). Layer 2, and are followed by MaxPool. oo od. The final output is flattened to vector x of size 32768. Then, compute vector of size 1024, ReL.U(fe (el.U[(fl(dropouf (dropout(:))). The final shift prediction is
ShiftNet architecture ShitNet has layers of (conv2D BatchNorm2d
15

---

Under review as a conference paper at ICLR 2020
Table 6: Train and validation scores for HighRes-net ShiftNet-Lanncos trained and tested with different references as imput. Lower is better.
Reference
train score
test score
None (no co-registration)
1.0131
1.0088
Mcan of LRs
0.9636
0.9690
Median or 9 LRs (base)
0.9501
0.9532
fc2 of size 2. The bulk of the parameters come from fcl, with 32768 x 1024 weights. These alone, account for 99% of ShiftNet's parameters. Adding MaxPoo 12d on top of layer 3, 3,5, halves the parameters of ShiftNet.
A.4 TOWARDS PERMUTATION INVARIANCE
desirable property a fusion model acting on an un-ordered set of images, is permutation invariance: the output of the model should be invariant to the order in which the LR views fused. An easy approach to encourage permutation invariant neural networks is to randomly shuffle the inputs at training time before feeding them a model (Vinyals al., 2015).
In addition to randomization, we still want to give more importance to clear LR views (with high clearance score), which can be done by sorting them by clearance. A good trade-off between uniform sampling and deterministic sorting by clearance, is to sample l: LR views without replacement and with bias towards higher clearance:
30,
p(i Cl...., C₂)
-==1 eBC;
where & is the total number of LR views, C is the clearance of LR; and B regulates the bias towards higher clearance scores,
When B 0, this sampling strategy corresponds uniform sampling and when 3 +inf, this corresponds to picking the k-clearest views in a deterministic way. Our default model was trained with 3 50 and our experiments are reported in Table 7.
Table 7: Validation scores vs. nviews for HighRes-net ShiftNet. Lower better.
Sampling strategy
train score
test score
3 = Do (k-clearest)
0.9386
0.9687
3 0 (uniform-k)
0.9638
0.9675
B 50 (base)
0.9501
0.9532
From Table Do reaches best training score and worst testing score. For 3 50 and 0, the train/test gap is much more reduced. This suggests that the deterministic strategy is overfitting and randomness prevents overfitting (diversity matters). the other hand, / performs sig- nificantly better than 3 0 suggesting that biasing model towards higher clearances could be beneficial i.c., clouds matter too.
16


Reviews:
[{"review_id": "rygMx9RTYS", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes a framework including recursive fusion to co-registration and registration loss to solve the problem that the super-resolution results and the high-resolution labels are not pixel aligned.  Besides, the method is able to achieve good performance in the Proba-V Kelvin dataset. However, I have some concerns about this paper:\n\n1) This paper lacks many references. Recently, many works focus on multi-frame super-resolution containing video super-resolution and stereo image super-resolution via deep learning.  They are using multiple low-resolution image to construct high-resolution image. For example:\n\nStereo super-resolution:\n\nJeon, Daniel S., et al. \"Enhancing the spatial resolution of stereo images using a parallax prior.\" *Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition*. 2018.\n\nWang, Longguang, et al. \"Learning parallax attention for stereo image super-resolution.\" *Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition*. 2019.\n\nVideo super-resolution:\n\nTao, Xin, et al. \"Detail-revealing deep video super-resolution.\" *Proceedings of the IEEE International Conference on Computer Vision*. 2017. \n\nFRVSR: Sajjadi, Mehdi SM, Raviteja Vemulapalli, and Matthew Brown. \"Frame-recurrent video super-resolution.\" *Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition*. 2018.\n\nFFCVSR: Yan, Bo, Chuming Lin, and Weimin Tan. \"Frame and Feature-Context Video Super-Resolution.\" *Proceedings of the AAAI Conference on Artificial Intelligence*. Vol. 33. 2019.\n\nEDVR: Wang, Xintao, et al. \"Edvr: Video restoration with enhanced deformable convolutional networks.\" *Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops*. 2019.\n\n2) Recursive fusion is aimed to fuse multiple low-resolution image information. Recently, more and more work utilize different methods to fuse multiple low-resolution image. For example, Tao et al proposes SPMC (Sub-pixel Motion Compensation) to align image, FRVSR uses unsupervised flow network that predicts optical flow to warp image, FFCVSR directly concatenate low-resolution image as the input of 2D convolutional network to fuse the information, and EDVR fuses multiple image features via utilizing deformable convolution. Thus, what is the advantage of recursive fusion compared to the above methods? This paper should discuss the difference between recursive fusion and the above methods.\n\n3) Registration loss is important in this paper and it can solve the problem the output SR is not pixel-wise aligned to the HR ground truth. Registration loss utilizes ShiftNet that is adapted from HomographyNet. Thus, what is the difference between ShiftNet and HomographyNet? This paper should add some details about ShiftNet and Lanczos interpolation. \n\n4) It is better to test more datasets and compare with more state-of-the-art methods. This paper only tests in a satellite image dataset. Some datasets can be considered such as VID4 dataset in video super-resolution.", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "Skl8XqOndH", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes an end-to-end multi-frame super-resolution algorithm, that relies on a pair-wise co-registrations and fusing blocks (convolutional residual blocks), embedded in a encoder-decoder network 'HighRes-net' that estimates the super-resolution image. Because the ground truth SR image is typically misaligned with the estimation SR image, the authors proposed to learn the shift with a neural network 'ShiftNet' in a cooperative setting with HighRes-net. The experiments were performed on the ESA challenge on satellite images, showing good results.\n\nOverall, I found this paper interesting, and the method described is both clever and efficient. While some points need to be clarified, I am in favor of accepting this paper to ICLR.\n\nPositive aspects:\n- the paper is very clear and easy to read, with nice figures.\n- a sensitivity analysis on many different parameters or types of inputs are made, which makes this paper an interesting research paper. For example, the tests on the type of reference image to stack at the beginning are very interesting.\n- While I am not an expert on super-resolution, I do see a clever algorithm, that can be for example used with different number of input views. \n- The end-to-end framework is also quite interesting as it allows to be spread easily across the very large satellite images users, with the code aldready publicly available.\n- Lastly, the results are good wrt to the state-of-the-art, as the algorithm was proposed during a 2019 challenge and was in the top ones on the private leaderboard.\n\nRemarks and clarifications:\n- After looking at the challenge website, it is quite strange to find that the results values in Table 1, in terms of public and private scores, differ from the actual leaderboards, even if the metric is still the same. Thus, I was not able to understand (i) if the authors really participate to the challenge (ii) what method they presented during the challenge as 3 or 4 are shown here (iii) what was their ranking in the real leaderboard. It is important to be precise. From what I see, they might be second on the private score (the one that matters), if so it can be clarified also in the abstract where the word 'topped' was used.\n- cPSNR metric : (i) can you please explain the acronym; (ii) it was the competition metric, so it means it is not 'we use' but 'the challenge used' - which will also show that it was not a choice to satisfy your results, but a standard metric.\n- Lanczos interpolation: can you please explain in a small sentence in what this interpolation differs from the others?\n- ShiftNet: what is this network? We only know tha it is adapted from HomographyNet, but we don't have any information about how it is composed. We just know from App. 1 that it has a very large number of parameters (34M). Why is that? Why a rigid registration needs such a large number of parameters?\n- median anchor image: this is one of the interesting points of the paper. Can you please just clarify that you take a naive median image? Such as: for pixel (i,j), med(i,j)= med( LR1(i,j), LR2(i,j) ,...) .\n- You are saying 'each imageset is padded to 32 views'. What do you mean? I thought your network was able to be used with different numbers of views. How did you pad your imageset?\n- you list 'several baselines': for me, a baseline is something to compare to - usually, it is even something easy, such as the ESA baseline. But in your list, you are mixing baselines, other state-of-the-art methods (some of which you don't beat, so it's not a 'baseline'...), and your methods. It is very difficult to know which ones are yours. Please make different lists and/or identify yours in the table 1.\n- What is the Ensemble method? It appears that in the paper, you explain the 'HighRes-net + shiftNet' method, but actually, the 'Ensemble' is the better one, while not clearly described. I don't understand what outputs are averaged - I thought HighRes-net and shiftNet were performed simultaneously.\n- How did you select the hyperparameters of your model?\n\n\nScientific questions: \n- Is it possible to have views of different sizes as inputs? Or views with missing parts?\n- Usually, we have a high resolution image that is different in terms of type of acquisition (like a different type of satellite, but also true in medical images for ex.). Is your network ready for that? I mean: can we constrain the method even if it is not the same type of acquisition as ground truth? In that case, is that possible to have a super-resolution of the type of the input LR images?\n\nTypos: \n- is comprised of -> is composed of\n- in Table 7, the bold number should be the beta=infinity as it is the best one. It will be clearer, even if of course, a good train score does not mean a good method because of overfitting.\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}, {"review_id": "SJx-WbR0KB", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents a multi-frame super-resolution method applied to satellite imagery. It first estimates a reference image for the multiple input LR images by median filtering. Then it pairwise encodes the reference image and each of the multiple images in a recursive fashion then fuses the corresponding feature maps with residual blocks and bottleneck layers until only one feature maps for the entire multiple images obtained. In other words, LR images are fused into a single global encoding. Then, it applies a standard upsampling network to obtain the super-resolved image this image is fed into a network that estimates only the translational shift, and the shifted image with the estimated translation parameters finally resampled. \n\nA major concern is the estimation of a single translational motion for the SR image at the end of the network after all multiple images are already fused. The fusion strategy disregards the underlying spatially varying motion. This explicitly assumes the images are on a flat surface, which perhaps an acceptable assumption for high-orbit satellite imagery where the ground surface depth variances might be negligible. Still, this is a very critical limitation of the method. Besides, I am not convinced that pair-wise fusion can handle significant translational fusion as the filters have shared parameters. How a single convolutional layer accomplishes a global encoding and compensates for any translation between any LR image pair is neither articulated nor convincing discussion and evaluations are provided. Of course, such a problematic approach needs at least some kind of motion compensation, which may explain the need for the ShiftNet layer at the end. Nevertheless, this seems quite problematic. \n\nEven assuming the method only applies to satellite imagery, it lacks mechanisms to compensate/distinguish cloud coverage and atmospheric distortions. Characterization of satellite imagery noise models (Weibull, etc.) common in such imagery as a prior also completely disregarded. For these reasons, the proposed method fails to be considered as a comprehensive approach for multi-image super-resolution of satellite imagery. \n\nNovelty-wise, there is very little as all modules have been commonly used for SR tasks. ", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a multi-frame super-resolution method including recursive fusion for co-registration and registration loss to solve the problem where the super-resolution results and the high-resolution labels are not pixel-wise aligned. While reviewer #1 is positive about this paper, reviewer #2 and #3 rated weak reject and reject respectively. Both reviewer #2 and #3 have extensive experience in the topic of image super-resolution. The major concerns raised by the reviewers include the lack of many references, the comparison of recursive fusion with related work, limited test databases, using a single translational motion for the SR images, and limited novelty on the network modules.  The authors provided detailed response to the concerns, however they did not change the overall rating of the reviewers. While the ACs agree that this work has merits, given the various concerns raised by the reviewers, this paper can not be accepted at its current state.
Decision: Reject