Less is More: Rethinking Few-Shot Learning and Recurrent Neural Nets
Authors: Deborah Pereg
Martin Villiger
Brett Bouma
Polina Golland
Venue: Submitted to ICLR 2023 (2023)

Abstract: The statistical supervised learning framework assumes an input-output set with a joint probability distribution that is reliably represented by the training dataset. The learner is then required to output a prediction rule learned from the training dataset's input-output pairs. In this work, we provide meaningful insights into the asymptotic equipartition property (AEP) \citep{Shannon:1948} in the context of machine learning, and illuminate some of its potential ramifications for few-shot learning. We provide theoretical guarantees for reliable learning under the information-theoretic AEP, and for the generalization error with respect to the sample size. We then focus on a highly efficient recurrent neural net (RNN) framework and propose a reduced-entropy algorithm for few-shot learning. We also propose a mathematical intuition for the RNN as an approximation of a sparse coding solver. We verify the applicability, robustness, and computational efficiency of the proposed approach with image deblurring and optical coherence tomography (OCT) speckle suppression. Our experimental results demonstrate significant potential for improving learning models' sample efficiency, generalization, and time complexity, that can therefore be leveraged for practical real-time applications. 

Under review as a conference paper at ICLR 2023
LESS IS MORE: RETHINKING FEW-SHOT LEARNING AND RECURRENT NEURAL NETS
A Annnymous authors
Paper under double-bling review
ABSTRACT
The statistical supervised learning framework assumes an input-output set with a joint probability distribution that is reliably represented by the training dataset. The learner is then required to output a prediction rule learned from the train- ing dataset's input-output pairs. In this work, we provide meaningful insights into the asymptotic equipartition property (A EP) (Shannon 1948) in the context of machine learning, and illuminate some its potential ramifications for few- shot leamning We provide theoretical guarantees for reliable learning under the information theoretic AEP, and for the generalization enror with respect to the sample size. We then focus on a highly efficient recurrent neural net (RNN) frame- work and propose a reduced- reduced-entropy algorithm for few-shot leaming. We also pro- pose mathematical intuition for the RNN an approximation of a sparse coding solver. We verify the applicability, robustness, and computational efficiency of the proposed approach with image deblurring and optical coherence tomography (OCT) speckle suppression. Our experimental results demonstrate significant po- tential for improving learning models' sample efficiency, generalization, and time complexity, that can therefore be leveraged for practical real-time applications.
INTRODUCTION
In recent years, machine learning (ML) methods have led to many state-of the-art results, spanning through various fields of knowledge. Nevertheless, clear theoretical understanding of important aspects of artificial intelligence (Al) is still missing. Furthermore, there are many challenges con- cemiing the deployment and implementation of AI algorithms in practical applications, primarily due to highly extensive computational complexity and insufficient generalization. Concems have also been raised regarding the effects of energy consumption of training large scale deep learning systems (Strubell et all, 2020). Improving sample efficiency and generalization, and the integration of physical models into ML have been the center of attention and efforts of many in the industrial and academic research community. Over the years significant progress has been made in training large models. Nevertheless, it has not yet been clear what makes a representation good for complex leaming systems (Bottou et all 2007: Vincent. et al., 2008 Bengio. 2009 Zhang et all, 2021).
Main Contributions. In this work we investigate the theoretical and empirical possibilities of few shot learning and the use of RNNs as powerful platform given limited ground truth training data. (1) Based on the information- -atitiotnntehriiccllll. asymptotic equipartition property (A EP) [Cover & Thomas 2006), we show that there exists relatively small set that can empirically rep- resent the output data distribution for leaming. (2) In light of the theoretical analysis, we promote the use of compact RNN-based framework, to demonstrate the applicability and effi ciency for few-shot learning for natural image debluriing and optical coherence tomography (OCT) speckle suppression. We demonstrate the use of single image training dataset, that generalizes well, as an analogue to universal source coding with a known dictionary. The method may be appli- cable to other learning architectures as well as other applications where the signal can be processed locally, such as speech and audio, video, seismic imaging, MRI, ultrasound, natural language pro- cessing and more. Training of the proposed framework is extremely time efficient. Training takes about 1-30 seconds on a GPU workstation and few minutes on CPU workstation (2-4 minutes), and thus does not require expensive computational resources.
(3) We propose an upgraded RNN framework incorporating receptive field normalization (RFN) (see [Pereg et al. 2021), A Apendix C

---

Under review as a conference paper at ICLR 2023
that decreases the input data distribution entropy, and improves visual quality in noisy environments. (4) We illuminate a possible optimization mechanism behind RNNs. We observe that an RNN can be viewed as sparse solver starting from an initial condition based on the previous time step. The proposed interpretation can be viewed as an intuitive explanation for the mathematical functionality behind the popularity and success of RNNs in solving practical problems.
2 DATA-GENERATION MODEL- A DIFFERENT PERSPECTIVE
Typically, in statistical learning Shalev-Shwartz & Dan-David 2014 Vapnik, 1999), it is assumed that the instances of the training data are generated by some probability distribution. For example, we can assume a training input set y = {(y; i= 1 V; ~ Py }, such that there is some correct target output x, unknown to the leamer, and each pair (y,, x;) in the training data is generated by first sampling a point y, according to Pr(,) and then labeling it. The examples in the training set are randomly chosen and, hence, independently and identically distributed (i.id.)) according to the distribution Py (.). We have access to the training enrrr (also referred to as empirical risk), which we normally try to minimize. The known phenomenon of overfitting is when the learning system fits perfectly to the training set and fails to generalize. In classification problems, probably approximately correct (PAC) learning defines the minimal size of a training set required to guarantee PAC solution. The sample complexity is function of the accuracy of the labels and a confidence parameter. It also depends on properties of the hypothesis class. To describe generalization we normally differentiate between the empirical risk (training enror) and the true risk. It is known that the curse of dimensionality renders learning in high dimension to always amount to extrapolation (out-of-sampp -aal-ll--appeeeeeeee extension) (Balestriero et all 2021).
INFORMATION THEORETIC PERSPECTIVE: CONNECTION TO AEP
Hereafter, we use the notation x" to denote a sequence Iι, 22 .... Tn. In information theory, a stationary stochastic process u" taking values in some finite alphabet 2 is called a source. In communication theory we often refer to discrete memoryless sources (DMS) (Kramer et al., 2208 Cover & Thomas, 2006). However, many signals, such as image patches, are usually modeled as entities belonging to some probability distribution forming statistical dependencies (e.g., a Markov- Random- Field (MRF) (Roth & Black 2009 Weiss Freeman, 2007)) describing the relations be- tween data points in close spatial or tem poral proximity. Here, we will briefly summarize the A AEP for ergodic sources with memory ussin, 2017). Although the formal definition of eryodic pro- cess is somewhat involved, the general idea is simple. In an ergodic process, every sequence that is produced by the process is the same in statistical properties (Shannon 1948). The symbol fre- quencies obtained from particular sequences will approach definite statistical limit as the length of the sequence is increased. More formally, we assume an ergodic source with memory that emits n symbols from a discrete and finite alphabet Ut with probability Pr (u1, U2, ...,, un). We recall a theorem (Breiman, 1957), here without proof.
Theorem 1 (Entropy and Ergodic Theory). Let U2, ..., un be a stationary ergodic process ranging over a finite alphabet U, then there is constant H such that
H =
lim logz (u,,u....,..)) n-000
H is the entropy rate of the source.
Intuitively, when we observe a source with memory over several time units, the uncertainty grows more slowly as n grows, because once we know the previous sources entries, the dependencies reduce the overall conditional uncertainty. The entropy rate H, which represents the average un- certainty per time unit, converges over time. This, of course, makes sense, as it is known that M(X,Y) < HT(X) - -YY)) In other words, the uncertainty of a joint event is less than or equal to the sum of the individual uncertainties. The generalization of the A EP to arbitrary ergodic sources is as following (McMillar 1953).
Theorem (Shannon McMillan (AEP)). For e > 0, the typical set A with respect to the ergodic process Pt (u) is the set of sequences (u1, U2, u2,,.., un)) L obeying
1. Pr(u E AT] > - e, for n sufficiently large.
2. 2 ++++ < (u) 22-
2

---

Under review as a conference paper at ICLR 2023
3. |A7]
= 2nd
|A| denotes the number of elements in the set A, and Pr[A] denotes the probability of the event A. In other words, if we draw a random sequence (uz, u2, .... N,,), the typical set has probability nearly 1, all elements of the typical set are nearly equally probable, and the number of elements of the typical set is nearly 2nd This property is called the asy asymptotic equipartition property (A EP). In information theory the A EP is the analog of the law of large numbers (Cover & Thomas 2006). The notion of a typical sequence was first introduced in 1948 by Shannon in his paper "A Mathematical Theory of Communication" (Shannon, 1948). Intuitively, the typical sequences u" are the sequences whose empirical probability distribution is close to (.).
As mentioned, the entropy rate is more often used for discrete memoryless sources (DMS), yet "every ergodic source has the AEP" (McMillan, 1953). Note that entropy typicality applies also to continuous random variables with a density Py Pe replacing the discrete probability (r((")) with the density value pu (u"). The APP leads to Shannon's s source coding theorem stating that the average number of bits required to specify symbol in sequence of length n, when we consider only the most probable sequences, is H. And it is the foundation for the known rate- nnte-distortion theory and channel capacity.
The AEP property divides the space of all possible sequences into two sets: the typical set, where the sample entropy is close to the true entropy, and the non-typical set that consists of the other sequences. We would like to show that most of our attention should be on the typical sequences, because any property that is true for the typical sequences will then be true with high probability and will determine the average behavior of a large sample.
Let us assume a training set V = {yi, x; } where x; E Rnx1 are sampled from A (Px) and paired with y, E Rdxi by some function as ground truth. The learning system is trained to output a prediction rule F yrd + X². As- sume an algorithm that trains the predictor by minimizing the training enrrr (empirical emor or empirical risk). A ssuming a discrete source (x(() that emits i.id sequences x" of symbols (for example: patches in an image, segments of an audio signal, etc.), the estimator has access only to the observed signal و² In an inverse problem yd would be a degraded signal originating in x", where the relationship between x. and 3 could be linear or non-linean linear, with or without additive noise, such that generally s² = 9xx") + e(z"), where g(-) and e(-) are functions of ,² Given 1/a we produce an estimate d". However, the following proofs are not restricted to this framework. Our problem setting is illustrated in Fig. [ For the sake of the following theoretical analysis we restrict the mapping J : y'd + X" to be a surjective func- tion. Namely, for every x", there is yd such that f(y²) = I". In other words, every element x" is the image of at least one element of y It is not required that 3² be unique. In the presence of noise this condition can only be met if the noise's 's power is under a certain threshold. The goal is to prove that leamnigg with A is sufficient for generalization of the entire distribution with the same gener- alization error. Denote sample size V m that is required to train predictor Fs Ddd A"", such that V ~ Px... An algorithm minimizes the training error (empirical error or empirical risk)
Learning System
encoder
decoder
Px(.)
Pri(()
Measured Data
Latent
Space
Figure 1: The learning system problem.
1 Σ ((Fo (y:,,
(F(Fo)= =
(1)
where 0 < x) < is some loss function. The empirical error over the training set at the end of the training, for the specific trained predictor hy is L (hrv) I Am << 1. The true error, or the generalization error, in this setting is
[(hv) = E(E.)) (hv
(2)
where F((x.)))PP.v (.) denotes the expectation over Px,.
Theorem (AEP leamning for systems with surjective mapping). ssume the generalization error of the trained predictor F )²d + x over the typical set A (Px) is at most oA. Then, the
generalization emor of the trained predictor F: ydd x over the entire distribution is at most.
3

---

Under review as a conference paper at ICLR 2023
Theorem 4 (sample size for leaming with surjective mapping). Assum a training set that is generated by randomly drawing samples from Px and labeling them by the target function g()),
V = { (yi, xi )=)="111 x; ~ Px: y; = g(x),, x; E R y, E Rdx1), }. g(-) is a deterministic function, and f(.) = g (-) a surjective function. ssume the predictor was trained successfully
to yield Cu (hw) I Am << 1. The sample size and the true error obey
((hv)= & Ams
m 2 2YH
((he) S A
2dH
1 -
A
m 24H()
Am SASI.
(3)
Am
In other words, as long as the sample size is larger or equal to the typical set size, we are guaranteed to have a generalization entor that is as small as the training ertor. Otherwise, the upper bound on the generalization error depends on the ratio between the sample size and the input typical set size. Theorem (sample size for learning with a surjective mapping in noisy environment). Asume a
training set that is generated by randomly drawing samples from Px and labeling them by the target function such that, V = {yy, x; M==1 X; ~ Px, y, = g(x;) + n, x; E [ Yi E R ). where g()) is a deterministic known or unknown function, and n, is an additive i.idd noise n, ~ N a.).. The sample size and the true enror obey
[(hv) S Am: m 2 2nl(Xit)
[(hv) S A,
1 1- Sm < 2
1 1- A
Am S SASI,
(4)
where ((X;Y)=H[) - - H(X)Y) is the mutual information between X and Y. Note that given EY² < of + o² we know ((XY)) < log(1 + oo./2))
In this case, we can resolve an input yf as x. they are jointly typical. For each (typical) output sequence x",, there are approximately (x possible y" sequences, all of them equally likely. We assume that no two x" sequences produce the same y output sequence, (otherwise, we will not be able to decide which x" sequence it originated from). Hence, the total number of possible (typical) yd sequences is approx 2n This set has to be divided into sets of size 2Y1()) corresponding to the different x sequences. The total number of disjoint sets is less than or equal to - ( Hence, we can have at most 2 distinguishable sequences of length n. The proofs for Theorems 3 can be found in A ppendix H
Remark 1. Theorem 4 is specific case of theorem 5
since in our setting, in the absence of noise
T(X; Y)= = HY)
Remark 2. The theorems and their proofs can be generalized to the continuous case with A (px) and differential entropy ((px).
Remark 3. is often possible to assume that X' and y are discrete alphabets as result of quantiza- tion of real values, and therefore we can discuss discrete entropy.
Remark 4. The size of the typical set is exponential by n. Therefore, one could claim the size of typical training set is still large relative to standard training sets that represent the entire probability distribution. However, it is known that data compression and source coding are based on the A E... And though the size of the typical set is exponential by n, compression algorithms, such as JPEG (Wallace, 1991), can compress an image by a factor of 10. In the noiseless case, there are 2 log, possible output sequences, where r = [ Using the A EP it should be possible to train with signif- icantly less examples, because we only try to generalize over 2Y) possible output sequences. Remark 5: Zontak & Irani (201)) postulate that patches of the same images are internally repeated, but unlikely to be found in other images. This notion might appear as a contradiction to the proposed analysis. However, note that although the probability of finding a sequence that belongs to AT is close to one, the probability of finding specific sequence in the typical set is very small.
The AEP tells us that there exists a relatively small group of training examples that would be suffi- cient for generalization. However, the AEP property does not define this set, nor the correct coding, leaming or prediction method. It just reassures us that there exists a set of the sort How do we find the typical learning set? One option may be by predefined or leamed dictionary coding: build a training set that represents the typical set, consisting of the most common structures, in a similar manner to universal source coding based on known dictionary (Cover & Thomas 2006).
4

---

Under review as a conference paper at ICLR 2023
3 PRELIMINARIES
3.1 RNN FRAMEWORK
ssume an observation data sequence y (I-, Ys E RNxI, E - IJ, and a corresponding output sequence x [xo, (Xo,X11.... XL-1,, , X E Rxxx, The RNN forms a map f yy + z, from the input data to the latent space variables. That is, for input yr and state z, of time step t, the RNN output is generally formulated as (o) PPacaan et al, 2013).
Hereafter, we will focus on the specific parametrization:
Z o (W +
b),
(5)
where a is an activation function, W zy E RNNx and W sy E R E IR x1 is the bias vector, assuming nu number of neurons in an RNN cell. Att = 0 previous outputs are zero. Here, we use the ReLU activation function, RcLU(=) max (0, =}. At this stage, we wrap the cell with a fully connected layer such that the desired final output is x, = FC(z)).
x are weight matrices and
Traditionally, RNNs are used for processing of time related signals, to predict future outcomes, and for natural language processing tasks such as handwriting recognition (Graves et all, 2009) and speech recognition (Graves et all 2013). In computer vision RNNs are less popular, due to gradient exploding and gradient vanishing issues (Pascanu et aL 2013), and their expensive computational complexity compared to CNNs. Liang & Hu (2015) proposed the use of recurrett convolutional networks (RCNNs) for object recogmition. Pixel- RNN (Van Den Oord et all, (016) sequentially pre- dicts pixels in an image along the two spatial dimensions.
3.2 SPARSE CODING & ITERATIVE SHRINKAGE A ALGORIHHSS
In sparse coding (SC) signal y IR is modeled sparse superposition of feature vectors (Elad, (2010: Chen et all 2001). Formally, the observation signal obeys Dz, where D E R N'x M .... M, and z IR M is sparse vector of the atoms weights. Over the years, many efforts have been invested in sparse coding, both in a noise free environment, or when allowing some error,
dictionary of MM atoms d, E IR Nx1
min ||z||2
s.t.
Ily Dzlla S
(6)
where ||z||a - Σ las ||z||2 on sparse coding are in Appendix A
2? and is the residual noise or error energy. Further details
Consider the cost function,
f(z) = 2 ly Dz + Al|zl11
for some scalar A. Following Majorization Minimization (MM) strategy, we can build a surrogate function (Daubechies et all 2004 Elad 2010)
(zz 20) = fz) + d(z, zn)
1
ly - Dz112 ||ll|| +
2 - z
1 Dzollă.
Dz
(8)
2
2
The parameter c is chosen such that the added expression
1 2/
d(z, 24)= = Q(z, 2n) f(z) = -11 20
Dz - Dz
(9)
2
D'D O. Therefore e > [[D' DD12 = amar (D" D), i.e., greater than the largest eigenvalue of the coherence matrix D''D. The term d(z, 2, 2o) is measure of proximity a previous solution zg. If the vector difference z 2 is spanned by D, the distance drops to nearly zero. A Alternatively, if D is not full rank and the change z 24 close to the null space of D, the distance is simply the approximate Euclidean distance between the current solution to the previous one. The sequence of iterative solutions minimizing Q(z, 2e) instead of fz), is generated by the recurrent formula Z = arg min, Q(z, 2,) where 8 E I is the iteration index. We can find a closed-fome solution for its global minimizer that can be intuitively viewed as an iterative projection the dictionary on the residual term, starting from the initial solution that is a thresholded projection of the dictionary on the observation signal (Z. 0):
is strictly convex, requiring its Hessian to be positive definite,
5

---

Under review as a conference paper at ICLR 2023
20+1 S (- DT (y- D2o) + +20) = (-DTy+ + +0+(1-1
(10)
where the Sz(2) = (|21 - 3) sgn(2) 2) is the soft threshold operator. Assuming the constant is large enough, it was shown in Daubechies al. (2004), that (10) is guaranteed to converge to its
global minimum. Over time, faster ex extesionns have been suggested, such as: Fast-ISTA (FISTA)
(Beck & Teboulle, 2009a), and Leamed-ISTA (LISTA) (Gregor & LeCun, 2010),
Ze+1 = S2 (Wy + Sza).
(11)
W and S are learned over set of training samples {y,, J w and S re-parametrize the matrices !DT and (I- DDDD), respectively. Note that W and S are no longer constrained by D.
4 RNN ANALYZED VIA SPARSE CODING
Observing the similar structure of 5 and we redefine the cost function 7
f(z,) 2 lly, -rzelll + Al|zel1
(12)
Now, building surrogate function
QZz 111 = f(z:) + d(zz. z(-1)
2 1ly: - Dzy +
21-1
Dz.
Dz 11
2
(13)
where the added term d(z,, 1) represents the distance between the current solution and the pre- vious solution at the preceding time step (rather than the previous iteration), yields
z, Sy D ( Dz.-1) + 21-1
(14)
which its learned version can be re- parametrized as,
2. = S3 ( ve + W -.
(15)
Clearly, 15 is equivalent to 5). In other words, RNN can be viewed as unfolding of one iteration
of learned sparse coder, based on an assumption that the solution at time is close to the solution in time 1. In subsection 3.1 the RNN state encodes the vector latent space. Then the FC net decodes the latent variable back to a space of the required dimensions. Given this interpretation, it may be claimed that RNN's use should not be restricted data with obvious time or depth relations. The RNN merely serves as an encoder providing a rough estimation of the sparse code of the input data. The RNN's memory serves the optimization process by starting the computation from a closer solution. Thus placing the initial solution in a "close neighborhood" or close proximity, and help- ing the optimization gravitate more easily towards the latent space sparse approximation. Clearly, convergence is not guaranteed.
5
FEW SHOT LEARNING VIA RNN
The setting described in this subsection has been previously employed for various applications. Biswas et all (2018) and Pereg et all (2020a) used a similar framework to facilitate automatic ve- locity analysis, where they used portion of the acquired data for training and let the system infer the rest of the missing velocities. [ereg et all 2020b) used a similar framework to perform seis- mic inversion with synthetic training data. Here, we expand and elaborate its application, while connecting it to the theoretical intuition in Section 2 The description below is formulated for two- dimensional (2D) input signals, but can be easily adapted to other input data dimensions. We use similar definitions and notations as previously described in (Pereg et al 2020at
Definition 1 (Analysis Patch): We define an analysis patch as a 2D patch of size L. x N enclosing L, time (depth) samples of N consecutive neighboring columns of the observed image S RL.X
ssume {nI., 7n N nL + AR N 1}. The analysis patch A (6.)) associated with an image point at location (i. f), such that element (A. 1) of A (6J) is
A (6.])
(Sitkjtt k, E Z, L O. TR 21 515 nn).

---

Under review as a conference paper at ICLR 2023
An analysis patch A (1-)) E R R N' is associated with pixel R,,, in the output image. To produce a point in the estimated R.,,, we set an input to the RNN as y A (6-) Each time step input is group of N neighboring pixels of the same corresponding time (depth). In other words, in our application yt = 15--L,+14,,---- ..... L,
1", We set the size of the output vector X2 to one expected pixel (P = 1), such that x is expected to be the corresponding segment, x = (f) R Lastly, we the first L, - 1 values of the output x and set the predicted pixel R. as the last one, LL,, The analysis patch moves through the imput image and produces all expected output points in the same manner. Each analysis patch and a corresponding output vector are an instance for the net. The size and shape of the analysis patch defines the geometrical distribution of columns and samples that are considered in each output point computation.
Note that, in the above framework the number of hidden units is typically much larger than the dimension of the input, nh >> N, (N- << M). The dimension of the latent space can be 3 orders of magnitude larger than the input space dimension. In accordance with that, it has been observed that over caatovee-pameee parametrized neural networks generalize better (Neyshabur et all 2018)) and that dictionary recovery may be facilitated with over rea-reeli models (Sulam et al. 2022).
Employing one projection assuming x is close to x; may throw us far away from the desired solution. RFN (Pereg et al. (2021) refers to local signal nomalization,, in an attempt to find the detected features' support in fewer iterations, by avoiding the liability of unbalanced energy. Since the RNN is essentially sparse solver, we propose to apply 2D-RFN, prior to applying the RNN. We then multiply the latent space support normalized weights by the non- normalized projection of the learned dictionary on the original input signal to regain the local energy. A ppendix C describes 2D-RFN, and provides the proposed method's forward pass in A Algorithm []] The RFN signal's ss distribution is confined to smaller set of values. Thus, RFN decreases the signal's 's entropy and the size of the typical set associated with the input distribution.
RFN-RNN.
6 NUMERICAL EXPERIMENTS
6.1 CASE STUDY: IMAGE DEBLURRING
We studied the RNN framework in Section and A AlgorihhmI(RNN--RNY for Gaussian debluring. First, we follow the same Gaussian deblurring experiment performed in Romano et all (2017) 1), using the set of natural images provided by the authors. All pixel values are in the range of [0,255] The images were convolved with a 2D Gaussian PSF 25 x 25 with standard deviation of 1.6. RGB images are converted to Y CbCr space. Inversion is applied to the luminance channel, and the result is converted back to the RGB domain. As a figure of merit, we used the peak signal to noise ratio (PSNR) and the SSIM, both computed on the estimated luminance channel of the ground truth and the estimated image. We investigated the basic RNN and RNN-RFN frameworks, and studied the effect of different training and testing images, with additive WGN with 0₂ = 22.
For each experiment, we used one of the images for training and the remaining 9 images for testing. The assumption that the latent space representation is sparse was verified in the numerical experi- ments.
the optimal PSNR and SSIM scores obtained for each image, and compares the PSNR scores to state-o of- the art image debluming methods: FISTA (Beck & Teboulle, 2009h),, NCSR (Dong et al. 2013) and RED (Romano et all, 2017 7). Note that most deblurring methods re- quire prior wedge of the degradation process. W Whereas the proposed approach requires only one example of the degraded image and its corresponding ground truth. The best scores were obtained by training with either one of the images: butterfly, boats, parrot, starfish and peppers. Hence, it is safe to assume that the patches in these images reliably represent the typical set associated with natural images to a certain degree. Figures 243 present examples of the proposed methods. Fig- ure the evolution of the PSNR scores for RNN and ENN-REN with increasing additive noise variance o E [0, 10 As the noise level is increased RFN suppresses noise better. Aditionna examples and training details are in Appendi [D.]
At this stage, we are not suggesting that the results are competitive with state- of denois- ers. Currently, the main advantage is substantial speed. Tensorflow bada et all, (016) imple- mentation converges in about 1700 iterations in an average of 14.29 seconds on a laptop GPU (NVIDIA GeForce GTX Ti 1650 with 4GB video memory) and 2.01 minutes on i-7 CPU. Pytorch
7

---

Under review as a conference paper at ICLR 2023
(a)
(b)
(c)
(d)
(e)
Figure 2: Visual comparison of deblurring of the image starfish: (a) Ground truth; (b) input, 24.8dB; (c)
RED-SD, PSNR=32.42dBB (d) RNN 29. (e) RNN-NN-RR RFN, 28.45dB.
(c)
(d)
(e)
Figure 3: Visual comparison of deblurring of the parrot image: (a) Ground truth; (b) input, 25.33dB; (c) RED-SD, PSNR==3.18dBB (d) RNN 30.10dB) (e) RNN-RFN, 29.64dB;
Paszke all (001) implementation training converges in 40 epochs on average in 30.73 seconds on GPU NVIDIA GeForce RTX 2080 Ti, and 2.73 minutes on i-7 CPU. Inference takes ~50 msec. For reference, RED-SD takes 15-20 minutes of processing for each image, on CPU.
As the focus of this study is to bridge between theory and application for few-shot learning, and results, verified the
to provide theoretical justifications, rather than to achieve state-of-tieg theoretical results in Theorem 5 for the image deblurring case. To this end, we trained the RNN network with varying sample sizes while trying to keep constant training error Ly (hv) S A.- The training data consists of patches belonging to a single image (starfish). We consider the recovery error, denoted as E(hy), over the other non- ton-training 9 images as an empirical approximation for the generalization error. Figure shows the evolution of the recovery emrrr with increasing sample size chosen uniformly in the range [1, 61 61752]. Evidently, L(he) ) 2 Lo (h+), while L (he) S A.. which implies that < (x:Y), In this region C(he) < 1 m2 ( A.) A (see proof). Hence, empirially we can deduce (X; Y) 2 0.33. Our code will be available upon acceptance.
6.2
CASE STUDY: OCT SPECKLE SUPPRESSION
OCT uses low coherence interferometry to produce cross- posss-ectional tomographic images of internal structure of biological tissue. It is routinely used for diagnostic imaging of the anterior eye, the retina, and, through fiber-optic catheter, the coronary arteries. Unfortunately, OCT images are degraded by the presence of speckle (Schmitt et al. 1999 Goodmar 2007), which appear as grain- like structures in the image, with size equivalent to the nominal spatial resolution of the OCT system. Speckle significantly degrade images and complicate interpretation and medical diagnosis by confounding tissue anatomy and masking changes in tissue scattering properties. Though speckle share many statistical properties with noise, it is essentially unresolved spatial information as a result of the interference of many sub- reboluootttion spaced scatterers (Curatolo et all 2201)..
Here, we show two examples of supervised learning speckle suppression with OCT experimental data, for demonstration. We used intensity OCT tomograms displaying the ocg-scaled squared norm of the complex-valuee tomogram. As ground truth for training and for evaluation of the general- ization, we used hardware- based speckle mitigation obtained by dense angular compounding, in a method similar to Desjardins et all (2007). We investigated two challenging cases of mismatch between the training data and the testing data: (1) Tissue type mismatch.
(2) Tissue type and ac- quisition system mismatch. Namely, in the first case, OCT tomograms of chicken tissue and a blueberry were acquired by the same system and used as training image and testing image, respec- tively. W heereas, in the second case, the training image was tomogram of chicken-skin acquired by one system, and the learning performance was tested on a tomogram of cucumber imaged with a different OCT system. Further details are in A ppendix D.2

---

Under review as a conference paper at ICLR 2023
(a)
(b)
(c)
(d)
(e)
Figure 4: OCT speckle suppression with tissue mismatch and identical imaging system: Training images: (a) Chicken muscle with speckle (PSNR 26.03 dB); (b) Ground truth, speckle-suppressed tomogram; (c)
B Blueberry with (PSNR = 21.39 dB) and (d) without speckle (ground truth); (e) Predicted speckle- suppressed tomogram (PSNR=31. IdB). Scale bars: 200
(a)
(b)
(c)
(d)
(e)
Figure 5: OCT speckle suppression with tissue and system mismatch: Training images: (a) Chicken skin with speckle (PSNR = 28.96 dB); (b) Ground truth, speckle-suppree tomogram; (c) Cucumber with (PSNR = 29.40 dB) and (d) without speckle (ground truth), imaged with inferior lateral resolution and lower lateral sam- pling than training image; (e) Predicted speckle-uuppressed tomogram (PSNR=31) 31.34dB). Scale bars: 200pm..
Figures 4 ()) show the ppeckle-corrppter and speckle- suueleeseppeee tomograms of chicken tissue used for training. The speckle- suppeessed tomogram in Fig. [bbb has homogeneous appearance, whereas the speckled signal in Fig. (a) is strongly textured, exhibiting a grainy pattern with spots of bright, high intensity signal, altemaatin with regions of very low signal. Figures [igures[Ac)-((( (c)-(d) show the speckle- comupted and speckle suppresspppp tomograms of the blueberry used for testing. As shown in the inferred result in Fig. e), the system is able to significantly reduce speckle while maintaining spatial resolution, despite the structural differences between the training and the testing images.
Figures 5 (bb show the original and compounded tomograms of chicken skin used for training. Visibly the training data non-stationary and of complex structure. Figures(5]c)- c)-(() show the orig- inal and compounded tomograms of cucumber used for testing. The cucumber image was acquired with a different sample objective lens resulting in twice poorer lateral resolution. The lateral sam- pling space was increased from 2.5 to 8ymm, resulting in system mismatch between training and testing data, in addition to the tissue mismatch. As can be seen, the learning system able to de- tect most features and to significantly remove speckle. Horizontal lines appear slightly sharper than vertical ones.
Computational resources. The proposed framework training time is only 35.00 seconds stan- dard CPU workstation equipped with an Intel(R) Core(TM) i7 -7820HQ CPU @ 2.30 GHz, or alternatively 4.56 seconds on average on a laptop GPU NVIDIA GeForce GTX Ti 1650 with 4GB video memory, and converyes typically in less than 700 iterations.
CONCLUSIONS
"The curse of dimensionality can be viewed either as the limitation on data analysis due to the large amount of data or parameters needed to analyze the data" (Bengio et all, (207)). In this work, we investigated the properties of the AEP in the context of supervised learning, and its practical ramifications in relieving computational complexity in learning systems. We focused on a RNN framework as a powerful tool for learning with limited training data. The low computational com- plexity can be leveraged for medical areas where there is lack of data and resources. Future work could extend the proposed work to other tasks and applications in other domains. Our future work will extend the A A.P hypothesis to the automatic design of the A EP training dataset for unsupervised medical imaging learning tasks.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Martín badi, Paul Barham, Jianmin Chen, Zhifeng Chen, A nny Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, and et al. Tensorflow: A system for large- scale machine learning. In Symposium on Operating Systems Design and Implementation,
pp. 265-283, 2016.
Tim
and ergodic theory, https://www http://www..at.t....mm.mmm. uc www.hh....a....aaa.a.coooo timm wwt./iittttppppee rntropy 21.
Austin.
Entropy
2017.
URL
Randall Balestriero, Jerome
Pesenti, and Yann LeCun. Learning in high dimension always amounts to extrapolation. arXiv preprint arXiv: 00.09485, 2021.
mir Beck and Marc Teboulle. fast iterative shrinkage- thresholding algorithm for linear inverse problems. SIAM J. Img. Sci., 2(1):183- 202, March 2009a.
A mir Beck and Marc Teboulle. Fast gradient based algorithms for constrained total variation image denoising and deblurring problems. IEEE transactions on image processing, 18(11):2419-2433 2009b.
T. Bendory, S. Dekel A. Bar-Zion, and A. Feuer. Stable support recovery of stream of pulses with application to ultrasound imaging. IEEE Transaction on Signal Processing, 64(14):3750 3759, 2016.
Yoshua Bengio. Learning deep architectures for AI. Found. Trends Mach. Learn., 2(1), 2009.
Yoshua Bengio, Yann LeCun, et al. Scaling learning algorithms towards AI. Large-scale kernel machines, 34(5):1-41, 2007.
Reetam Biswas, Anthony Vassiliou, Rodney Strombery, and Mrinal K. Sen. Stacking velocity esti- mation using recurrent neural network. In SEG Technical Program Expanded Abstracts 2018, pp. 2241-2245, 2018.
Léon Bottou, Olivier Chapelle, Dennis DeCoste, and Jason Weston, Scaling Learning Algorithms toward AI, pp. 321-359. 2007.
Leo Breiman. The individual ergodic theorem of information theory. The Annals of Mathematical Statistics, 28(3):809-88 -11, 1957.
Emmanuel J. Candès, Yonina C. Eldar, Deanna Needell, and Paige Randall. Compressed sensing with coherent and redundant dictionaries. Applied and Computational Harmonic Analysis, 31()): 59 - 73, 2011.
S. S. Chen, D. L. Donoho, and M. A. Saunders. A tomic decomposition by basis pursuit. SIAM Rev., 43(1):129- 159, 2001.
Thomas M. Cover and Joy A. Thomas. Elements of Information Theory (Wiley Series Telecom- munications and Signal Processing). VilleyIIntesscien USA, 2006.
Andrea Curatolo, Brendan F Kennedy, David D Sampson, and TR Hillman. Speckle in optical coherence tomography. Taylor & Francis, 2013.
I. Daubechies, M. Defrise, and C. De Mol. An iterative thresholding algorithm for linear inverse problems with a sparsity constraint. Communications on Pure and Applied Mathematics, 57(11): 1413-143 1457, 2004.
A. E. Desjardins, B. Vakoc, W. Y. Oh, S. M. R. Motaghiannezam, G. J. Tearney, and B. E. Bouma. Angle- resolved optical coherence tomography with sequential angular selectivity for speckle re- duction. pt Express, 15(10):6200- 6209, May 2007.
Weisheng Dong, Lei Zhang, Guangming Shi, and X Li. Nonlocally centralized sparse representa- tion for image restoration, IEEE Transactions on Image Processing, 22(4):1620-163) 2013.
D.L. Donoho. Compressed sensing. JEEE Transactions on Information Theory, 52(():1229- 2006.
10

---

Under review as a conference paper at ICLR 2023
M. Elad. Sparse and Redundant Representations. Springer, 2010.
urélien Géron. Hands-on on machine learning with Scikit- Learn and TensorFlow concepts, tools, and techniques to build intelligent systems. O' O'Reilly Media, 2017.
J.W. Goodman. Speckle phenomena in optics: theory and applications. Englewood, Colo, Roberts & Co, 2007.
Alex Graves, Marcus Liwicki, Santiago Fernández, Roman Bertolami, Horst Bunke, and Jürgen Schmidhuber. novel connectionist system for unconstrained handwriting recognition. IEEE Transactions Pattern Analysis and Machine Intelligence, 31(5):855- 168, 2009. doi: 10.1109/ TPAM1.2008.137
Alex Graves, A Abdel-rahman Mohamed, and Geoffrey Hinton. Speech recognition with deep recur- rent neural networks. In 2013 IEEE International Conference Acoustics, Speech and Signal Processing, pp. 6645-6649, 01 2013.
Robert M Gray. Entropy and information theory. Springer Science & Business Media, 2011.
Karol Gregorand Yann LeCun. Learning fast approximmaios of sparse coding. In Proceedings of the 27th International Conference on International Conference on Machine Learning, pp. 399-406. Omnipress, 2010.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-temy memory. Neural Computation, 9(8): 1735- 1780, 1997.
K. Jamett, K. Kavukouoglu, M. Ranzato, and Y. LeCun. What is the best multi- stage architecture for object recognition? 2009 IEEE 12th International Conference on Computer Vision, pp. 2146- 2153, 2009.
Gerhard Kramer et al. Topics in multi-user information theory. Foundations and Trends in Commu- nications and Information Theory, 4(4-5):263 444,4444,44 2008
Honglak Lee, Roger Grosse, Rajesh Ranganath, and Andrew Y. Ng. Convolutional Deep Belief Net- works for Scalable Unsupervised Learning of Hierarchical Representations, pp. 609-616. ACM New York, 2009.
Ming Liang and Xiaolin Hu. Recurrent convolutional neural network for object recognition. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3367-3375, 2015.
rerogiiittocoooottoo In 2015
Brockway McMillan, The basic theorems of information theory. The Annals of mathematical statis- tics, pp. 196- 219, 1953.
B. K. Natarajan. Sparse approximate solutions linear systems. SIAM Comput., 24(2):227-234, 1995.
Behnam Neyshabur, Zhiyuan Li, Srinadh Bhojanapalli, Yann LeCun, and Nathan Srebro. To- wards understanding the role of parametaratiiiiiooooon in generalization of neural networks. CoRR, abs/1805.12076, 2018. URL
12076
Bruno Olshausen and David Field. Emergence simple-cet cell receptive field properties by learning sparse code for natural images. Nature, 381:607-9, 07 1996.
Vardan Papyan, Yaniv Romano, and Michael Elad. Convolutional neural networks analyzed via convolutional sparse coding. Mach. Learn. Res., 18(1):2887-2933 January 2017. ISSN 1532- 4435.
Razvan Pascanu, Tomas Mikolov, and Yoshua Bengio. On the difficulty of training recurrent neural networks. In Proceedings of the 30th International Conference on Machine Learning, volume 28 of Proceedings of Machine Learning Research, pp. 1310-1311 1318, tlanta, Georgia, USA, 17-19 Jun 2013.
11

---

Under review as a conference paper at ICLR 2023
A dam Paszke, Sam Gross, Francisco Massa, A dam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca A ntiga, A Abaan Desmaison, A Adreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-pent nigh-perfomance deep leaming library. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d AlchBuc, E. Fox, and R. Gamett (eds.), Advances in Neural Information Processing Systems 32, pp. 8024-8035. 2019.
Deborah Pereg and Israel Cohen. Seismic signal recovery based on earth Q model. Signal Process- ing, 137:373 386, 2017. doi: http:/dx..di.ong/// http:///......20000000
Deborah Pereg, Israel Cohen, and A nthony A Vassiliou. Multichannel sparse spike inversion. Jour- nal of Geophssics and Engineering, (4(5):1290-111 09 2017.
Deborah Pereg, Israel Cohen, and A nthony A. Vassiliou. Three-dimension dimensiona sparse seismic decon- volution based on earth Q model. Signal P Processing, 154:97 - 107, 2019.
Deborah Pereg, Israel Cohen, and Anthony A. Vassiliou. Sparse seismic deconvolution via recurrent neural network. Journal of Applied Geophysics, 175:103979, 2020a. ISSN 0926-9851.
Deborah Pereg, Israel Cohen, A nthony A. Vassiliou, and Rod Stromberg. Prestack time migration velocity analysis using recurrent neural networks. Journal of Applied Geophysics, 181:104160, 2020b. ISSN 0926-9851
Deborah Pereg, Israel Cohen, and A Anhhony A. Vassiliou. Convolutional sparse coding fast approxi- mation with application to seismic reflectivity estimation. IEEE Transactions on Geoscience and Remote Sensing, pp. 1-19, 2021. doi: 10.1109/TGRS 20t...2221.11500
Yaniv Romano, Michael Elad, and Peyman Milanfar. The little engine that could: Regularization by denoising (red). SIAM Journal on Imaging Sciences, 10(4):1804- 1844, 2017.
Stefan Roth and Michael J Black. Fields of experts. International Journal of Computer Vision, 82 (2):205-229, 2009.
Joseph M. Schmitt, S. H. Xiang, and Kin Man Yung. Speckle in optical coherence tomography. Journal of Biomedical Optics, 4, 1999.
Shai Mhalev-Shwartz and Shai Dev-David. Understanding Machine Learning From Theory to Algorithms. Cambridge University Press, 2014.
Claude Elwood Shannon. A mathematical theory of communication. The Bell System Technical Journal, 27:379-423, 1948.
Emma Strubell, A nanya Ganesh, and A ndrew McCallum, Energy and policy considerations for modern deep learning research. Proceedings of the AAAI Conference on Artificial Intelligence, 34(09):13693- 13696, Apr. 2020.
Jeremias Sulam, Chong You, and Zhihui Zhu. Recovery and generalization in over-realized dictionary leaming. Journal of Machine Learning Research, 23(135):1-23, 2022. URL
hhttp://jmlr.. http::///mml.ooorpppprss////////
1360. html
Aäron Van Den Oord, Nal Kalchbrenner, and Koray Kavukcuoglu. Pixel recurrent neural networks. In International conference on machine learning, pp. 1 1756. PMLR, 2016.
Vladimir N. Vapnik. The Nature Statistical Learning Theory. Springer; 1999.
Pascal Vincent, Hugo Larochelle, Yoshua Bengio, and Pierre- Atoine Manzagol. Extracting and composing robust features with denoising autoencoders. ICML '08, pp. 1096-1103 New York, NY, USA, 2008. A ssociation for Computing Machinery.
Gregory K Wallace. The Tpej still picture compression standard. Communications of the ACM, 34 (4):30-44, 1991.
Yair Weiss and William T Freeman. What makes good model of natural images? In 2007 IEEE Conference on Computer Vision and Pattern Recognition, pp. 1-8. IEEE, 2007.
12

---

Under review as a conference paper at ICLR 2023
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning (still) requires rethinking generalization. Communications of the ACM, 64(3):107- 115, 2021.
Maria Zontak and Michal Irani. Internal statistics of a single natural image. In CVPR 2011, pp. 977-984, 2011. doi: PR.10///
A SPARSE REPRESENTATIONS
Sparse coding (SC) is a popular task in many fields, such as: image processing EElad, 2010). computer vision aTarrttt et all 2009), compressed sensing (Donoho, 2006), ultrasound imaging (Bendory et all 2016), seismology (Pereg & Cohen, 2017: Pereg et al. 2017: 2019 2021), and vi- sual neurosciense Olshausen & Field 1996: Lee et al. 2009).
A sparse representations model (Elad, 2010) assumes signal E Rdx1 that is analyzed as a sparse linear combination of some dictionary basis components:
y= Dz,
(16)
where D E Rdx is a matrix called the dictionary, built of the atoms d; E Rdx1, i= I..., M, as its columns, and z E Rdxx1 is the sparse vector of the atoms weights. Sparse coding, that is, the recovery of z, has been the center of tremendous research efforts. Finding the sparsest solution, the one with the smallest b-- norm, is basically attempting to solve
(Po)
min ||2||o
S.L.
y Dz,
(17)
where |z]]g denotes the number of non-zeros in z. Unfortunately, Po is in general NP-Hard Natarajan 1995), therefore the no-mrmm is often replaced with the -oom
(Pi):
min |||||
s.t.
y = Dz,
(18)
where |z||
Σ, |2,1. In many real-life scenarios, such as in the presence of noise or when some enroris is allowed, we solve
(P1,))
min |z||a
s.t. ly
Dz S e,
(19)
The sparsest solution to Po and Pi is unique under certain conditions, and can be obtained with known algorithms, such as orthonormal matching pursuit (OMP) or basis pursuit (BP), depending on the dictionary's properties and the degree of sparsity of z. That is, when
where |/z||2
||z|lo- < & + m()) where ((D) is the mutual coherence defined as
d d ||d:||2 | 11
(DD
max (f)
(20)
the true sparse code z can be perfectly recovered (Candès et all 2011).
An intuitive way to recover z is to project y on the dictionary, and then extract the atoms with the strongest response by taking a hard a soft threshold, i.e., z = HD" y) orz = (D(TT)) y), where
the hard threshold and the soft threshold operators are respectively defined as
and Sp(z)= + B,
2 38 |2| S 8 2 >> 3
2,
B
Hz(z))
0,
| S
2 - B,
Note that the ReLU activation function obeys
0. 253
ReLU(2 (3) max(= 33, = (z)
2 22- B, 2 2>
Therefore, the soft threshold solution can be also written as
= SD - -
(- ReLU(DD B3 ReeLU Re -B).
13

---

Under review as a conference paper at ICLR 2023
It is possible to assume a nonnegative sparse code such that the weights are solely positive, es- sentially assuming a compounded dictionary [D, -D] (Papyan et all, 2017). Hence a nonnegative model does not affect the expressiveness of the model. Perfect support recovery by simple threshold-
ing is guaranteed only when ||z||o (1+ n(D) mnnx
I mnin. ),
where iz min, and 2 max are the minimum
and maximum values of the vector |z| on the support, implying that this approach may have stability
issues when the data is unbalanced.
In the special case where D is a convolutional dictionary, the task of extracting z is referred to as convolutional sparse coding (CSC). In this case, the dictionary D is convolutional matrix con- structed by shifting a local matrix of filters in all possible positions. PPapyan et al. (2017) show that the forward pass of CNNs is equivalent the layered thresholding algorithm designed to solve the CSC problem.
B PROOFS
B.1 PROOF OF THEOREM K
We assume training set V where (x. ym are sampled from A (P'x) and labeled by some function as ground truth, V = {(y,, { }== an algorithm that minimizes the training emrr (empirical error or empirical risk)
x; E AT (Px)). The output of a predictor y -> X'. A ssume
1 Σ (FF())))
(F(Fu)
(21)
A ssuming discrete source (-) that emits i.idd sequences xt of symbols (for example: patches in an image, segments of an audio signal, etc.). The estimator has access only to the observed signal v². (In inverse problem y" would be degraded signal originating in x", where the relationship between x. and yd could be linear or non-linear, deterministic or stochastic. Generally, estimate ,².h. However the following proof is not restricted to this framework. For the sake of the following theoretical analysis we restrict the mapping F yrd X to be surjective function. In other words, for every x", there is a yd such that f(v") = x=. In other words, every element x² is the image of at least one element of yd. It is not required that yd be unique. In the presence of noise this condition can only be met if the noise's s power is under a certain threshold. The goal is to prove that learning with A. (Px) is equivalent to training with the entire distribution with the same generalization error.
yd )xx") + e(x:))) Given yd we produce
A sequence yd of symbols is passed to the learning system. For example, it is possible that this system is built as an encoder- decoder, such that the encoder "compresses" yd into a latent space representation vector z and sends 2* into the decoder. The decoder reconstructs x" from 25, as (**).. Generally speaking, the predictor reconstruct a. from o and is said to be successful fn z". We consider the case where every source sequence xt is assigned unique 2* Therefore one can reconstruct x. perfectly. Note that the same latent space x can represent different observed sequences पن This assumption is true if and only if the mapping F : - X is unique. In other words, every x² can be mapped to more than one yd but every nd can only be mapped to one x.. The goal is to prove that learning by training over the typical set (Px) is sufficient.
Denote A = {-= : x= E (Px)), B = xa & (Px))),, B = (n).. We train the predictor with a training set V/ over . A is at most e Now, given some test input yd if x==(""" (y²) (P the encoder passes to the decoder the ²² that represents this sequence. In the general case, the predictor deciphers yd as trained by generalization. However if x= (v²) & A (Px) we can assume some unknown output fnd (the encoder sends to the decoder some unknown " generated by the trained leaming system), with error S.. The average error is upperbounded by
" of E A}, such that the generalization error (risk)
((hw) S Pr(x" E AT(x))) + Pr(x" & AP S 5 5.(N)E
(22)
But since (n) - 0 oc, we have ((hv) S ST
14

---

Under review as a conference paper at ICLR 2023
An alternative way to derive the same result is as following. The generalization emorr of the trained predictor he is
[(hw) = P(xx
E(hy (v),
E
Px, (2. v) E(hy (v).
= Σ Px(x) Σ Pro(( (h)
ImPx
v-Prix
Σ Px(x)
E (x(ll)) P-Prxx
Ca(he) S SEA.
(E))(Px)
where L (hag denotes the generalization emor over the typical set (Px) The third equality
follows from Prix" E AA (Px)] 1. Throughout the proofs we sometimes omit the superscript for simplicity. Note that this derivation is symmetric for and y, therefore it is possible to build training set by drawing samples yd E (Py) and paining them with the corresponding x", under
the assumption that the mapping F y + d is surjective function. A Alternatively, it possible to define the jointly typical set B {(x", ((x",²") (s", ,1") E A (Px,y), (Pr), E
A (Px)) and assume the training input-outpon pairs are drawn from B.
B.2 PROOF OF THEOREM !
A ssume supervised leaming algorithm with a training set NP, sampled from an unknown distribu- tion Px and paired with by some target function g(-), to learn a predictor ho 3dd + X (Here, the subscript V/ emphasizes that the output predictor depends on V..) The goal of the algorithm is to find Fy that minimizes the error with respect to the unknown Px,y over X x yd Since the leamer does not know (-), the true error is not available to the learner. A useful notion of error that can be calculated by the leamer is the training error (the emrrr over the training sample), a.k.a the empirical error or empirical risk,
Lu (1) = 1 Σ (h(x)).)))
(23)
where m = | and f(r, -t) is some loss function, such as: MSE, l norm, etc. The training sample only a glimpse into the world that is available to the leamer. This learning paradigm coming up with a predictor that minimizes Ly ( (F is called Empirical Risk Minimization or ERM for short (Shalev-Shee Shweets & Ben-David, 2014 We define the true enrr, or the generalization error, in our
framework as
((he) = E(a.)) (1),
(24)
where E(x.))~xx.v (.) denotes the expectation over Px,Y.
Let the leamer's s output sequence x E [ be a finite sequence whose i'th entry x;; takes on values in discrete and finite alphabet X.. We write x for the Cartesian product of the set X' with itself times. In other words, our hypothesis class H. obeys H = {h. ydd + X²). J. For a training sample y the trained predictor hy, minimizing
hw arg min Lo (h) hER
ssume the predictor was trained successfully to yield L.v (hv) S Am. << Recall the sequences that serve as the examples in the training set are independently and identically distributed (i...d.). We are interested in finding the sample size m sequences of instances that will lead to a bounded generalization emror for the specific trained predictor, L(hw) S A. Namely,
[(h+) = -( ((hu(y), = Σ Px,x (x.z) (1h()))
2.8
= (()))))(h))))) =
Σ
Py(y) el (v), f(v)),
VEA)(PY)
the second equation follows from P(X,Y) = P(X)) Recall that we assumed g(-) is a deterministic function, and that f(.) is a subjective function, therefore P(X f(v)lY = y) 1.
15

---

Under review as a conference paper at ICLR 2023
And the third equation follows from Prly" A 1. If m 2 2dt((), and since [A]( 2df((), and (v") 2 2- then clearty ((hy) Lv (hw) S Am. Otherwise, m 2dt1(()) and
(hwv S (2 22)2) m4m dH(M) < AA, where we have used the fact that (xx, . - 1. We now have
1 - m2 12 (x) Am) = A
(25)
And since Am <<1,
m 2 (YH 1 A
R 24H + Am),
(26)
or approximately
m 2 (1 - A).
(27)
Equivalently,
A YY) A.).
(28)
Hence the ratio between the sample size and the input data typical set determines the upper bound on the generalization enoor
L
B.3 PROOF OF THEOREM 5
Since y g(x) + n, each of the possible y sequences induces a probability distribution on the sequences. Since two different y" may originate in the same x" sequence, the xt are confusable. Fora "nof-coofusablee subset of input sequences, such that with high probability there is only one highly likely x" that could have caused the particular output, we can reconstruct the sequence at the output with negligible probability of error, by mapping the observations into the appropri- ate "widely spaced" hidden sequences. We can define the conditional entropy H (X) assuming they are eryodic and have stationary coupling (Gravl 2011). Defining their mutual information
I( = HY) H(\{X). their jointly typical set follows similar properties (Cover & Thomas, 2006). Define
((x"."" : (a" y²) E (Px, yd (P))), E A7 and AT (Pxxy (.")) {1" (x", ,"" (P(xx))) Observe that (Px (x") = 0 if x² & (Px)
K Kamerr et al. 2008).
Pr[Y E
1X" = x
(29)
Pr[(=", ((("""v") E B 2 2 (-x(())))
(30)
B 2nI(X:
(31)
Roughly speaking, we can resolve an input vf as x. if they are jointly typical. For each (typical)
output z" sequence, there are approximately 2y((() possible y" sequences, all of them equally likely. We wish to ensure that no two x= sequences produce the same y" output sequence, otherwise, we will be able to decide which xn sequence it was originated from. The total number of possible (typical) nd sequences is approx 2nH(Y)) This set has to be divided into sets of size ( (X) corresponding to the different input X sequences. The total number of disjoint sets is less than or equal to (M(H(Y)-HQV) x)) 22)(Xi)), Hence, we can have at most 2n/(X.Y) distinguishable sequences of length n. Therefore,
C(h+) EE (h)()))
Σ 2M.
Σ
ZNEB
where the last equation follows from (29 If m > 2 and since |B| 2 and Pr[(=" v²") E B] 2 (xxx),, then clearly L(hv) L (hv) A.. Otherwise, when m < 2n/(x;Y), we have
((hw) & (2 - mm)-mm)1-e ml(.)) + mmmmmmmmleee m SA.
assuming ((x, 2) < 1. We now have
m2 m2- m2-nn(XiY)( - A 2 1 A
(32)
16

---

Under review as a conference paper at ICLR 2023
Assuming 1 Am = 1,
> 2n1(X: A).
(33)
Equivalently
A A21- 2nI(X:Y] (1 - Am).
m
(34)
C RECEPTIVE FIELD NORMALIZATION
Pereg et al. (2021) propose fast altemative algorithm, inspired by the classic iterative thresholding algorithms, that produces a relatively good approximation of a convolutional sparse code. Most solvers are slowed down by the use of one global threshold (bias) to detect each local feature shift along the signal, or a predetermined constant local threshold. Whe When we apply a global threshold at each iteration, the threshold is too high, weak expressions are annihilated, and strong expressions can "shadow" over low-energy regions in the signal, which in tum, can be interpreted as false- positive support locations. On the other hand, if the threshold is very small, as often the case in ISTA, many iterations are required to compensate for false detections of early iterations. This issue is aggravated in the presence of noise, and in real- time applications due to model perturbations. The proposed remedy is to normalize each data point by locally focused data energy measure, before applying a threshold. In other words, each receptive field of the data is scaled with respect to the can still use common bias
local eneryy. This way even when the data is inherently unbalanced, for all receptive fields, without requiring many iterations to detect the features support.
We briefly reformulate RFN for 2D signals.
Definition C.1 (2D Receptive Field Normalization Kernel); A kernel h[k, 4, k,, I E Z, can be referred to as a receptive field normalization kemel if
1. The kernel is positive: li[k, I 2 Vk,
2. The kernel is symmetric: lt[k; [] = hl-k, -1] Vk: L.
3. The kernel's global maximum is at its center: 10, 0) 2 [k. vk, 0.
4. The kernel's energy is finite: Σ., Mi[ks [] < co.
Definition C.2 (2D Receptive Field Normalizationll We define the local weighted energy centered around S(k: I], sample of 2D observed signal S RL,x.
os [| &
(35)
where [ik:: d] is a RFN window function of size Ln x Ln, Ln << L, is an odd number of samples. used truncated Gaussian shaped window, but one can use any other window
For our application function depending on the application, such as: a rectangular window, Epanechnikov window, etc. The choice of the normalization window and its length affects the thresholding parameters. If [k. I] is a rectangular window, then og kk. I] is simply the Frobenius norm of a data patch centered around the [k, I] 1h location. Otherwise, if the chosen RFN window attenuating, then the eneryy focused in the center of the receptive field, and possible events at the margins are repressed. RFN is em- ployed by dividing each data point by the local weighted eneryy, before projecting the signal on the dictionary and taking threshold. We compute local weighted energy as defined in (35). Namely,
ag[k, [] = hhkk, * S²Ik, C.
(36)
where li(k: I] receptive field normalization window, and denotes the convolution operation. Then, we nommalize the signal by dividing each data point by the conrespondigg receptive field eneryy. In order to avoid amplification of low energy regions, we use a clipped version of [k(k). Namely,
aas [
las kk, I]] T os k, [] <
āglkt, []
(37)
17

---

Under review as a conference paper at ICLR 2023
where 7 0 is predetemmined threshold. Empirically, for our application 0.15 $ t S 0.4 works well. The RFN image $ image is
Skk: ! ãs |k
S []
(38)
A sssming an analysis patch {y E R from RFN image $ RL.XJ (as defined in Def- 5)) given a normalized convolutional dictionary D, an initial solution support (at the first iteration)
0, the detected
qi = Za, (D" (D"y.), y:), where Z is an element- wise thresholding indicator function
(39)
Is, (Tx) = {
| > 81
(40)
0
| < B1.
Algorithm 1: 2D REN-RNN (forward pass)
input : Input image S E RL,xJ, RFN-kernel E Rlaxhn, W
W 221
Init: Zo O, qu 0, ==0 compute:
os [K, nl)
Vhlk, (.
0s kt 1 {1))) [o,lk, [K
los ki !] 2 las !] <r'
S1k, 1| = [/oskK/
for ic to L. do
for to J do
set: analysis patch (y; JL E R from image S E RJ and its corresponding analysis patch (fr) 1 RAXN from RFN image S RI,XJ
for to T do
q. WT
Vt WT 2 4-11+ b)
WT 2y
Yt
2, q. Ft
end
end end
The implied assumption is that W.y represents 1 DD, therefore, it is also possible to add additional regularization loss terms enforcing = Dz,, Y: Dž,, where y, is the RFN-input signal and ž the RFN latent supports vector.
D EXPERIMENTS DETAILS
D.1
IMAGE DEBLURRING
Table 1: PSNR (dB) and SSIM scores for natural image deblurring
Bullerly
Boels
C. MaM
House
Parrol
Lena
Barbara
Starfish
Peppers
Leaves
Avg
Imput PSNK
22.84
26.40
23.30
28.52
27.05
26.33
23.84
24.80
25.97
22.26
25,20 26
FISTA
30.36
29.36
26.81
31.50
31.23
29.47
25.03
29.05
29.42
29.36
29.22
NUSH
30.84
31.49
28.34
33.63
33.39
31.26
27.91
32.27
30.16
31.57
31.00
TNRD
31.57
31.53
26.31
33.71
33.19
26.62
32.46
29.98
31.95
RNN
27.80
29.40
25.44
31.27
30.25
29.00
27.75
29.25
28.17
26.82
28.52
0.93
0.91
0.88
0.93
0.95
0.92
0.82
0.92
0.90
0.94
0.91
RNN
25.16
28.45
25.16
30.65
29.01
24.47
28.73
28.01
27.56
0.91
0.90
0.88
0.92
0.94
0.91
0.81
0.92
0.91
0.91
0.90
To test the systems' performance under more challenging setting we tested RNN and RNN- RFN
with increasing additive noise variance o, in the range [0, 10 We trained the network with
18

---

Under review as a conference paper at ICLR 2023
single example of the blureed image boat contaminated by Gaussian noise with variance on = v22 As can be seen in Fig. [ as the noise level increased RFN suppresses noise better. Figures provide additional examples of the reconstruction image starfish for on = 2v2 2 and on 7v2.
PSKR
29
28
PSRR (RNN-FM
27
26
24
22
21
10 12 14
Figure 6: PSNR[dB] as function of number of noise variance on-
(b)
(c)
(d)
Figure 7: Visual comparison of deblurring of the image starfish for different noise levels: (a) Ground truth; (b)
input, PSNR=24 22.2dBB o.. 1.24; (c) RNN result, PSNR=28.07dBB (d) RNN- REN result, PSNR=27.91dB
XX (d)
(a)
(b)
(c)
Figure 8: Visual comparison of deblurring of the image starfish for different noise levels: (a) Ground truth; (b) input, PSNR=23.02dB, o. 9.90; (c) RNN result, PSNR=25.02dB (d) RNN-RET BNN-RFN result, PSNR=25.07dB
D.2 OCT SPECKLE SUPREESION
The learning system is the RNN framework described in section 5 a cropped single intensity image of coherent tomogram aligned with a speckle suppressed to- mogram R E LJ We then define an analysis patch a 2D patch of size L, N enclos- ing L, time (depth) samples of N consecutive traces of the observed image S. An analysis patch A (i.)) associated with pixel R.,, in the speckle suppressed intensity tomogram. To produce a point in the final tomogram R.,,, we set the input to the RNN as y A (4J),, Each time step in- put is group
We denote S LR as
neighboring pixels of the same conresponding time (depth). other words,
y, = [S; I -+t.j-nn S 1+t.jttn is row of the corresonndigg analysis patch. We set
the size of the output vector x, to one expected pixel: 1. Such that is the expected segment
19

---

Under review as a conference paper at ICLR 2023
4.5
(tbb) Ca(he)
fim)
33
<10
Figure 9: Recovery envr L(hy) and training enror Ly (hw) < Am (logı0 scale) as a function of the sample size m. Empirically, Ly (hw) < am, where = (x(x) - Am), for Y() 2 0.33.
of the corresponding enhanced tomogram, R (L, 11.j. RR, Lastly, we ignore the first L values of the output x, and set the predicted pixel R,,,, as the last one, xL.
The analysis patch moves through the image and produces all predicted speckle suppressed intensity tomogram points in the same manner. Each output pixel is recovered by taking into consideration its relations to neighboring pixels: TA-1 preceding pixels in the axial direction at the same lateral position, and (N 1) L, pixels from N - laterally neighboring surrounding pixels (in space), from adjacent A lines. For experiments, we set the number neurons as n. 1000. Increasing the number of neurons did not improve the results significantly, but increases training time.
In recurrent neural network the nodes of the graph are connected by both feedforward connections and feedback connections (Hochreiter & Schmidhuben [197: Geror, 2017): At a current state, the current output depends on current inputs, and on outputs at previous states. In a sense, we can say that the network is able to make decisions that are based also on the memory of its past decisions, that it "remembers" its past outputs and takes them into consideration when computing the current output. We believe RNN fits this task because it is able to "remember" both in space and time dimensions. Most OCT images have layered structure, and exhibit string relations along the axial and lateral axes. RNNs can efficiently capture those relations and exploit them.
Figures 4 a)- (b) show the speckle-comeeeee coruupted and speckle-suppressed suppressed tomograms of chicken tissue of size 341 x 691 used for training. The analysis patch size of size [L, .] [9,9], 9]. The axial and lateral spacing for both training and testing data are Ag = 3.06pm, A = 6ym.
In Figs. (J))-(b) tomogram of size 300 643 spatial sampling spacing are Ax 2.5pm, A₂ = 4.78pm, respectively. Whereas, the cucumber axial and lateral spacing are Ax Sym, A 4. 4.7mm respectively. Note that, the cucumber image is sampled at significantly lower rate along the lateral direction. Due to the system mismatch between the training data and the testing data, we set the analysis patch size to 7, match the testing data parameters, rather than the optimal analysis patch size that would fit the training data.
used for training. Here, the lateral and axial
20


Reviews:
[{"review_id": "ypWV3msJICc", "reviewer": "Reviewer_WSnj", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "First, the authors make some proofs (whose correctness I haven't fully verified) based on  the asymptotic equipartition property that show there exists a small set that can represent the whole data distribution. Then, the authors make some connections between RNNs and sparse coding. Finally, authors test RNNs and RNNs with receptive field normalization in some ad-hoc tasks without implementing proper baselines.", "summary_of_the_review": "The paper needs to have a clear focus and structure. The theoretical results don't seem insightful. On top of that, the empirical evaluation of the results is far from enough to make any serious claims about the approach.", "strength_and_weaknesses": "The empirical evaluation of the paper is very very limited. If the authors want to make a convincing argument they will have to use more standard datasets and make comparisons to better approaches. The RNN model performs very close to the RNN-RFN according to Table 1. I wouldn't be surprised if dropout or l2 regularization would push the performance of a basic RNN beyond the other approaches.\n\nI haven't fully understood the mathematical results, but they seem overly complicated without adding much to the proposed approach (I spent quite some time going through them).", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "v8wH9dpcS6", "reviewer": "Reviewer_WHvp", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper analyzes the learning problem in three aspects. 1. From the perspective of information theory, when the loss is bounded and the entropy of the data distribution is small, one can learn only in the typical set; 2. Modeling RNN propagation as the iterative method that solves sparse sensing and RNN training is equivalent to dictionary learning; 3. Proposed the RNN algorithm with weighted image. Finally there are experiments verifying the points.", "summary_of_the_review": "Given my confusions and questions above, I would like to see more authors' response and clarification, and would definitely be willing to increase the score if it's convincing.", "strength_and_weaknesses": "This paper is broad that touches three aspects of the learning problem, and the insight from info theory provides a different perspective to ML community. I believe the math proof is correct and there are illustrative experiments as well. Below are detailed questions.\n\n- The introduction is not clear. I would suggest moving Figure 1 to the front. Now when I read it, seems the beginning part starts from introducing the last two blocks of the figure, not mentioning the left part. I’m confused about the setup yet, is it like, there is a system that encodes x as y, and we want to find the decoder/inverse map from y to x? How is it relevant to RNN? \n- I think the logic is not quite clear to readers. The connection among three parts seems to be weak. For example, the first part is relevant to sample complexity, but the second part is analysis of RNN forward algorithm. How is it relevant to typical set stuff?\nAlthough there are a few theorems in the first part, I think most of them are textbook knowledge. The entropy is directly relevant to how hard the learning is and the typical set represents the most likely appearing instances following the distribution. I don’t find much novelty here.\n- The justification of the Algo 1, especially the novelty, is very short and I cannot understand it. \n- The paper has the title “few-shot”, but I cannot see how this paper contributes to it, i.e. how to learn from “few” data. The typical set stuff seems to be relevant, but perhaps I did not catch its relation to the other sections. \n- Section 3,4 constructs the equivalence of iterative lasso solver and RNN, but I cannot see why this connection is important and correct. For example, the whole sequence of RNN is important, but are the iterations in the middle of lasso solver important, what do they mean? I feel in lasso, the iterations would be more and more sparse, is it an expected and important property of RNN sequence? Slight differences between lasso and RNN exist, like the bias $b$, and the thresholding parameter $\\lambda$, etc. \n- Critical: Comparing (14) and (15), is $W_{zz}$ forced to be a PD matrix as assumed in lasso solver, is it enforced that $I -c W_{zy}W_{zy}^T = W_{zz}$? If $W_{zy}$ and $W_{zz}$ represent $D^T/c$ and $I - D^TD/c$ then they have to follow that equality. One needs to justify and properly use the similarity between lasso solver and RNN, otherwise it just looks like a coincidence.\n- Page 5, $cI-D^TD \\succ 0$.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "ay7bvMDixP", "reviewer": "Reviewer_uFRZ", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors provide some theoretical statements showing that a relatively small set that can empirically represent the input-output relationship for learning.\nThey then pointed out that sparse coding with iterative solutions can be modeled by RNN computation.\nThey applied the proposed method to few-shot learning.\nExperimental results on image deblurring and optical coherence tomography (OCT) speckle suppression demonstrate that the proposed method performs well.", "summary_of_the_review": "The relationship between the theoretical part and the actual implementation part is unclear. Clarity is low.", "strength_and_weaknesses": "*Strength\n- Experimental results of the proposed method showed promising performance.\n\n*Weaknesses\n- The relationship between the theoretical part (Section 2) and the actual implementation part (Sections 3-5) is unclear. The theory of sparse coding itself might work for this instead of the discussion of AEP.\n- Clarity is low.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "1: The main claims of the paper are incorrect or not at all supported by theory or empirical results.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "Uj4yWqlVX6Z", "reviewer": "Reviewer_GAhp", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper targets the understanding of few-shot learning under the image reconstruction task. Specifically, it focuses on OCT speckle suppression and image deblurring, which use local information to clean the speckle/noise from the full image. The performance on the two tasks has successfully demonstrated the effectiveness of the approach though the tasks are kind of simple and may be of limited impact. Regarding the approach side, it first provides a theoretical analysis of few-shot learning from the perspective of asymptotic equipartition property and shows that there is a relatively small set (i.e., typical set) whose sample entropy is close to the true entropy and can represent the whole data distribution. Then, it proposes to connect RNN with sparse coding and treat the RNN as an unfolding of one iteration of the sparse coder. Finally, the RNN network is used in few-shot learning for related tasks. ", "summary_of_the_review": "Though detailed analyses are provided, the experiment is not quite promising, and it is hard to measure the impact of this approach. In addition, as mentioned above, further discussion is needed to determine whether the image patches can be formulated as sequences.", "strength_and_weaknesses": "Strength:\n1.\tIt is interesting to find the connection between RNN with a sparse coder, which is then used for few-shot learning in speckle suppression and image deblurring tasks.\n2.\tIt is surprising to obtain reasonably good performance by training the model on a single training image set.\n3.\tClear explanation about the newly proposed perspective of data generation w.r.t. AEP.\n\nWeakness:\n1.\tAt Page 4, right after Eq.3, it is claimed that the sample size should be necessarily large. However, I fail to see clear connection between the sample size and the proposed RNN-based approach. \n2.\tThough RNN can be used as an unfolding of an iterative algorithm, it still requires the assumption that different columns in single patches are formulated as a sequence with clear order. However, this may not be true. At least a bi-directional RNN considering the order of columns along two complementary directions should be considered.\n3.\tThe experiments are not comprehensive. In Table 1, the performance difference between RNN & RNN-RFN is not significant. A more detailed explanation should be provided. Also, the PSNR and SSIM are not clearly labeled (I assume only the bottom approaches have SSIM scores).\n4.\tWhat are the compared approaches? The author should at least give a reference for approaches listed in Table 1 and explain why the performance underperforms the compared approaches by a large margin.\n5.\tAs mentioned above, the tasks used for evaluation are too simple. In addition to the simple cases, evaluation on a more challenging task (e.g., a more challenging noise situation added in image deblurring) should be performed.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject