Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: First, the authors make some proofs (whose correctness I haven't fully verified) based on  the asymptotic equipartition property that show there exists a small set that can represent the whole data distribution. Then, the authors make some connections between RNNs and sparse coding. Finally, authors test RNNs and RNNs with receptive field normalization in some ad-hoc tasks without implementing proper baselines.
strength_and_weaknesses: The empirical evaluation of the paper is very very limited. If the authors want to make a convincing argument they will have to use more standard datasets and make comparisons to better approaches. The RNN model performs very close to the RNN-RFN according to Table 1. I wouldn't be surprised if dropout or l2 regularization would push the performance of a basic RNN beyond the other approaches.

I haven't fully understood the mathematical results, but they seem overly complicated without adding much to the proposed approach (I spent quite some time going through them).
clarity,_quality,_novelty_and_reproducibility: The paper is unnecessarily complicated to read and doesn't focus on it's topic. Sections 3 and 4 give a lot of details about sparse coding that do not add much to the proposed approach.
summary_of_the_review: The paper needs to have a clear focus and structure. The theoretical results don't seem insightful. On top of that, the empirical evaluation of the results is far from enough to make any serious claims about the approach.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The authors provide some theoretical statements showing that a relatively small set that can empirically represent the input-output relationship for learning.
They then pointed out that sparse coding with iterative solutions can be modeled by RNN computation.
They applied the proposed method to few-shot learning.
Experimental results on image deblurring and optical coherence tomography (OCT) speckle suppression demonstrate that the proposed method performs well.
strength_and_weaknesses: *Strength
- Experimental results of the proposed method showed promising performance.

*Weaknesses
- The relationship between the theoretical part (Section 2) and the actual implementation part (Sections 3-5) is unclear. The theory of sparse coding itself might work for this instead of the discussion of AEP.
- Clarity is low.
clarity,_quality,_novelty_and_reproducibility: *Clarity
- In p.3, there are no definitions for g(x) and e(x).
- In p.3, n has multiple meanings for the superscript n for x^n and the number of dimensions of x, n, which is confusing.
- In Eq.1 and 2, x and y are not bold. Is it a typo? Or are they different from other bold x and y?
- In Eq.3, there is no definition for Delta.
- The character lambda is duplicately used in different meanings as the parameter in Eq.7 and eigenvalue.
- In Eq.15, there is no bias term compared to Eq.5. Are there any analyses or discussions with that?
- In Eq. 11 and 15, what can we interpret W and S as (like D is a dictionary)?
- How do the authors optimize W and S? What are training samples for z?
- Definition 1 is hard to follow. Providing some illustrations can help.

*Quality
- Please see the above comments.

*Novelty
- The proposed method might be novel.

*Reproducibility
- Code is unavailable.
summary_of_the_review: The relationship between the theoretical part and the actual implementation part is unclear. Clarity is low.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper targets the understanding of few-shot learning under the image reconstruction task. Specifically, it focuses on OCT speckle suppression and image deblurring, which use local information to clean the speckle/noise from the full image. The performance on the two tasks has successfully demonstrated the effectiveness of the approach though the tasks are kind of simple and may be of limited impact. Regarding the approach side, it first provides a theoretical analysis of few-shot learning from the perspective of asymptotic equipartition property and shows that there is a relatively small set (i.e., typical set) whose sample entropy is close to the true entropy and can represent the whole data distribution. Then, it proposes to connect RNN with sparse coding and treat the RNN as an unfolding of one iteration of the sparse coder. Finally, the RNN network is used in few-shot learning for related tasks. 
strength_and_weaknesses: Strength:
1.	It is interesting to find the connection between RNN with a sparse coder, which is then used for few-shot learning in speckle suppression and image deblurring tasks.
2.	It is surprising to obtain reasonably good performance by training the model on a single training image set.
3.	Clear explanation about the newly proposed perspective of data generation w.r.t. AEP.

Weakness:
1.	At Page 4, right after Eq.3, it is claimed that the sample size should be necessarily large. However, I fail to see clear connection between the sample size and the proposed RNN-based approach. 
2.	Though RNN can be used as an unfolding of an iterative algorithm, it still requires the assumption that different columns in single patches are formulated as a sequence with clear order. However, this may not be true. At least a bi-directional RNN considering the order of columns along two complementary directions should be considered.
3.	The experiments are not comprehensive. In Table 1, the performance difference between RNN & RNN-RFN is not significant. A more detailed explanation should be provided. Also, the PSNR and SSIM are not clearly labeled (I assume only the bottom approaches have SSIM scores).
4.	What are the compared approaches? The author should at least give a reference for approaches listed in Table 1 and explain why the performance underperforms the compared approaches by a large margin.
5.	As mentioned above, the tasks used for evaluation are too simple. In addition to the simple cases, evaluation on a more challenging task (e.g., a more challenging noise situation added in image deblurring) should be performed.

clarity,_quality,_novelty_and_reproducibility: Clarity: 
The concept is clear.

Qaulity: 
The idea is realtively novel but the task is simple.

Reproducibility:
Need some more details
summary_of_the_review: Though detailed analyses are provided, the experiment is not quite promising, and it is hard to measure the impact of this approach. In addition, as mentioned above, further discussion is needed to determine whether the image patches can be formulated as sequences.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper analyzes the learning problem in three aspects. 1. From the perspective of information theory, when the loss is bounded and the entropy of the data distribution is small, one can learn only in the typical set; 2. Modeling RNN propagation as the iterative method that solves sparse sensing and RNN training is equivalent to dictionary learning; 3. Proposed the RNN algorithm with weighted image. Finally there are experiments verifying the points.
strength_and_weaknesses: This paper is broad that touches three aspects of the learning problem, and the insight from info theory provides a different perspective to ML community. I believe the math proof is correct and there are illustrative experiments as well. Below are detailed questions.

- The introduction is not clear. I would suggest moving Figure 1 to the front. Now when I read it, seems the beginning part starts from introducing the last two blocks of the figure, not mentioning the left part. I’m confused about the setup yet, is it like, there is a system that encodes x as y, and we want to find the decoder/inverse map from y to x? How is it relevant to RNN? 
- I think the logic is not quite clear to readers. The connection among three parts seems to be weak. For example, the first part is relevant to sample complexity, but the second part is analysis of RNN forward algorithm. How is it relevant to typical set stuff?
Although there are a few theorems in the first part, I think most of them are textbook knowledge. The entropy is directly relevant to how hard the learning is and the typical set represents the most likely appearing instances following the distribution. I don’t find much novelty here.
- The justification of the Algo 1, especially the novelty, is very short and I cannot understand it. 
- The paper has the title “few-shot”, but I cannot see how this paper contributes to it, i.e. how to learn from “few” data. The typical set stuff seems to be relevant, but perhaps I did not catch its relation to the other sections. 
- Section 3,4 constructs the equivalence of iterative lasso solver and RNN, but I cannot see why this connection is important and correct. For example, the whole sequence of RNN is important, but are the iterations in the middle of lasso solver important, what do they mean? I feel in lasso, the iterations would be more and more sparse, is it an expected and important property of RNN sequence? Slight differences between lasso and RNN exist, like the bias $b$, and the thresholding parameter $\lambda$, etc. 
- Critical: Comparing (14) and (15), is $W_{zz}$ forced to be a PD matrix as assumed in lasso solver, is it enforced that $I -c W_{zy}W_{zy}^T = W_{zz}$? If $W_{zy}$ and $W_{zz}$ represent $D^T/c$ and $I - D^TD/c$ then they have to follow that equality. One needs to justify and properly use the similarity between lasso solver and RNN, otherwise it just looks like a coincidence.
- Page 5, $cI-D^TD \succ 0$.
clarity,_quality,_novelty_and_reproducibility: I think the paper is original, but a clearer connection among sections and better explanation would be helpful.
summary_of_the_review: Given my confusions and questions above, I would like to see more authors' response and clarification, and would definitely be willing to increase the score if it's convincing.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This manuscript studies few-shot learning under the image reconstruction task. Specifically, it focuses on OCT speckle suppression and image deblurring, which use local information to clean the speckle/noise from the full image. It first provides a theoretical analysis of few-shot learning from the perspective of asymptotic equipartition property and shows that there is a relatively small set (i.e., typical set) whose sample entropy is close to the true entropy and can represent the whole data distribution. Then, it proposes to connect RNN with sparse coding and treat the RNN as an unfolding of one iteration of the sparse coder. Finally, the RNN network is used in few-shot learning for related tasks. 

Although it is interesting to see the connection of RNN with sparse coding iterative algorithms, the reviewers have main concerns on the evaluation of the methodology and clarity of the logic flow. The reviewers unanimously rate the current version of the manuscript below the accept line, so is the decision. 
justification_for_why_not_higher_score: The reviewers unanimously rate the current version of the manuscript below the accept line, so is the decision.
Decision: Reject