Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper considers the problem of training image classification for meta-learning/few-shot learning models with large images. The authors point out that many categories of meta-learning algorithms have difficulty using large images because of memory constraints brought up by computation of gradients computed during meta-training. They decompose 3 major categories of meta-learning models (CNAP, MAML, and Proto-Nets) into a common gradient update that is troublesome w/r/t memory. Their idea to make training more feasible is to use all the support set items in the forward pass but approximate the backward pass using a small random subsample of the examples in the support set. They compare training their versions of memory-efficient, large input meta-learning models against recent work on ORBIT (a few-shot object recognition task) and VTAB+MD (a newer benchmark for few-shot classification that is based on Meta-Dataset and VTAB datasets). On both benchmarks, they show that training using larger images using their proposed method improves performance.
main_review: Originality: the paper is considering an interesting aspect of the meta-learning problem (holding constant the method but using larger input images) that as far as I know hasn't been considered before.

Quality: the proposed method and its motivating analysis seem sound. The experimental comparison is also fair and highlights the benefit of using the large image input based on their proposed approximation. However, I believe the experiments could've been expanded. Why not consider mini-ImageNet or tiered-ImageNet benchmarks in which most work uses 84x84 inputs? It would be interesting to see the benefit of larger inputs on those commonly used benchmarks. Additionally, I think an ablation experiment showing how the quality and speed of training varies with the approximation parameter H would also have been useful to evaluate the method. Lastly, I also think first-order maml with a larger input would be a useful baseline to see the effect of using larger inputs in a different way than what the authors propose.

Clarity: the paper is very well-written. It motives the problem well and has a clear description of the proposed solution. The experiments also seem to be conducted fairly and are informative.

Significance: I think the results in the paper could be useful to researchers as they show a way to improve quality of meta-learning models simply by increasing the image resolution. One concern I have is that the authors include MAML in the class of methods for which the proposed approximation applies but in the end, do not implement or show results for the method since "...because of the memory required to retain the inner loop’s computation graph, training may still need to be scaled to smaller images or smaller tasks." I am wondering if its actually useful to include discussion of MAML then in the paper if the paper doesn't have a concrete proposal/implementation of the approximation of second-order MAML that allows it to be trained on larger images?
limitations_and_societal_impact: Yes.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: Meta-learning algorithms are not memory friendly since they have to load all the support samples during the inner-loop. When the support data are high-resolution images, or need huge support samples (i.e. 1000), the meta-training is hard to work on a single GPU. As a result, this paper aims to propose an efficient meta-training algorithm which only rely on the gradients of subsets of the support data. And they show the can achieve competitive results against transfer learning approaches but at a fraction of the test-time computational cost on multiple benchmarks.
main_review: The motivation of the paper is clear and the paper is easy to follow. The proposed algorithm is extremely simple and can be applied to almost all the meta-learners. The results on benchmarks which need large support samples seems promising. It is interesting to make the meta-learners more efficient but I am not sure how significant the problem is. In addition, I think more experiments are needed to show the effectiveness of the proposed algorithm.
limitations_and_societal_impact: 1)  Are the performance of the meta-learners sensitive to the choice of H?  Can we have a table or a figure about that, like accuracy v.s. different choice of H. I am interested in how meta-learning performs when H is the full support size or when H=0 (Just detach the gradient of the support samples).

2) Which one is more important to achieve a better performance? A larger support size or the gradient we needed for the support samples? Can we have the results where we have only the maximum number of support samples with 224 * 224 resolution to fit the memory?

3) The experiments only show that the proposed algorithm works, but lacks experiments like how this unbiased estimator performs against the full gradient. In addition, we need to show the gradient of the support samples are important.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The authors propose a method that enables few-shot learners to be trained on 1 GPU without memory issues. This is very important since most few-shot learners need to process the whole train set of a task before evaluating the loss on the task. They show mathematically that their method gives an unbiased estimation of the true gradient (if all samples in training were back-propagated to update the model). They evaluate their method on CNAPS and ProtoNets and show that it can achieve better results since it can be trained with larger images.  
main_review: The paper is well-written and well-structured. The problem that they try to solve is very important and also very challenging. The idea seems to be generalizable to different meta-learning techniques. They show it for CNAPS and ProtoNets. Their method is based on the idea to update the parameters of the model with the whole train set of the task but backpropagate through only a subset of the task. I think the mathematical study of this really helps the paper. Since this is mathematically correct, it might open new directions for new meta-learning algorithms. 

I have two concerns. First, optimization-based algorithms like MAML that require several inner steps are still cannot get much improvement from this method. I looked at the supplementary material and the part that discussed MAML, however, I could not understand why it is not possible to limit the back-propagation on only a few samples from the train set. Can you please explain more about this?

My second concern is that some other related work like (Meta-learning with implicit gradients) can be used to solve the same problem. It seems that related work did not thoroughly cover other approaches for solving the memory issue in meta-learning. Has there been any other work that tackles this issue? I am aware that it might not be possible to compare directly, however, it is good to discuss them.  

======Post rebuttal======

I thank the authors for clarifying reviewers' questions. I strongly suggest the authors modify the paper based on reviewers' requests. Especially, point 5 in reviewer wfg9's comment and experiments that were done for reviewer ES9N's comments. After discussing with other reviewers and reading their comments, I decided to stick with my initial review score of 6.
limitations_and_societal_impact: The authors described the limitations and societal impacts of their work.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: Many meta-learning approaches have been trouble with the scalability issue due to the large memory-required computation during the training process. To overcome such a problem, this paper proposes LITE that is memory efficient episodic training scheme allowing meta-learning methods to use the benefit of large-size images of tasks on a single GPU. On two benchmark datasets (ORBIT and VTAB+MB), meta-learning methods combined with LITE outperform meta-learning methods without LITE and Simple CNAPs (meta-learning method) + LITE shows competitive performance compared with transfer learning methods.
main_review: I think this paper is nice since it tackles the memory issue that has been a problem for a long time in the meta-learning field and the proposed solution is generally applicable for many meta-learning methods. However, I wish some clarity problems and analysis are improved during the rebuttal period.


- Strength
    - This paper is well written and easy to follow.
    - This paper tackles an important problem in the meta-learning domain, which is limited scalability due to the memory-intensive computation of training of support sets of tasks.
    - By overcoming the above issue, this paper achieves the SOTA performance on two few-shot benchmark datasets.


- Comments (Weakness)

    (order of importance)

    - Could the authors report memory savings and whether the performance drops for the same image sizes (e.g., 84) when we use LITE?
    - I wonder how much we can enlarge the image size or task size by using LITE with a single GPU?
    - In Table 1, could the authors report GPU hours (or Wall Clock Time) at the test time of each method? I wonder how much we can save time when we use the meta-learning methods compared with transfer learning as the authors described in the introduction.
    - In Table 1, could the authors report the results with 224 EN-B0?
    - Could the authors add discussion on [1] that is a method for large-scale meta-learning?

    [1] Shin et al. Large-Scale Meta-Learning with Continual Trajectory Shifting, ICML 2021.
limitations_and_societal_impact: Please refer to Comments in Main Review.
review_title: null

Meta-review: The paper is addressing the problem of meta-learning with large-scale images. In this setting, many of the existing algorithms require an intractable amount of memory. The proposed method addresses this challenge by simply sub-sampling the support set in the backward pass. Although the method is sensible, the empirical gain is marginal. All reviewers agree on the merit of the paper to be accepted after an active discussion phase.
Decision: Accept (Poster)