Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper introduces a novel setup called Anytime Learning at Macroscale. In this setup the learner receives the examples as a sequence of large batches, and is required to output a model after processing each batch. This model is used to give prediction for the next batch. The overall performance is then sum of the average losses on the individual batches.
main_review: My main concern is with the motivation of the setup. In particular, it does not seem to be very different from online learning. The paper does discuss the differences between the two setup, however the arguments are a bit superficial. In particular, it is claimed that in online learning the examples are streamed one at the time as opposed to receiving them in large batches - but this does not seem to make the online learning methods completely handicapped. Randomized algorithms (such as Thompson-sampling based methods) should perform reasonably. (For each example in batch i, apply the model obtained after processing all the examples in batches 1,2,..., i-1.) It would be great if the authors could elaborate on this.
summary_of_the_review: Concern with the motivation: the setup is not that different from online learning after all.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The authors describe a framework to perform empirical evaluation of an anytime learning setting where data is available in a streaming minibatch fashion. With a primary aim to measure performance of a classifier across variety of practical settings of such streaming data to not only achieve high accuracy, but also provide non-trivial prediction anytime using limited computational resources. Using multiple benchmark datasets, the paper concludes that methods with intermediate parameter updates are better on the accuracy to computational efficiency tradeoff, and larger models generalize better.
main_review: Paper is well written. The authors document the approach they have considered, and the metrics used to evaluate the various experimental settings used. 

As I started to read the paper, the problem setting seemed very similar to the ones used in data stream mining research over the past decade. Though the authors state that the primary differentiator to the stream setting is that the models use what they call as "meta-batches" as streams rather than streaming single data instances, I fail to understand any theoretical or empirical difference in the two approaches. There exists multiple popular data stream frameworks such as MOA (Massive Online Analysis) that is used exactly for the problem setting described in the paper. So, the primary contribution of the paper seems to be in extensively evaluating the model complexity and approaches over various data and problem settings. Moreover, majority of the future questions that the authors hope to answer have been studied across various papers (in similar forms). Please refer to Lu, Jie, et al. "Learning under concept drift: A review." IEEE Transactions on Knowledge and Data Engineering 31.12 (2018): 2346-236.

The second objection of the paper is that the authors seem to use simple datasets from today's standard to derive their conclusion. Though the conclusions in the paper are fair and not surprising, a more complex set of datasets may provide a stronger result. Furthermore, it is important to note that there are other factors that influence the classifier performance, more than the batch size and data size available for training. The data itself may be imbalanced, non standard etc. So, by using more datasets, these issues can potentially be elevated.
summary_of_the_review: In summary, the original contribution is not very clear. The authors have ignored to discuss comparisons with a branch of data stream mining reseach that has provided similar conclusion. And the empirical evaluation is on simple datasets, and may need further evidential support from more complex datasets.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors consider a batch learning problem, in which large batches of data arrive in series. They explore the performance of several types of algorithms in terms of their computational cost, model size, and error rate. 
main_review: The problem presented by the authors is relevant to applied ML/AI problems, which are always a work in progress. Further, improving the efficiency of learning is desirable. So, the problem is reasonably well motivated.

1. I'm skeptical about the value of the cumulative error rate. From a practical point of view a data engineer might be concerned with the questions; how good is the model I have now? What would be impact of further data collection? If one has collected a set of batches, the performance of prior models is not terribly relevant. 

2. The non-iid nature of the data is mentioned, and it is mentioned that cross-validation is carried out only on the current batch. These concepts could be explored more thoroughly. What are the issues with evaluation in this setting? Should I hold out a portion of every batch to use for evaluation? What other evaluation strategies are possible? What are their strengths and weaknesses?

2. In continuous streaming settings, there may be distribution shift over time. That is not addressed in this work. 

3. The authors make a big point about the scale of the problem. Obviously this makes naive approaches less appealing. But how does the problem scale? What really separates (if anything), the macroscale problem from more mundane sized problems? Are there underlying scaling laws at work that cause shifts the performance of each learning strategy?
summary_of_the_review: The paper addresses a real need, but more insight is needed.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: Summary: This paper proposes anytime learning at macroscale (ALMA), which is anytime learning under the assumption that data is observed as a sequence of large batches. This paper introduces metrics that can be used to access the error rate, memory, and compute throughput the entire learning process. They evaluate multiple learning models on different datasets in the ALMA setting. They observe that methods that update parameters at a moderate rate tend to yield a better tradeoff, while bigger models tend to generalize better.
main_review: The problem setting of anytime learning at macroscale is interesting and novel to me. How to efficiently learn data in a streaming fashion is a practical challenge. The proposed learning setting targets the level of the entire sequence of large datasets. 

Although the method overall is valuable and interesting, the paper is poorly organized and thus hard to understand. It is difficult to find some critical details or notation definitions that are related but separated apart in the paper. The method lacks an integrated and principal formulation from which the techniques are derived. Though they show some theoretical results, it is hard to relate them to the objective and the algorithm steps explicitly and tightly. The contribution is unclear. Here are some detailed comments:

- What does "organically generated" mean?
- " both training ... and finetuning ... are not satisfying" should use "neither...nor..."
- What does "constrained capacity" ?
- What is the definition of macroscale?
- Unclear the difference between ALMA and other learning frameworks.
- Related work compares ALMA to lots of different prior work, but it is poorly organized and I am not sure why those prior work should be considered as comparisons.
- It is suspicious to say that "extensions to regression and unsupervised learning (where y is missing) are trivial".
- In fixed architecture, why "A potential drawback of Ens is that evaluation and training are inconsistent"?
- In Growing Mixture of Experts, "Compared to Ens, MoE has exponentially many more components". I am not sure where "exponentially" comes from.


Quality: The submission is technically sound. The claims in the contribution are supported by empirically results. It is a complete piece of work.

Clarity: The experimental details are also very specific, such that reproducing the results should be possible. 
summary_of_the_review: This paper proposes a novel learning setting called anytime learning at macroscale (ALMA). The proposed idea is simple and technically sound. Extensive evaluations are conducted. However, the contribution is unclear, and the presentation needs improvement.
review_title: null

Meta-review: The work presented in this submission is focused on a new approach for learning a model that can perform well at any point in time, and called Anytime Learning at Macroscale (ALMA). The algorithm processes data through a series of training batches, each of these processing steps being followed to a model evaluation. The total loss is the average (or sum) of the losses computed at each step.

Reviewers agreed that the paper is not ready for acceptance at ICLR 2022 as the presentation of the work lacks of clarity, especially w.r.t. to the similarities with online learning and the learning of streams of data, and the fundamental difference between small or moderate batch sizes and very large batches.
Decision: Reject