Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper investigates the problem of minimizing prequential description lengths for image datasets with neural networks, primarily dealing with the issue of computational overhead. The proposed method (Mini-batch Incremental Training with Replay Streams, MI/RS) is inspired by  continual learning and its effectiveness is demonstrated empirically across traditional image classification datasets and neural network architectures.
strength_and_weaknesses: It makes an encouraging attempt to bridge the gap between theory and practice for minimum description length.
clarity,_quality,_novelty_and_reproducibility: This paper is well written and clearly presented. 

The focus of the paper is investigating the trade-off between minimizing prequential description lengths and computational overhead. Several heuristic methods are compared empirically compared in the experiment section. 

However, the significance of minimizing prequential description lengths is not sufficiently clear, which is merely discussed in the introduction section with a toy case. In this sense, it is difficult to evaluate the paper in a broader view. 

The code should be made public to validate reproducibility owing to the fact that implementation details are missing in the text.
summary_of_the_review: See “Strength And Weaknesses” and “Clarity, Quality, Novelty And Reproducibility”.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes two new techniques: *forward-calibration* and *replay streams* for efficiently computing prequential description lengths. It particularly focuses on the computational costs associated with these methods and claims to show that they improve upon previously reported results by large margins.
strength_and_weaknesses: **Strengths:**
- The experiments compare several methods across multiple datasets and networks.
- The paper does multiple ablation studies to better understand different components of the algorithm, although this study is relegated to the appendix.

**Weaknesses:**
1. The idea of replay streams has been well studied in the Reinforcement Learning literature but has not been cited in this work.
2. The organization of the paper can be improved. Equations are introduced without defining all of the notations. I would recommend adding a preliminaries section where notations and prerequisites are first defined.
3. The MDL principle and prequential description lengths are properly introduced only in the appendix. I would recommend a short version of this be introduced in the main text.
4. Under $Chunk Incremental/Continual Fine-tuning (CI/CF)$, a claim is made "However, recent research suggests that..." - this statement does not have a reference. Also, following it, the authors state that they run an ablation study but do not state where this ablation study is, nor the results of it.
5. Under *mini-batch incremental/replay streams* they make an assumption "We instead assume that the data is stored in its original order in permananet storage and we maintain K replay streams..." - this is an extremely strong assumption and cannot hope to hold in practice. The authors emphasize that this framework is for the setting where the i.i.d assumption fails, typically in streaming settings and therefore, I feel this assumption cannot hold in practice.
6. Interpreration of Results:
    - I am not sure what to take away from Table 1. It seems to me like ERM outperforms MI/RS in most settings which makes sense since these settings are artificially created from the i.i.d setting.
    - In Figure 1, the scale of the loss as well as errors is different for ERM compared to Prequential MDL training. THerefore, I am unsure how to compare these two. It could be argued that the difference in the plots is purely due to scale since the absolutely differences is what we compare.
clarity,_quality,_novelty_and_reproducibility: **Clarity, Quality, Novelty:**
- I think the writing of the paper can be improved significantly in terms of succintness and clarity. The results are hard to interpret and notations are not clearly described.
- The work utilizes ideas from related fields in a novel manner.
summary_of_the_review: ~The paper in its current state is hard to read and it is not clear that the improvements are significant. Therefore, I cannot recommend acceptance. However, if changes are made and my questions are clarified, I would be willing to change my mind.~

Since the authors have improved their presentation and clarified several of my concerns, I am increasing my score to 6.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This work extensively investigates continuous learning based methods for prequential minimum description length. 

More precisely, the authors train MDL models using online learning with rehearsal, and propose two techniques for improving the results. 
- Forward-calibration, to optimize a calibration parameter $\beta$ by using each new batch of samples for evaluation, then single gradient step on $\beta$, then training the other parameters $\theta$, before being placed in a replay buffer. 
- Replay streams, to avoid implementing large replay buffers. Replay streams rely on the data stored in its original order, and maintain K pointers to some positions in the ordered data. When the k-th replay stream is called, it yield the corresponding sample and increments the pointer by one. The authors also propose a mechanism to randomly reset the pointers depending on what's needed.

The authors extensively evaluate their method compared to classical prequential MDL (models trained from scratch or fine-tuned on increasingly larger chunks of data) and compare to ERM on various datasets and neural network architectures (without learning rate schedule). They show their method to perform reasonably well against ERM given the same computational budget and better than previous baselines for prequential MDL, and perform some ablations on techniques such as forward calibration, label smoothing and weight standardization.
strength_and_weaknesses: Strength:
- Finding alternatives to ERM, whose central hypothesis is that data is i.i.d. is an important problem to apply machine learning in many interesting settings. This work investigates a potential candidate, namely prequential MDL.
- The paper is very well-written and motivated.
- The experimental protocol seems thorough: many different architectures and datasets are considered, and ablations are done.
- Experimental results are promising: the proposed method performs better than other MDL baselines, and rival with ERM without learning schedule.

Weakness:
- It seems to me that some baselines are missing: it could be worth evaluating the test accuracy for MDL baselines on ImageNet, and mentioning the results of ERM with a learning schedule (I am surprised by the discrepancy of ERM results with/without learning schedule).
- Although MNIST, CIFAR and ImageNet are iid, wouldn't it be possible to make it non-iid, train on a smaller number of epochs and see how your method performs vs. ERM? I think it may be a more appropriate setting to use continual learning.

Minor:
- Typo: negaive (last line before Figure 4).
clarity,_quality,_novelty_and_reproducibility: Clarity:

The paper is well-written and self-contained, with nice introduction to MDL in Appendix with some caveats:

- The cumulative loss is lower with the proposed method but it is not clear to me how this is the right metric to evaluate the model: if the dataset is a classification dataset, what matters is rather the test accuracy? If so, the model does less good than ERM and one of the two MDL baselines. Could you clarify this?

Quality:
- The experimental protocol is thorough, with potentially missing baselines (see above).

Novelty:
- To the best of my knowledge, applying and extensively evaluating continual learning techniques to Prequential MDL is new, as well as the proposed techniques.

Reproducibility:
- No code is provided but the Appendix contain many experimental details.
summary_of_the_review: This work investigates an interesting alternative to ERM. It is well-written and the experimental protocol is thorough, although some baseline or experiments may be in my opinion missing to convince that the proposed method is very convincing. I wouldn't recommend acceptance for now but would increase my score if my concerns (add baselines or explain why they are not appropriate, meaning of the cumulative loss w.r.t. test accuracy) are answered.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper studies the problem of training deep neural networks with the minimum description length objective. The paper connects the MDL objective training to the ERM objective training but under the settings of continual learning. The paper conducts extensive experiments and compares several methods to achieve the best MDL performance with FLOP cost constraints.
strength_and_weaknesses: Strengths:
1. The paper studies the problem of MDL objective training. This setting addresses many problems in practice where assumptions for ERM objectives are not valid.

2. The paper draws the connection between MDL objective training and ERM objective training under continual learning settings. 

3. The paper conducts extensive experiments and compares several methods training under the MDL objective. The paper also proposes a practical method to achieve approximate-random in rehearsal training. 

4. The paper gives a nice overview of MDL in the appendix.


Weaknesses:
1. The novelty and algorithmic contributions of the paper are limited. The proposed/studied methods are all existing approaches or simple modifications of existing ones. The paper also lacks theoretical justifications.

2. As mentioned in the paper, the experiments do not account for the learning rate annealing, which is a widely used technique for ERM training and significantly affects the final performance.

clarity,_quality,_novelty_and_reproducibility: Clarity:
I really like the writing and the presentation of the paper. The Pareto front figures are very clear and informative. I also have a few suggestions, and please check the minor comments below.

Quality:
The paper shows extensive experimental results on different datasets using various network architectures. The empirical results are of high quality and could be very useful to future research.

Novelty:
As discussed in the strengths and weaknesses part, the novelty of the paper is limited. The experimental results are convincing but non-surprising. Overall I think it's useful empirical work for MDL objective training, which could be critical for applying machine learning models to more practical scenarios.

Reproducibility:
The paper contains many details about the experiment settings and the reproducibility seems solid.


Minor Comments:
1. "Continuous learning" and "continual learning" are both used, and I think they correspond to the same thing. "Continual learning" is probably a more widely used term.

2. The regret is not precisely defined. Given multiple forms of regret are commonly used in online learning literature, I suggest the authors provide a precise definition of the regret used in the paper.
summary_of_the_review: The paper studies a practically very important problem, conducts extensive experiments, and provides convincing arguments to understand the experimental observations. The paper's presentation is very clear. The paper is weak on the methodology and novelty side, but still very useful given its strong and solid empirical results.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper presents two methods, (1) forward calibration and (2) replay streams, for computing prequential description lengths in continuous learning-based methods.
 
Main strength: 
+ Thorough experimental validation. Promising results. The paper shows results better than other MDL baselines.
+ Clearly written paper.

Weakness:
- Novelty and algorithmic contributions are limited. These methods are well-studied in the Reinforcement Learning literature but not cited.
 
The paper initially received borderline scores (including two marginally above the acceptance threshold and two marginally above the acceptance threshold). After the authors’ responses, the reviewers’ concerns are largely alleviated. 
- “… largely clarified by questions and therefore, I am increasing my score” (Reviewer zftM)
- “Overall I think the paper has some limitations but overall provides many useful empirical results.” (Reviewer zGnF)
- “Thank your for the clarifications, I think they make the submission stronger. I will raise my score since my own concerns were answered” (Reviewer LsM9)
 
While the paper was initially rated as borderline, the reviewers are now all leaning positively toward this work. The AC thus recommends accepting the paper.

justification_for_why_not_higher_score: The paper was initially rated as borderline, given the limited novelty and technical contributions. The authors' responses clarified some of the concerns of the reviewers. The reviewers are now leaning toward accepting the work. The AC, however, does not see even stronger support from the reviewers to justify a higher score.
justification_for_why_not_lower_score: The reviewers increased their scores after the authors' responses.
Decision: Accept: poster