Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper proposes a new theoretical framework to study the task-free continual learning problem. The contribution is twofold:
* An extension of the domain adaptation theory allowing for multiple time steps and yielding time-dependent generalization bounds: This analysis explains the forgetting behavior with the discrepancy distance between the memory (source) and targets domains. 
* An algorithm inspired by the theoretical results: The discrepancy distance is used to first design a dynamic expansion mechanism. This distance provides a criterion to detect when to add new components to the model.  It is also used to propose a sample selection procedure that insures probabilistic diversity of the information stored in the memory. 
This algorithm is then tested on different benchmarks and compared to prior task-free continual learning approaches, showing a substantial improvement. 
strengths_and_weaknesses: Strengths: 
* The paper is generally clear and well structured. 
* The take of the authors on the problem is novel to the best of my knowledge. 
* The theoretical analysis proposed in this paper is sound, well motivated and well structured. It is of significance to the community and can shed the light on novel insights for the challenging problem it is focusing on. 
* The algorithm derived from the theory makes sense, and the experiments seem to validate its efficiency. 

Weaknesses: 
* Minor: There is an overload of abbreviation in the paper (e.g. SM for supplementary materials, GB for generalization bounds seems non-necessary), and some inconsistency in the notations (e.g.: The indices in the preliminary (3.1) seem wrong at some places, tasks are sometime indexed by t and sometimes by i. This can make reading the paper quite hard. 
* While the training phases of the expansion model are clear, it is quite unclear how the model is used at inference time. It would be great if this can be made clearer. It seems that the VAE is used to decide which trained component to use. Is this the case? 
* While the experiments are diverse and show an improvement over prior works, it is unclear how the methods compare from the computational (and less importantly the memory) cost point of view. 
questions: * While the theory has been developed for the task-free case, can it be adapted to also account for the case with known task boundaries? 
* It seems in the initial definitions that it is assumed that the data is presented to the model ordered by category. Is this the case? If yes: While this is common in the literature, how important is this assumption to the analysis? Would it be possible to extend the analysis to also hold for the case where the data distribution is shifting in a smoother way (model exposed to a slowly changing mixture of categories for example)? 
* Can the authors comment on how he model deals with a case where knowledge previously seen is encountered again after initiating a new model? Given that the memory is erased when a new component is initiated, wouldn't this lead to some redundancy in the learnt models? Would it be possible to add a phase to recombine the knowledge encoded in the different components of the model? 
* Can the authors comment of the computational cost of their method, and how the model grows over time, as compared to the prior works? 
* The field has long focused on catastrophic forgetting. The analysis in this paper is also built around fighting this behavior. When training on a stream of data, however, one could argue that the present and future performance should have a higher weight. What do the authors think about this? Can their analysis and proposed method be adapted to look into forward transfer rather than preventing the model from forgetting? 
limitations: The authors do not discuss the limitations in the paper. As mentioned above, it would be interesting to consider analysing the computational cost of the proposed method.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: - The authors derive the first generalisation bound for Task Free Continual Learning. The analysis is inspired by domain adaptation theory.
- The authors leverage the discrepancy measure to derive a continual learning algorithm which balances between model complexity, generalisation and catastrophic forgetting.
- In order to do so, the model is designed as a mixture of smaller classifiers. To each classifier is attached a VAE. The VAE is used to flush the memory and model selection. At test time, the generated samples by the VAE are used to compute the discrepancy with the test samples, in order to identify which model to use for inference. 
strengths_and_weaknesses: ## Originality
### 1- Are the tasks or methods new?
The following elements are new : 
- The generalisation bound for Task-Free Continual Learning.
- Leveraging the tools of domain adaptation theory for continual learning.
- The final algorithm ODDL with the model expansion, model selection, sample selection and discrepancy as a primitive.

### 2- Is the work a novel combination of well-known techniques?
- The work builds upon the out of distribution theory, and expands it in meaningful ways to derive an algorithm for Task-free Continual Learning. 

### 3- Is it clear how this work differs from previous contributions?
- The way the contributions of this work stand out is clear.

### 4- Is related work adequately cited ?
- To my knowledge, the related work is adequately cited.

## Quality
### 1- Is the submission technically sound ?
- The submission is technically sound.

### 2- Are claims well supported (e.g., by theoretical analysis or experimental results) ?
- The theoretical results are supported by proofs.
- 

### 3- Are the methods used appropriate ?
- Using the VAE to "summarise" the distribution of a subcategory looks relevant.
- Also, the way discrepancy distance is used in the algorithm is theoretically motivated.

### 4- Is this a complete piece of work or work in progress ?
- It is a complete piece of work.

### 5- Are the authors careful and honest about evaluating both the strengths and weaknesses of their work ?
- The authors benchmarked their method ODDL against several baselines in Table 1.
- The linear growth of the number of models in case of a very large number of tasks, was briefly discussed in the appendix, section G.. 

## Clarity:
### 1- Is the submission clearly written?
- I read the submission a few times, in order to understand the high level picture of the algorithm, and how its components fit.
- One way to improve clarity is maybe by improving the diagram in Figure 2, by making it more verbose ?

### 2- Is it well organised? (If not, please make constructive suggestions for improving its clarity.)
- I suggest presenting a big picture of the algorithm first with the main building blocks at a high level, then presenting each element in detail. While reading through section 4.3, I realised I didn't read section 4.1 and 4.2 through the right angle, so I had to go back and forth between both sections a few times to get the right picture.

### 3- Does it adequately inform the reader? (Note that a superbly written paper provides enough information for an expert reader to reproduce its results.)
- The hyperparameters, code and required resources are shared by the authors.

## Significance:
### 1- Are the results important? 
- The results are important in the following ways : 
   - A theoretically grounded approach for Task-Free Continual Learning.
   - An improvement of 7% of average accuracy against the top baseline, on CIFAR-10 and CIFAR-100.
   - A modular algorithm, which is easy to interpret and debug.

### 2- Are others (researchers or practitioners) likely to use the ideas or build on them?
- Given the improvements over the baselines in the experiments (Table 1), it is likely that practitioners will benchmark this method. However, given that it requires a non-negligible amount of engineering and hyperparameter tuning, it is likely that practitioners will first benchmark simpler to implement methods.
- Given that the way the approach is designed is clearly explained and grounded on theory, it make it more likely that researchers will build upon it.  

### 3- Does it advance the state of the art in a demonstrable way? 
- The results advance the state of the art by 7% of average accuracy over the top baseline for CIFAR-10 and CIFAR-100. (Table 1)
questions: - In the appendix, Table 3, it looks like the average accuracy gain with the method decreases with the complexity or scale of the dataset. Do you have any intuition why the improvement on Split MImageNet is much smaller than for CIFAR-10 and Split-MNIST ? 
- Do you have any indications for practitioners on how to set the threshold lambda ? Specifically in the case of very long streams, where the optimal lambda may vary over time.

A few typos : 
- Replace "CIFRA" by "CIFAR"
- Replace "Memroy" by "Memory" in Figure 2
limitations: The limitations and potential negative societal impact are discussed adequately in the section G of the Appendix.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper proposes a theoretical analysis framework for task free continual learning. The presented framework  gives generalization bounds in terms on discrepancies between distributions. Then, the authors propose an algorithm for task free continual learning that maintains samples in a memory buffer and selects the samples measuring the discrepancy between distributions. 
strengths_and_weaknesses: The paper is well written. There are points that should be included in the paper. For example, explain the relationship of the presented bounds with existing bounds for scenarios with changing distribution over time, compare with more methods designed for task free continual learning, and describe the complexity of the method. I include the questions in the following section.

There are some typos in the paper: 
	- The authors use the acronym for the supplementary material (SM) in line 147 and define the acronym in line 160.
	- Figure 1 appears far from the text.
	- There is a repeated "the" in line 320

questions: In numerical experiments, the proposed algorithms is compared with continual learning methods, is it fair the comparison with methods that are not designed for task free continual learning?

You evaluate the method on datasets where the distribution changes in a different manner. For example, in the Split CIFAR10 dataset each task contains new classes, while in the Permuted MNIST dataset each task assigns a random permutation. For which scenarios would it be more appropriate to use your algorithm? For example, for those where the distribution changes smoothly, scenarios where only the joint distribution of labels and instances changes, scenarios where the conditional distribution changes?

It is common in continual learning to assume that tasks are iid or have a common representation, would it make sense to compare your algorithm with methods designed for supervised classification under concept drift? As well as using datasets with concept drift?

Increasing the complexity of the model would lead to tighter bounds, but what is the model's complexity?

Can the bounds presented be compared with those described in papers designed for scenarios with changing distribution? For instance, “New analysis and algorithm for learning with drifting  distributions.”, “Lifelong learning with non-i.i.d. tasks”, or “Minimax Classification under Concept Drift with Multidimensional Adaptation and Performance Guarantees”.
limitations: Yes
review_title: null

Meta-review: metareview: This paper is well-written. All reviewers agree the paper has solid contribution for task-free continual learning by providing theoretical foundation first, and the developing methods inspired by their theory. Empirical studies by comparing with a number of baseline methods demonstrate the effectiveness of the proposed method. All reviewers' concerns during the discussion phase have been addressed. Hence, I recommend acceptance. 
confidence: Certain
Decision: Accept