Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 8: Accept
review_content: The paper proposes a solution for using meta-learning methods without the need of determining the task segmentation a priori. Authors identify that the task segmented setting is very similar to the problem of change-point detection (CPD) and particularly, they connect the generative model of the meta-learning approach to the Bayesian recursive method of Adams and MacKay 2007. The presentation of the meta-learning problem is well done and I understand its importance within the difficulties for modelling new unobserved tasks. The notation and description of the generative model included in the problem statement section is clearly understandable for any reader, this is a very positive point. They demonstrate a deep comprehension of the BOCPD model of Adams, and its extension for the meta-learning approach is original to me. Lastly, the presentation of the MOCA meta-learning algorithm is useful for reproducibility and I see it completely applicable to other scenarios. There is a noticeable effort of describing the solution for both regression and classification problems and the empirical results conclude a positive performance of MOCA.

Overall, I consider that the paper is well-written with thorough explanations. Of course, there are some details that could be improved (I will comment this later). If done, I would be willing to increase my score. The contribution of the paper is significant to meta-learning models and I think it will help to spread the Adams’ model to other type of problems.

There is an important contribution in the paper that I have to mention and it is also relevant for the future application of the Adams model. If one reads the original BOCPD paper, in particular Eq (1), where the predictive posterior p(x_t+1|x_{1:t}) is defined from a marginalisation over the run length values, it is noticeable that this equation is not used in the final recursion of the CPD method. This is because the posterior p(r_t|x_{1:t}) is sufficient for determining if there is a CP on the t time-step or not. So the predictive posterior is never used in practice (in the original paper). When I first read Adams 2007, this detail was clear to me. Surprisingly, I find that the authors have find a practical use of this equation (Eq. (7)) and it is in the main core of the meta-learning algorithm, this is fantastic.

The details I think should be improved are:

- The conditional run-length prior p(r_t|r_{t-1}) barely appears and without any detailed description is not obvious for non-familiar readers with the Bayesian CPD approach.
- The first time I read the manuscript, the use of z_t in the BOCPD presentation made me feel a bit lost. Why not use x or y? At least specify that is a toy variable for the explanation. Later on, authors change again to the x,y notation.
- Equations 3 and 4 are too similar, this looks a bit repetitive. Why not reusing one of them or say the change from one term to another? 
- If reading the literature of the BOCPD and posterior extensions, the likelihood model of the detector is often referred as the underlying predictive model (UPM). Using a similar term would help for orienting familiar readers into the solution.
- As authors should have noted on their experiments, as one makes the run-length higher, the number of parameters \eta[r_t] increases. A clear state on how this is solved would help.
- In the PCOC subsection, I find the definition of y \sim q(y) a bit weird. Using Cat(_) or Multinomial likelihood notation would be a bit better.

review_title: Official Blind Review #2

final_score: rating: 6: Weak Accept
review_content: This paper pushes meta-learning towards task-unsegmented settings. Different from the traditional offline meta-learning phase with explicit task segmentation, MOCA adopts a Bayesian changepoint estimation scheme for task change detection. The setting is novel and deserves research in-depth, and the idea is easy to understand. The proposed method can learn the meta-learning model and changepoint detection model simultaneously. Besides, the MOCA framework is not designed specifically for one algorithm and can be easily combined with other meta-learning models.

However, I got some questions about this paper:
Q1: Is the ‘Hazard’ in the experiment the same to $\lambda$ in eq.1? I think notations should be consistent if they are the same.
Q2: Can other changepoint models be compared in the experiment? I found many in the related work.
Q3: The running times in Figure 2 should be reported to demonstrate the efficiency of MOCA, since the method is proposing online streams, efficiency should be promised for quickly processing.
Q4: What is ‘T’ in Algorithm 1? Are experiment results sensitive to it? Experiments about this should be conducted and reported.

Lastly, I think [1] should be cited as related work about continual learning for proposing task-free continual learning, which is very similar to the setting in this paper.
[1] Rahaf Aljundi, Klaas Kelchtermans, Tinne Tuytelaars. Task-Free Continual Learning. CVPR 2019
review_title: Official Blind Review #1

final_score: rating: 3: Weak Reject
review_content: The paper considers the meta-learning in the task un-segmented setting and apply bayesian online change point detection with meta-learning. The task un-segmented is claimed to exist in real applications and the paper explains the idea in a clear way. 

My major concerns and questions are the following:

1) In Eq(4), it requires a computation of normalization constant that needs to sum over support of r_t. The support gets larger and larger with the length of sequence increases over time. Does this method scale well with long sequence?

2) The part I feel most confused are the experiments.  The paper tries to solve a problem in the task un-segmented setting, but why there is no such setting in experiments that the meta-training set cannot be segmented to different tasks?

3) The Figure 1 is hard to read. What do the red and green points in the left panel mean? The hazard rate is not defined before the experiment which makes Figure 3 also hard to understand. 

4) What is the meta-learning algorithm used in the experiements, MAML, NP or RNN-based methods?  It claims MOCA can be used with any meta-learning algorithms and how does it show in experiments?

5) In Rainbow MNIST, why in high hazard rates, all models perform comparable to “train on everything”? Does it mean meta-learning does not work in this configuration and why? Does “train on everything” includes fine-tuning on the meta-test training set?

6) In Mini-IMAGENET, what does it mean by "we associate each class with a semantic label that is consistent between tasks”? Why it needs to form “super-class”? 

I think the proposed method can be useful in the task un-segmented setting. But before the above-mention questions are solved and the experimental section gets more clear, I will give a conservative rating. 


######################
Post-rebuttal review:
Thanks for the authors' feedback and it resolves some confusion parts in the paper. Though the author claims the reason of experiments setting, I believe it is necessary to have one experiment where task-segmentation is impossible and compare with standard sequential learning methods in that setting. Otherwise it remains a question whether the proposed method works only when the problem basically has task-segmentation.  And if the problem has task-segmentation, why not using traditional meta-learning methods, as shown as oracle methods with better performance in paper? 
Since the major contribution of the paper is providing a meta-learning method to work in the problems where task-segmentation is unavailable, not having an experiment in this setting (withholding segmentation information does not exactly fall in this setting because it adds a condition that the task-segmentation information is originally accessible) is a major reason for my current evaluation.  I recognize the careful design of the MOCA and agree with some positive points raised by other reviewers. I would not be bothered if the paper is accepted while I tend to maintain the current rating because of the above-mentioned concern.

review_title: Official Blind Review #3

Meta-review: In this paper the authors view meta-learning under a general, less studied viewpoint, which does not make the typical assumption that task segmentation is provided. In this context, change-point analysis is used as a tool to complement meta-learning in this expanded domain. 
 
The expansion of meta-learning in this more general and often more practical context is significant and the paper is generally well written. However, considering this particular (non)segmentation setting is not an entirely novel idea; for example the reviewers have already pointed out [1] (which the authors agreed to discuss), but also [2] is another relevant work. The authors are highly encouraged to incorporate results, or at least a discussion, with respect to at least [2]. It seems likely that inferring boundaries could be more powerful, but it is important to better motivate this for a final paper. 
 
Moreover, the paper could be strengthened by significantly expanding the discussion about practical usefulness of the approach. R3 provides a suggestion towards this direction, that is, to explore the performance in a situation where task segmentation is truly unavailable. 
 
[1] Rahaf et el. "Task-Free Continual Learning". 
[2] Riemer et al. "Learning to learn without forgetting by maximizing transfer and minimizing interference".
 

Decision: Reject