Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: Paper summary:
This paper makes the observation that a curriculum need not depend on the difficulty of examples, as most (maybe all) prior works do. They suggest instead a curriculum based on learning one class at a time, starting with one and masking the label of all others as 'unknown' (i.e. treating them as negative examples), and unmasking classes as learning progresses. This is the "incremental labels" part. They make another observation, that label smoothing is applied to all examples regardless of difficulty, and propose an alternative "adaptive" version where labels are smoothed only for difficult examples. This is the "adaptive compensation" part.

Paper contributions: 
- the two observations described above are both interesting, and methods addressing these seem like good ideas in light of the observations
 - the explanation of the method and experiments are very clear.
 - ablation and exploration studies are well-done to further investigate the proposed methods

Review summary & decision: 
I like the core of this paper a lot, and recommend acceptance. It makes some insightful observations, reasonable suggestions, explains things clearly, and does reasonable experiments. The observations and proposed methods are both valuable contributions to the field. If the related work and clarity of abstract/intro are improved, along with addressing false claims and some other relatively minor things, I think this could be a very good paper, and I would be happy to increase my score.

Reasons for decision:
 - Interesting observations are made and the approaches taken are interesting and well-motivated.
 - The paper overall is well structured, easy to understand, and thorough in the experiments.
 - Some related work (detailed below) in other fields and just about curriculum learning seems to be missing. Very strange claim (which is false, as far as I know) that curriculum learning has only been used in shallow networks emphasizes that the related work is lacking. 
 - The abstract and intro summary of contributions didn't do a very good job of conveying what the methods do, although they are clearly explained elsewhere (see suggestions below)
 - A lot of time is spent on detail of experiments and long, clear explanations (this is great), but makes it read a bit like a lab report. Figure 2 and the "properties of LILAC" section are nice, but could be improved with more discussion and reference to other works/fields to provide insight about _why_ LILAC works, not just _that_ it works.
 - Results seem very incremental to me (improvements over other methods are in the decimal places), and I think many people will criticize or dismiss the methods on that basis. I hesitated about this, and in the end decided that the interesting observations are the most valuable part of the paper, not pushing SOTA (while of course it's valuable to report numbers, I think our field should focus less on SOTA numbers in general). 
 - Misleading results are presented for CIFAR-10; ShakeDrop is not SOTA, and no reason is provided for why the particular citations in that table are there (there are many others that could be...). I don't mind that you don't get SOTA, but I mind being misled. If I've misunderstood something here, please clarify! :)

Feedback/suggestions/nits (not necessarily part of decision assessment): 
1. Briefly review some continual learning work; incremental labels are very similar to open set learning; this is worth mentioning. Would also be nice to mention anomaly detection here. 
2. Briefly review negative mining (cases where this improves learning e.g. hard negative mining in text). 
3. More discussion of motivation and why you think that LILAC works would be nice; connections to the above-mentioned fields could help. 
4. Space could be made for the things I suggest here by decreasing spacing in the "main contributions" bullets and reducing the size of Figure 1 (the coloured tiles are very large and there's a lot of unnecessary white space. The size of text is mostly good, although the legend could also be decreased in size) 
5. Incremental labels are well-explained in the dedicated section, but not until then. In the abstract, intro, and first section on "LILAC", I didn't really understand what was going on. I would suggest not using the words mask/unmask; this term is overloaded and gives the wrong connotation to me (that you don't use the predictions for the masked out classes). If you can't find a different term, I'd suggest explaining more clearly what it means to "mask" in this context in the abstract. It would also be good in the abstract to have a sentence motivating your approach (after "... learning difficult samples"). The next sentence sort of tries to do this "... starting point from which" but I think this is really unclear and makes it sound like you're learning an init. 
6. Adaptive compensation is well explained in the abstract, but very confusingly in the intro summary of contributions. 
7. I think it would be clearer to call it "adaptive smoothing". I know LILAC is a nice name, but so is LILAS. 
8. Cite first sentence in abstract or reword
 9. "in this work, we propose.... which introduces" propose/introduce is repetitive
10. "to the best of our knowledge," [insert "all" here]
11. "curriculum learning approaches have only been tested using shallow networks" This is just false as far as I know; apart from the misleading presentation of CIFAR-10 results, addressing this is probably the most important suggestion I have. e.g. here's a whole thesis on doing CL with deep convnets that I found from a few minutes googling: http://www.diva-portal.org/smash/get/diva2:878140/FULLTEXT01.pdf  
12. LILACto -> LILAC to
13. inconsistent use of italics vs. bold for emphasis in the text. Typically use italics; bold is for keywords and subtitles. 
14. Mention tsne hyperparameters in an appendix 
15. Conclusion is more of a summary; the first couple sentences are good but the repeating of the explanation of both methods could be moved to the abstract and/or intro (these sentences are much clearer than the ones used there), making more room to discuss future work and connections to other fields. 
16. CIFAR-10 table should list method names, not just citations (they're referred to by name in the caption, which makes it hard to read the table)

 Questions:
1. What other explorative experiments could you do to investigate properties of IL and AC?
2. Did I misunderstand something about the CIFAR-10 SOTA presentation (is there some reason for the choice of numbers cited here?)
review_title: Official Blind Review #3

final_score: rating: 1: Reject
review_content: ---- Paper summary ----
This paper proposes a curriculum learning approach for classification. The proposed curriculum consists of two phases:
(1) a “label introduction phase”, in which the model is able to see and learn to classify only subset of labels (the model still trains on the samples belonging to the “unseen” classes, but their label is now set to a default class). The subset of seen labels is expanded incrementally, until the entire label set is observed. 
(2) an “adaptive compensation phase”, where the model trains on all labels, but the targets for each class are replaced from 1-hot vectors to smooth version. This only applies to the classes on which the model has made mistakes in a previous training round.
This method is tested on 3 image classification datasets, and a single neural network architecture is tested per dataset (either ResNet18 or ResNet34).

---- Overall opinion ----
While the ideas introduced in this paper may have merit, I believe the experimental evidence is quite limited. Based on the results shown in the paper, I am not convinced that this approach is better than the baselines it compares to. Moreover, since the claim is that this approach is a general curriculum learning method, I find the setting it was tested on very limited (3 datasets, 1 model per dataset), especially since there are no theoretical results to complement the empirical evidence. Finally, the method introduces several parameters (b, m, E, T, eta) that are treated is a somewhat hand-wavy manner, without a proper analysis on the effect of such parameters and how one should set them. Details on this, and other major issues, can be found below. For these reasons, I believe the paper in its current form is not yet ready for publication.

---- Major issues ----
1. The paper simply mentions that the unseen labels are set to a default label, which Figure 1 implies (and is not otherwise clarified) is one of the labels in the dataset. I am not sure intuitively why it makes sense to force the model in the beginning to map a sample’s inputs to another label from the dataset, which in the end it has to learn is wrong. Doesn’t it make more sense to map the unseen labels to a new, fictional label? If not, then how do you decide which of the M labels to choose as the fake label?

2. The method introduces several hyperparameters, such as b (number of visible labels in the beginning), m (number of labels to reveal at each step), E (number of epochs in each incremental phase), T, epsilon. The specific numbers used in the experiments are reported, but it is not clear how these were chosen, and how one would choose them for a new dataset or model.

3. In Table 1, the LILAC results are bolded with a caption saying that bold they means “the best case scenario”, and the main text also claims that “, LILAC is the only one to consistently increase classification accuracy and decrease the standard deviation across all datasets compared to batch learning”. However, from Table 1 it seems LILAC has neither the highest accuracy (label smoothing has overall higher accuracy), neither the lowest std. It may seem that the authors arbitrarily decided what is the best accuracy/std tradeoff which makes their method seem the best. Please define clearly the criteria for establishing the best method, and explain in what setting this criteria is a valid choice.

4. Aside from the issue mentioned above, the differences in accuracy or std in Table 1 seem minor (e.g., within 0.10% on Cifar10, and within 1% in the others). Please provide more evidence that these differences are significant.

5. How was the consistency in table 1 decided? Please provide the specific metric.

6. Regarding the choice of models and datasets, the method was only tested on image datasets. This is could be enough as a contribution, but in this case the introduction and abstract should not claim a generality that has not been tested. Similarly, the paper only considers 1 model per dataset. Does this work for other models too? 

7. From Table 3, it looks like the LILAC w/o AC is actually worse than the baseline. In this case, what is the benefit of having the label introduction phase? Why not just have the AC component alone? If there is a reason, please include the results for AC alone in Table 3.

8. I find the comparison in Figure 2 potentially misleading, because I believe 1 epoch of batch training is not equal in terms of amount of training as 1 full span of the incremental phase. I believe these embeddings should be shown at convergence time.

9. In Table 4 and accompanying text, the authors conclude that “LILAC is relatively invariant to the order of the label introduction”. However, to me both random and ascending order seem actually random with respect to how this order is used. I advise the authors to try other orderings too, such as sorting them by the error of an initial training of a classifier, or other more difficulty-based orders.

---- Minor issues ----
1. In Algorithm 1, there is a missing equation (“Eqn. ??”), also I’m not sure why the first line says “Write here the result”.

2. “small batch sizes help achieve more generalizable solutions, but do not scale as well to vast computational resources as large mini-batches” → this is a bit confusing. How do large mini-batches scale better, and what is the difference between “small batch sizes” and “large mini-batches”? 

---- Questions ----
Please see the major issues questions above.
review_title: Official Blind Review #2

final_score: rating: 3: Weak Reject
review_content: 1. Summary:

This paper proposes a novel direction for curriculum learning.  Previous works in the area of curriculum learning focused on choosing easier samples first and harder samples later when learning the neural network models.  This is problematic since we need to first compute how difficult each samples are, which introduces computational overheads.  In this work, the paper propose to gradually learn with a class-wise perspective instead.  The neural network has only access to the labels of certain classes (chosen randomly) in the beginning, and the samples that belong to the rest of the classes are treated as unseen samples but with a label forced into the last class.  Then, the true labels of unseen classes are gradually revealed, and this is repeated until in the final incremental step, all labels are revealed.  The method further has an adaptive compensation step, which use a less peaked distribution label for supervision only for the incorrectly predicted samples.  The experiments show that with only the first step, the proposed method is worse than the original batch learning, but by adding the second label smoothing step, there is improvement over the original batch learning setup.


2. Decision:

Weak Reject -- The class-wise idea for curriculum learning is interesting but the motivation and intuition behind the design of the proposed method is weak.


3. Reasons for decision:

Pros:

The class-wise idea used in this paper seems to add a new direction to the area of curriculum learning.  The experiments are well designed, with ablation study to understand the behavior of the proposed method in depth.

Cons:

The motivation behind the ideas of the paper and the design of the procedure was not so clear.  Why would it be beneficial to start learning from a few classes in the beginning and then gradually expanding the class labels?  I was also curious if this will be more beneficial with a dataset with more classes (maybe not so different by looking at the comparison of CIFAR10 and 100 according to Table 1).

If my understanding was correct, the masked labels from unseen samples are all forced into the last class.  Does this design cause new issues, since the last class will have a lot of label noise until the final incremental step?  If yes, is it possible to consider other ways to add a fake label?   A naive way might be: for unseen samples, attach a fake soft label that has uniform probability over masked classes (M-b) and zero probability for the seen classes (in b).

In the experiments, it seems that the proposed method has consistency but no other baselines have consistency.  However, if you just look at the best mean accuracy, a naive label smoothing is often better than the proposed method.

4. Additional comments not related to the decision of the paper:

In Algorithm 1 in appendix A, the equation link is lost: “Eqn. ??”

In appendix B, what is the decay rate for LR milestones?



After response:

Thank you for answering my questions and for the comments.  Even though towards the end of the learning stage, the true labels are given for all samples, I feel these noisy steps go against the spirit of curriculum learning.  Starting from small classes seems to be a good curriculum and is intuitive, but the noisy part seems to be adding a bad curriculum.  This paper can be much stronger if this part is solved.
review_title: Official Blind Review #1

Meta-review: While the reviewers appreciated the ideas presented in the paper and their novelty, there were major concerns raised about the experimental evaluation. Due to the serious doubts that the reviewers raised about the effectiveness of the proposed approach, I do not think that the paper is quite ready for publication at this time, though I would encourage the authors to revise and resubmit the work at the next opportunity.
Decision: Reject