Venue: ICLR 2018 conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: The paper proposes to study the influence of ordering in the Curriculum and Self paced learning. The paper is mainly based on empirical justification and observation. The results on 36 data sets show that to some extent the ordering of the training instances in the Curriculum and Self paced learning is not important. The paper involves some interesting ideas and experimental results. I still have some comments.

1.	The empirical results show that different orderings still have different impact for data sets. How to adaptively select an appropriate ordering for given data set?
2.	The empirical results show that some ordering has negative impact. How to avoid the negative impact? This question is not answered in the paper.
3.	The ROGS is still clearly inferior to SPLI. It seems that such an observation does not strongly support the claim that ‘random is good enough’. 

review_title: The paper proposes to study the influence of ordering in the Curriculum and Self paced learning. The paper is mainly based on empirical justification and observation. The results on 36 data sets show that to some extent the ordering of the training instances in the Curriculum and Self paced learning is not important. The paper involves some interesting ideas and experimental results. I still have some comments.

final_score: rating: 4: Ok but not good enough - rejection
review_content: Summary: 
The paper proposes an algorithm to do incremental learning, by successively growing the training set in phases. However as opposed to training using curriculum learning or self paced learning, the authors propose to simply add training samples without any order to their "complexity". The authors claim that their approach, which is called ROGS, is better than the classical method and comparable to curriculum/self paced learning. The experiments are conducted on the UCI dataset with mixed results. 

Review: 
My overall assessment of the paper is that it is extremely weak, both in terms of the novelty of method proposed, its impact, and the results of the experiments. Successively increasing the training set size in an arbitrary order is the first thing that one would try when learning incrementally. Furthermore, the paper does not clearly explain what does it mean by a method to "win" or "lose". Is some training algorithm A a winner over some training algorithm B, if A reaches the same accuracy as B in lesser number of epochs? In such a case, how do we decide on what accuracy is the upper bound. Also, do we tune the hyper-parameters of the model along the way? There are so many variables to account for here, which the paper completely ignores. 

Furthermore, even under the limited set of experiments the authors conducted, the results are highly inconclusive. While the authors test their proposed methodology on 36 UCI datasets, there is no clear indication whether the proposed approach has any superiority over the previous proposed ones, such as, CL and SPLI. 

Given the above weaknesses of the paper i think the impact of this research is extremely marginal. 

The paper is generally well written and easy to understand. There are some minor issues though. For example, I think Assumption (a) is quite strong and may not necessary hold in many cases. 
review_title: No new idea with inconclusive experiments

final_score: rating: 4: Ok but not good enough - rejection
review_content: This paper addresses an interesting problem of curriculum/self-paced versus random order of samples for faster learning. Specifically, the authors argue that adding samples in random order is as beneficial as adding them with some curriculum strategy, i.e. from easiest to hardest, or reverse. 
The main learning strategy considered in this work is learning with growing sets, i.e. at each next stage a new portion of samples is added to the current available training set. At the last stage, all training samples are considered. The classifier is re-learned on each stage, where optimized weights in the previous stage are given as initial weights in the next stage. 

The work has several flaws. 
-First of all, it is not surprising that learning with more training samples at each next stage (growing sets) gets better - this is the basic principle of learning. The question is how fast the current classifier converges to the optimal Bayes level when using Curriculum strategy versus Random strategy. The empirical evaluations do not show evidence/disprove regarding this matter. For example, it could happen that the classifier converges to the optimal on the first stage already, so there is no difference when training in random versus curriculum order with growing sets. 
-Secondly, easyness/hardness of the samples are defined w.r.t. some pre-trained (external) ensemble method. It is not clear how this definition of easiness/hardness translates when training the 3-layer neural network (final classifier). For example, it could well happen that all the samples are equally easy for training the final classifier, so the curriculum order would be the same as random order. In the original work on self-paced learning, Kumar et al (2010), easiness of the samples is re-computed on each stage of the classifier learning. 
-The empirical evaluations are not clear. Just showing the wins across datasets without actual performance is not convincing (Table 2). 
-I wonder whether the section with theoretical explanation is needed. What is the main advantage of learning with growing sets (when re-training the classifier)  and (traditional) learning when using the whole training dataset (last stage, in this work)? 


review_title: This paper is trying to analyse different learning strategies, curriculum learning versus learning in random order, arguing that the latter one can achieve the same competitive performance as the former one.  The proposed approach of learning with growing sets as well as empirical results are not convincing. 

Meta-review: The authors give evidence that is certain cases, the ordering of sample inclusion in a curriculum is not important.  However, the reviewers believe the experiments are inconclusive, both in the sense that as reported, they do not demonstrate the authors' hypothesis, and that they may leave out many relevant factors of variation (such as hyper-parameter tuning). 
Decision: Reject