Venue: ICLR 2017 conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper presents a thorough analysis of different methods to do curriculum learning. The major issue I have with it is that the dataset used seems very specific and does not necessarily justified, as mentioned by AnonReviewer3. It would have been great to see experiments on more standard tasks. Also, I really can't understand how the performance of FFNN models can be so good, please elaborate on this (see last comment).
However, the paper is well written, the comparisons of the described methods are interesting and would probably apply to some other datasets as well.

The paper is way too long (18 pages!). Please reduce it or move some of the results to an appendix section.

The method described is extremely similar to the one described in Reinforcement learning neural turing machines (Zaremba et al., 2016, https://arxiv.org/pdf/1505.00521v3.pdf) where the authors progressively increase the length of training examples until the performance exceeds a given threshold. Maybe you should mention it.

Could you explain very briefly in the paper what "4-connected" and "8-connected" mean, for people not familiar with these terms?

I agree that having gold pen stroke sequences would be nice and probably very good features to have for image classification. But how accurate are the constructed ones? Typically, the example given in figure 1 does not represent the way people write a "3". I'm just concerned about the validity of the proposed dataset and what these sequences really represent (although I agree that it can still be relevant as a sequence learning dataset, even if it does not reflect the way people write).

In figure 5, for the blue curve, I was expecting to see an increase of the error when new data are added to the set, but there doesn't seem to be much correlation between these two phenomenons. Can you explain why? Also, could you explain the important error rate increase at about 7e+07 steps for the regular sequence learning?

The method used to test the H1 hypothesis is interesting, but did you try something even simpler like not using batch (ie batch size of 1 sequence)? This would alleviate this "different number of points by batch" effect and the results would probably very different than in figure 5.

The performance of the FFNN models seem too good compared to the RNN ones. How is this possible? RNN models should perform at least as well. Even the "Incremental sequence learning" RNN barely beats its FFNN equivalent. Do the "dx" and "dy" values always take values in [-1, 0, 1]? If so, the number of possible mappings is very small (from [-1, 0, 1] to [-1, 0, 1]), how could a mapping between two successive points be so accurate without looking at the history? Please clarify on this.
review_title: null

final_score: rating: 5: Marginally below acceptance threshold
review_content: The submitted paper proposes a new way of learning sequence predictors. In the lines of incremental learning and curriculum learning, easier samples are presented first and the complexity is increased during training. The particularity here is that the complexity is defined as the length of the sequences given for training, the premise being is that longer sequences are harder to learn, since they need a more complex internal representation.

The targeted application is sequence prediction from primed prefixes, tested on a single dataset, which the authors extract themselves from MNIST.

The idea in the paper is interesting and worth reading. There are also many interesting aspects of evaluation part, as the authors perform several ablation studies to rule out side-effects of the tests. The proposed learning strategy is compared to other strategies.

However, my biggest concern is still with evaluation. The authors tested the method on a single dataset, which is non standard and derived from MNIST. Given the general nature of the claim, in order to confirm the interest of the proposed algorithm, it need to be tested on other datasets, public datasets, and on a different application.

The paper is too long and should be trimmed significantly.

The transfer learning part (from prediction to classification) is a different story and I do not see a clear connection to the main contribution of the paper.

The presentation and organization of the paper could be improved. It is quite sequentially written and sometimes reads like a student's report.

The loss given in the long unnumbered equation on page 6 should be better explained: provide explanations for each term, and make clearer what the different symbols mean. Learning is supervised, so which variables are predictions, and which are observations from the data (ground truth).

Names in table 2 do not correspond to the descriptions in section 4.

review_title: Interesting idea, long experiments, but only a single non-standard dataset

final_score: rating: null
review_content: 1. Any reason why use such a specific dataset for this problem? Why not also use a standard sequential dataset, such as, for language modeling? In my experience this approach of incremental learning in the language modeling task does not work. You think it could be because of large vocabulary size? 

2. In figure 4, the red lines which correspond to the standard algorithm (without incremental learning), the curve is really jumpy? I never experience such a thing when working with task such as language modeling. Why is that the case? You think it is dataset specific? 
review_title: Choice of the task

final_score: rating: null
review_content: You mention that image to stroke conversion can be done solving a travelling salesman problem. This is only correct if there is a single drawing gesture and the pen does not leave the paper, which seems to be a stringend restriction.
review_title: TSP

final_score: rating: null
review_content: You mention the "Best value for the average over 10 runs".
Can you explain what you calculated here?
review_title: Max of AVG

final_score: rating: 3: Clear rejection
review_content: First up, I want to point out that this paper is really long. Like 17 pages long -- without any supplementary material. While ICLR does not have an official page limit, it would be nice if authors put themselves in the reviewer's shoes and did not take undue advantage of this rule. Having 1 or 2 pages in addition to the conventional 8 page limit is ok, but more than doubling the pages is quite unfair. 

Now for the review: The paper proposes a new artificial dataset for sequence learning. I call it artificial because it was artificially generated from the original MNIST dataset which is a smallish dataset of real images of handwritten digits. In addition to the dataset, the authors propose to train recurrent networks using a schedule over the length of the sequence, which they call "incremental learning". The experiments show that their proposed schedule is better than not having any schedule on this data set. Furthermore, they also show that their proposed schedule is better than a few other intuitive schedules. The authors verify this by doing some ablation studies over the model on the proposed dataset. 

I have following issues with this paper: 

-- I did not find anything novel in this paper. The proposed incremental learning schedule is nothing new and is a natural thing to try when learning sequences. Similar idea have already been tried by a number of authors, including Bengio 2015, and Ranzato 2015. The only new piece of work is the ablation studies which the authors conduct to tease out and verify that indeed the improvement in performance is due to the curriculum used. 

-- Furthermore, the authors only test their hypothesis on a single dataset which they propose and is artificially generated. Why not use it on a real sequential dataset, such as, language modeling. Does the technique not work in that scenario? In fact I am quite positive that for language modeling where the vocabulary size is huge, the performance gains will be no where close to the 74% reported in the paper.

-- I'm not convinced about the value of having this artificial dataset. Already there are so many real world sequential dataset available, including in text, speech, finance and other areas. What exactly does this dataset bring to the table is not super clear to me. While having another dataset may not be a bad thing in itself, I almost felt that this dataset was created for the sole purpose of making the proposed ideas work. It would have been so much better had the authors shown experiments on other datasets. 

-- As I said, the paper is way too long. A significant part of the length of the paper is due to a collection of experiments which are completely un-related to the main message of the paper. For instance, the experiment in Section 6.2 is completely unrelated to the story of the paper. Same is true with the transfer learning experiments of Section 6.4.

review_title: Really long paper with not a lot of impact

Meta-review: This is an empirical paper which compares three different instantiations of a kind of incremental/curriculum learning for sequences.
 
 The reviews from R1 and R3 (which gave confidence scores of 4) were negative. The main concerns addressed by the reviewers:
 * Paper is too long -- 17 pages -- and length is due to experiments (e.g. transfer learning) which are tangential to the main message of the paper (R3, R1) 
 * Lack of novelty (R3)
 * Tests only on single, synthetic, small dataset and questioning the claim that this new synthetic dataset is helpful to the community (R3, R1)
 
 However, R3 and R1 both pointed out that they found the ablation studies interesting. R4 (who gave a confidence score of 3) was gave a more positive score but also expressed similar concerns with R1 & R3 (page length, similarity to existing work, dataset too specific and not necessarily justified).
 
 The author argued for the novelty of the paper, agreed to reduce the paper length and also argued that the data was indeed helpful (giving a specific case of another researcher who was extending the data). The author also provided a "twitter trail" countering the argument that the dataset was created for the sole purpose of showing that the method works.
 
 After engaging the reviewers in discussion, R4 admitted they were originally too generous with their score and downgraded to 5. The AC has decided that, while the paper has merits as acknowledged by the reviewers, it's not strong enough for acceptance in its present form. The AC encourages the author to work on an improved version (perhaps with experiments on an additional real dataset) and organize it with the audience in mind.
Decision: Reject