Title: Incremental Sequence Learning
Authors: Edwin D. de Jong
Venue: ICLR 2017
Keywords: Deep learning
Supervised Learning

Abstract: Deep learning research over the past years has shown that by increasing the scope or difficulty of the learning problem over time, increasingly complex learning problems can be addressed. We study incremental learning in the context of sequence learning, using generative RNNs in the form of multi-layer recurrent Mixture Density Networks. While the potential of incremental or curriculum learning to enhance learning is known, indiscriminate application of the principle does not necessarily lead to improvement, and it is essential therefore to know which forms of incremental or curriculum learning have a positive effect. This research contributes to that aim by comparing three instantiations of incremental or curriculum learning.

We introduce Incremental Sequence Learning, a simple incremental approach to sequence learning. Incremental Sequence Learning starts out by using only the first few steps of each sequence as training data. Each time a performance criterion has been reached, the length of the parts of the sequences used for training is increased.

We introduce and make available a novel sequence learning task and data set: predicting and classifying MNIST pen stroke sequences. We find that Incremental Sequence Learning greatly speeds up sequence learning and reaches the best test performance level of regular sequence learning 20 times faster, reduces the test error by 74%, and in general performs more robustly; it displays lower variance and achieves sustained progress after all three comparison methods have stopped improving. The other instantiations of curriculum learning do not result in any noticeable improvement. A trained sequence prediction model is also used in transfer learning to the task of sequence classification, where it is found that transfer learning realizes improved classification performance compared to methods that learn to classify from scratch.


TL;DR: We investigate a technique for sequence learning where the initial parts of the sequences are learned first; this is found to not only greatly speed up learning, but moreover to strongly improve generalization performance.

Review discussion:
Updated version:
- added a section with generative results, including movies showing what the network has learned over the course of training
- used the ICLR style file
- Clarified the contribution (see abstract, intro, and conclusion)
- Added figures to illustrate the architecture of the network and the difference between training and generation
- Adapted the selection of experiments in Section 6.4 (more logical selection of experimental settings)
- Some textual edits
Article length: the current version of the article aims to provide a description of the work that is as clear and complete as possible.
As a result the number of pages is higher than requested for final versions (this is in line with the ICLR submission policy, which permits submitting longer articles). If accepted, a condensed version will be produced that satisfies the ICLR recommended guidelines ("8 pages, plus 1 page for the references and as many pages as needed in an appendix section").
1. Any reason why use such a specific dataset for this problem? Why not also use a standard sequential dataset, such as, for language modeling? In my experience this approach of incremental learning in the language modeling task does not work. You think it could be because of large vocabulary size?
2. In figure 4, the red lines which correspond to the standard algorithm (without incremental learning), the curve is really jumpy? I never experience such a thing when working with task such as language modeling. Why is that the case? You think it is dataset specific?
Hi,
Thanks, nice questions! Here my responses:
1. There is no particular reason for selecting this specific data set; the method could have been tried on any sequence learning data set, and evaluating it on additional problems will be a good next step. As noted in the article, this work is inspired by Alex Graves's article on sequence generation (https://arxiv.org/abs/1308.0850). There, the IAM online handwriting database is used, but I had read that using this data set requires requesting permission first (actually, when checking the IAM page just now, it looks like registering is sufficient). To enable other researchers to easily reproduce the experiments, I wanted to use a data set that can be directly downloaded.
Given the need to identify a sequence learning data set, two considerations were:
- The MNIST digits data set is by far the most well known data set in machine learning; almost anyone who knows about ML will be familiar with it.
- To represent letters or digits, sequences are a very natural representation; much more so than bitmaps. I try to express that in this blog post:
https://edwin-de-jong.github.io/blog/mnist-sequence-data/
I find representing the MNIST digits as sequences interesting since a more natural representation for digits should in principle benefit the classification of the sequences. (To get the most out of this benefit, it would be better to have actual data of the original pen movements that created the digits; as far as I know, the pen movements that produces the MNIST digits were not captured, so that data is unavailable).
Since a dataset of of MNIST digits represented as stroke sequences was not available yet, I created one as a preliminary step for this research:
https://github.com/edwin-de-jong/mnist-digits-as-stroke-sequences/wiki/MNIST-digits-as-stroke-sequences-(code)
On whether this works in language modeling: I would expect that whenever forming a compact representation of the preceding part of the sequence is required for adequate prediction of the next sequence element, the approach may bring benefit, as developing this internal state for shorter prefixes is quicker and easier to learn, and must be learned before longer prefixes can be handled;
but the performance on other data sets such as language will need to be evaluated in experiments to say something more concrete about this, and this would be a great topic for follow-up research.
2. Yes, indeed the learning process sometimes goes off in directions that lead to higher errors, even when using regular sequence learning. What I found in earlier experiments is that using two layers of LSTM units makes learning a bit more unstable, but tends to produce better results in the end, hence that is what I used in the experiments;
when using an RNN with only one layer, the results were more stable.
Hope this clarifies; please let me know in case you have further questions or any of the above is unclear.
Best regards,
Edwin
You mention that image to stroke conversion can be done solving a travelling salesman problem. This is only correct if there is a single drawing gesture and the pen does not leave the paper, which seems to be a stringend restriction.
The Traveling Salesman approach also works for digits consisting of multiple strokes; it is used here to find a path visiting _all_ the points.
When two subsequent points in the sequence are more than 1 pixel apart, this implies a new stroke is starting, i.e. the pen leaves the paper. In the resulting sequence data this is encoded as EOS=1.
An example can be seen in the sequence for the number '4' here:
https://edwin-de-jong.github.io/blog/mnist-sequence-data/
You mention the "Best value for the average over 10 runs".
Can you explain what you calculated here?
Hi,
Yes, the caption of the test set error table specifies: "Best value for the average over 10 runs of the test set error". The way this is calculated is:
For each run, the test error over the complete test set is reported at different points during the run. This yields, for each run, the test set error over time.
Next, this is averaged of the ten runs;
so we have the average test error over the course of a run.
Finally, the lowest value for this average test error is determined; this is the value reported in the table. Since different runs achieve their test error at different points in time, individual runs can obtain a lower test error; to obtain a robust statistic of performance however, we report these averaged values.
This paper presents a thorough analysis of different methods to do curriculum learning. The major issue I have with it is that the dataset used seems very specific and does not necessarily justified, as mentioned by AnonReviewer3. It would have been great to see experiments on more standard tasks.
Also, I really can't understand how the performance of FFNN models can be so good, please elaborate on this (see last comment).
However, the paper is well written, the comparisons of the described methods are interesting and would probably apply to some other datasets as well.
The paper is way too long (18 pages!). Please reduce it or move some of the results to an appendix section.
The method described is extremely similar to the one described in Reinforcement learning neural turing machines (Zaremba et al., 2016, https://arxiv.org/pdf/1505.00521v3.pdf) where the authors progressively increase the length of training examples until the performance exceeds a given threshold. Maybe you should mention it.
Could you explain very briefly in the paper what "4-connected" and "8-connected" mean, for people not familiar with these terms?
I agree that having gold pen stroke sequences would be nice and probably very good features to have for image classification.
But how accurate are the constructed ones? Typically, the example given in figure 1 does not represent the way people write a "3". I'm just concerned about the validity of the proposed dataset and what these sequences really represent (although I agree that it can still be relevant as a sequence learning dataset, even if it does not reflect the way people write).
In figure 5, for the blue curve, I was expecting to see an increase of the error when new data are added to the set, but there doesn't seem to be much correlation between these two phenomenons. Can you explain why?
Also, could you explain the important error rate increase at about 7e+07 steps for the regular sequence learning?
The method used to test the H1 hypothesis is interesting, but did you try something even simpler like not using batch (ie batch size of 1 sequence)? This would alleviate this "different number of points by batch" effect and the results would probably very different than in figure 5.
The performance of the FFNN models seem too good compared to the RNN ones. How is this possible? RNN models should perform at least as well. Even the "Incremental sequence learning" RNN barely beats its FFNN equivalent. Do the "dx" and "dy" values always take values in [-1, 0, 1]? If so, the number of possible mappings is very small (from [-1, 0, 1] to [-1, 0, 1]), how could a mapping between two successive points be so accurate without looking at the history? Please clarify on this.
Dataset: please see Graves '13 on which this research was based; it uses the IAM handwriting data set. As noted below, I had read that using this data set requires requesting permission first.
Therefore I took it on me to create a similar data set and make it publicly available, so that other researchers can easily check and replicate the research.
FFNN performance: great question, I was also surprised by this. At an earlier stage I therefore did some exploratory analysis, and it appears that there is a substantial part of the initial error that can be reduced by FFNN, which makes sense; examples of what FFNNs can learn here is the right approximate scale (means and standard deviations) of the distibutions, but also that e.g. after a stroke to the right, a stroke to the left (opposite direction) is unlikely.
When you look at the later experiments though, it can be seen that RNNs can improve a lot further beyond the level achieved by the FFNNs.
Length: Fully agree, I will make sure that the final version (if accepted) respects the ICLR page limits, where supplementary material can move to the appendix.
Zaremba et al., 2016: Thanks for this relevant reference, certainly worth mentioning, I'll add a citation.
However, the approach there clearly differs in that:
- the length of input sequences is not varied; it is the maximal length of the _desired output_ to _typical inputs_ that is varied, which is different.
- the task family and method family are both different. We are concerned with sequence _prediction_, where the next element is to be predicted given the current one, and an RNN is used. The article you mention is concerned with a set of tasks involving copying sequences that have already been received, and uses reinforcement learning rather than sequence learning. Very interesting, but different.
4-connected: yes, will add an explanation; this simply means we look at either the four directly neighboring pixels (North, East, South, West), or all 8 neighboring pixels (including NE, SE, SW, NW).
"Accuracy" of constructed sequences: there is no claim or aim that the sequences use the same strokes as the humans drawing the digits. The aim is to obtain an efficient representation that reconstruct the thinned MNIST images; the dataset succeeds in providing this.
Fig 5: For the black line, where the increases all happen around the start of the run, this is clearly visible, see the large increases and decreases in the solid black line at the start. I think part of the reason this is less visible for the other methods is that the jumps are spread over the entire run; bear in mind the results (also the jumps) are averaged over 10 runs.
Error increase around 7e7: as noted below the RNNs are sometimes unstable, due to using 2 layers. One interesting finding is that the Incremental Sequence Learning approach clearly improves the stability. Apparently it goes to a different region of the weight / state space; I have no explanation so far for why this improvement occurs.
The method used to test the H1 hypothesis is interesting, but did you try something even simpler like not using batch (ie batch size of 1 sequence)? This would alleviate this "different number of points by batch" effect and the results would probably very different than in figure 5.
H1: A batch of 1 sequence would not be comparable; the regular method would then train on all (e.g. 40) points of the sequences, whereas Incremental Sequence Learning would initially train only 2 points, which means the amount of computation and training per batch is not comparable. Or do you mean using a single point of a single sequence per batch? Theoretically a good idea, but I expect that would be impossibly slow, as the amount of overhead in creating and handling batches is then greatly increased compared to the amount of training per batch.
FFNN vs RNN: I think the relatively good performance of FFNNs is because FFNNs are easier to train. The absolute performance of FFNNs is not spectacular however; note that there are still several points of error at the end of the run. As you note the offsets in the data are mostly ( [ -1 | 0 | 1 ], [ -1 | 0 | 1] ), but note that the loss function during training is based on the predicted probability distribution (more specifically on the resulting likelihood of the observed true offsets);
so always predicting (0,0) with a high probability would not work. The reported error however, as described in the article, is based on the RMSE, and the challenge (see figure 9) is in getting the error substantially below e.g. 2. That takes an order of magnitude more CPU cycles; as the chart shows, this performance level is passed around 1e8.
The submitted paper proposes a new way of learning sequence predictors. In the lines of incremental learning and curriculum learning, easier samples are presented first and the complexity is increased during training. The particularity here is that the complexity is defined as the length of the sequences given for training, the premise being is that longer sequences are harder to learn, since they need a more complex internal representation.
The targeted application is sequence prediction from primed prefixes, tested on a single dataset, which the authors extract themselves from MNIST.
The idea in the paper is interesting and worth reading. There are also many interesting aspects of evaluation part, as the authors perform several ablation studies to rule out side-effects of the tests. The proposed learning strategy is compared to other strategies.
However, my biggest concern is still with evaluation. The authors tested the method on a single dataset, which is non standard and derived from MNIST. Given the general nature of the claim, in order to confirm the interest of the proposed algorithm, it need to be tested on other datasets, public datasets, and on a different application.
The paper is too long and should be trimmed significantly.
The transfer learning part (from prediction to classification) is a different story and I do not see a clear connection to the main contribution of the paper.
The presentation and organization of the paper could be improved. It is quite sequentially written and sometimes reads like a student's report.
The loss given in the long unnumbered equation on page 6 should be better explained: provide explanations for each term, and make clearer what the different symbols mean. Learning is supervised, so which variables are predictions, and which are observations from the data (ground truth).
Names in table 2 do not correspond to the descriptions in section 4.
Testing on a second application / data set: I agree that that is a good idea, and will try to find time for this. Since the task of language modeling was proposed by several of the reviewers, I plan to use that.
Length: agreed, I will resolve this.
Transfer learning: indeed that's a bonus, performed and presented out of interest, but not critical to the main contribution. I will remove it from the main body of the article, which will also improve the organization.
Loss: ok, will clarify.
Names in table 2: will fix.
Thanks for your helpful suggestions.
First up, I want to point out that this paper is really long.
Like 17 pages long -- without any supplementary material.
While ICLR does not have an official page limit, it would be nice if authors put themselves in the reviewer's shoes and did not take undue advantage of this rule. Having 1 or 2 pages in addition to the conventional 8 page limit is ok, but more than doubling the pages is quite unfair.
Now for the review: The paper proposes a new artificial dataset for sequence learning. I call it artificial because it was artificially generated from the original MNIST dataset which is a smallish dataset of real images of handwritten digits.
In addition to the dataset, the authors propose to train recurrent networks using a schedule over the length of the sequence, which they call "incremental learning". The experiments show that their proposed schedule is better than not having any schedule on this data set.
Furthermore, they also show that their proposed schedule is better than a few other intuitive schedules. The authors verify this by doing some ablation studies over the model on the proposed dataset.
I have following issues with this paper:
-- I did not find anything novel in this paper. The proposed incremental learning schedule is nothing new and is a natural thing to try when learning sequences. Similar idea have already been tried by a number of authors, including Bengio 2015, and Ranzato 2015. The only new piece of work is the ablation studies which the authors conduct to tease out and verify that indeed the improvement in performance is due to the curriculum used.
-- Furthermore, the authors only test their hypothesis on a single dataset which they propose and is artificially generated. Why not use it on a real sequential dataset, such as, language modeling. Does the technique not work in that scenario?
In fact I am quite positive that for language modeling where the vocabulary size is huge, the performance gains will be no where close to the 74% reported in the paper.
-- I'm not convinced about the value of having this artificial dataset. Already there are so many real world sequential dataset available, including in text, speech, finance and other areas. What exactly does this dataset bring to the table is not super clear to me.
While having another dataset may not be a bad thing in itself, I almost felt that this dataset was created for the sole purpose of making the proposed ideas work. It would have been so much better had the authors shown experiments on other datasets.
-- As I said, the paper is way too long. A significant part of the length of the paper is due to a collection of experiments which are completely un-related to the main message of the paper.
For instance, the experiment in Section 6.2 is completely unrelated to the story of the paper. Same is true with the transfer learning experiments of Section 6.4.
Dear Reviewer3,
Thank you for looking into this article. It appears to me that there is a misunderstanding about the contribution. The current version of the article clearly describes the contribution (this explicit formulation was not included in the original submission, please consult the current version therefore). From the abstract:
"While the potential of incremental or curriculum learning to enhance learning is known, indiscriminate application of the principle does not necessarily lead to improvement, and it is essential therefore to know {\em which forms} of incremental or curriculum learning have a positive effect. This research contributes to that aim by comparing three instantiations of incremental or curriculum learning."
This contribution is novel. If you disagree, could you please describe where this contribution has been made earlier? You mention the following two references: Bengio 2015, and Ranzato 2015. To be clear, I assume you are referring to:
1) Bengio, Samy and Vinyals, Oriol and Jaitly, Navdeep and Shazeer, Noam, NIPS 2015. Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks.
2) Marc’Aurelio Ranzato, Sumit Chopra, Michael Auli, Wojciech Zaremba, ICLR 2016. Sequence level training with recurrent neural networks
If so: both articles are about making increasing use of the model's own predictions during training;
that is an interesting and important idea, but unrelated to the method proposed in the current article.
The first article is already cited, and is about switching between training on output vs target data, which is unrelated to varying sequence length. The second article varies the part of the sequences for which cross-entropy loss is used.
That is very different from using only a small part of the sequences for training. Would you agree? If not, could you please elaborate?
Responses to your points:
Length: my apologies, indeed the original submission is too long. Just to clarify how this came about: I wrote the article before I had ICLR in mind, and wrote it to be as clear and complete a description of the research as possible. Given that it became an ICLR submission, I will reduce the length of the main body to 8 pages, and use the appendix for the remaining parts. That will not help to reduce the time you invested in reading this as a reviewer, but it will help for all future readers.
Artificial data set: yes, the data set was created using an automated transformation procedure of the MNIST data. It aims to provide an efficient representation of the essence of the digit shapes (the thresholded and thinned images). It does not claim to reconstruct the strokes drawn by the human writers. The value of this data set is that it represents the digit shapes in an efficient way, and therefore provides a good basis for learning to generate (and classify) the digits.
Moreover, I made this data set and the code to create it publicly available so that any future researchers interested in sequence learning (or in reproducing the results in the article) can freely make use of it. At NIPS I already met a researcher who found this useful, and is working on creating a variant of the data set.
Novelty: The idea of curriculum learning is known, but there are many possible ways to translate this idea to the context of sequence learning; how would you select _which particular form_ to use?
The core contribution of the article is that it investigates which of several possible forms of curriculum learning work well in the context of sequence learning, and demonstrates that one particular such form (for reasons that are both explained and analyzed) performs particularly well. The latest version of the article clearly defines this contribution (see the abstract and conclusion), but the original submission did not; was this review perhaps based on that earlier version? The data set, though useful in its own right, is only a side contribution.
This is a novel, useful and important contribution that can help to improve sequence learning research and applications; a researcher who is new to sequence learning currently has no source that compares or recommends this approach, yet it clearly outperforms the other options in the experiments reported in this article, and therefore deserves further use and analysis.
In particular, it will be valuable to perform experiments on additional problems, to determine to what extent this result extends to other sequence learning domains and problems.
Evaluating on an additional data set, such as language modeling: good idea, I agree this is a useful and important next step, and I'm happy to do so (have not tried this so far; these were the first experiments with the proposed approach). I will see if I can find time to perform additional experiments before the discussion phase ends (Jan 20).
Choice of the data set: the setup of the experiments is based on Graves '13, as noted in the article. The most striking results in that article are handwritten character sequence prediction. To stay close to these original experiments, the aim was to also apply the method to sequences representing handwritten characters. I had read that using the IAM data set requires requesting permission first.
Therefore I decided to create a similar data set and make it publicly available, so that other researchers can easily check and replicate the research.
So, to summarize: the motivation for creating this data set was simply to obtain a publicly available data set that is comparable to the learning problem in the above article.
Another reason why this new data set is valuable is that it provides a more efficient representation of the essence (thresholded, thinned images) of the MNIST images.
"I almost felt that this dataset was created for the sole purpose of making the proposed ideas work"
I can assure you (and prove) that this is not the case. I created the data set first in order to be able to start doing RNN sequence learning experiments, and before I came up with the Incremental Sequence Learning idea. Once the data set was created, I made the data and the code to produce it publicly available on Sept 16:
https://twitter.com/EdwinDdeJong/status/777093726829174784
I then performed many experiments with RNNs on this data, and found the reported method to work surprisingly well. The first experiments with this were completed in October (submitted to a NIPS workshop, and accepted for an oral presentation there). I completed the current longer writeup only in November:
https://twitter.com/EdwinDdeJong/status/797013075274694656
Length and focus: I agree, and as noted above, I will split the article into a main part that fits within the ICLR page limit (8 pages), and place remaining materials in the appendix.
Two reviewers had questions about which problems the approach is expected to work for. I did not find the time to do further experiments, but I did in the mean time think about when the approach can be expected to confer benefit.
When there are long range dependencies where it is necessary to learn relations with a substantial part of the history, it is to be expected that gradually growing the length of the history over which relations are learned can bring benefit.
As noted, I am willing to reduce the paper to the suggested 8 pages, and use an appendix for remaining material.

Meta-review:
role: PC
data: comment: This is an empirical paper which compares three different instantiations of a kind of incremental/curriculum learning for sequences.
 
 The reviews from R1 and R3 (which gave confidence scores of 4) were negative. The main concerns addressed by the reviewers:
 * Paper is too long -- 17 pages -- and length is due to experiments (e.g. transfer learning) which are tangential to the main message of the paper (R3, R1) 
 * Lack of novelty (R3)
 * Tests only on single, synthetic, small dataset and questioning the claim that this new synthetic dataset is helpful to the community (R3, R1)
 
 However, R3 and R1 both pointed out that they found the ablation studies interesting. R4 (who gave a confidence score of 3) was gave a more positive score but also expressed similar concerns with R1 & R3 (page length, similarity to existing work, dataset too specific and not necessarily justified).
 
 The author argued for the novelty of the paper, agreed to reduce the paper length and also argued that the data was indeed helpful (giving a specific case of another researcher who was extending the data). The author also provided a "twitter trail" countering the argument that the dataset was created for the sole purpose of showing that the method works.
 
 After engaging the reviewers in discussion, R4 admitted they were originally too generous with their score and downgraded to 5. The AC has decided that, while the paper has merits as acknowledged by the reviewers, it's not strong enough for acceptance in its present form. The AC encourages the author to work on an improved version (perhaps with experiments on an additional real dataset) and organize it with the audience in mind.

Decision: Reject