Title: Zoneout: Regularizing RNNs by Randomly Preserving Hidden Activations
Authors: David Krueger
Tegan Maharaj
Janos Kramar
Mohammad Pezeshki
Nicolas Ballas
Nan Rosemary Ke
Anirudh  Goyal
Yoshua Bengio
Aaron Courville
Christopher Pal
Venue: ICLR 2017
Keywords: Deep learning

Abstract: We propose zoneout, a novel method for regularizing RNNs.
At each timestep, zoneout stochastically forces some hidden units to maintain their previous values.
Like dropout, zoneout uses random noise to train a pseudo-ensemble, improving generalization.
But by preserving instead of dropping hidden units, gradient information and state information are more readily propagated through time, as in feedforward stochastic depth networks.
We perform an empirical investigation of various RNN regularizers, and find that zoneout gives significant performance improvements across tasks. We achieve competitive results with relatively simple models in character- and word-level language modelling on the Penn Treebank and Text8 datasets, and combining with recurrent batch normalization yields state-of-the-art results on permuted sequential MNIST.

TL;DR: Zoneout is like dropout (for RNNs) but uses identity masks instead of zero masks

Review discussion:
I'm glad to see that the setup for the word level PTB task is similar to many other related works, making it easy to compare at a glance. Do you have any intuition as to the different between a zoneout LSTM and the variational LSTM when it comes to perplexity? You mention elsewhere (conclusion?) that you don't perform extensive hyperparameter tuning, which Gal et al. 2015 do for the variational LSTM. Intuitively I prefer zoneout to the variational LSTM as the entire hidden state remains active, essentially retaining more "bandwidth", but it's unfortunate the results don't quite line up there.
I am confused over the training of the character level PTB task too. You state that you train LSTMs, GRUs, and tanh-RNNs on non-overlapping sequences of 100 in batches of 32. In the next sentence however you then state that you train on overlapping sequences.
The investigation into the properties of gradient flow are interesting and help justify the claim that gradient propagation benefits are an inherent property of zoneout. I hope to see these types of analyses become standard when investigating new techniques, especially when they may impact gradient propagation.
Overall this is a good paper with many experiments to support the paper's claims. The introduction of zoneout is a positive contribution to the community - I'm looking forward to seeing it used more extensively and being added as a standard component in various frameworks.
In addition to variational dropout, (Gal et al. 2015) drop out rows of the embedding layer (i.e. words of the input), and use weight decay.  It is possible this additional regularization helps Gal et al. (2015) achieve superior performance on this task.
Thank you for pointing out the inconsistency in the text wrt (non-)overlapping sequences; we’ll update the paper to clarify.
In fact, we originally ran all models without overlapping sequences.
Then to compare with Cooijmans et al. (2016), we used overlapping sequences as data augmentation for the LSTM experiments (only), which reduced BPC from 1.29 to 1.27.
Zoneout is both a regularizer (due to the noise) and a better optimizer (due to the ability to pass gradients back through the identity connections). Can any conclusions be drawn about how much each factor is helping ? Is there a way to tease apart the contributions to the improvements from these two factors ?
Our focus is the regularization effect of zoneout, which provides consistent improvements in generalization (validation and test set cost) in our experiments.
Our experiment on gradient propagation (section 4.4) does not demonstrate that zoneout improves optimization, only that zoneout (unlike a naive implementation of dropout on recurrent connections) does not suffer from vanishing gradients.
The effect on optimization (i.e. training curve) is mixed: train cost decreases faster on character-level Penn Treebank, but not on other datasets. In general, we mostly observe lower training costs with unregularized models than models with zoneout, as is often the case for regularization - by reduce overfitting, it’s common to see a rise in training cost.
Very nice paper. Simple idea, good results and a strong writeup.
From what I understand from the conclusion, no hyper parameter search was conducted.
Instead, the hyper parameters from other works are taken over.
I am undecided whether this is the best for readers. Being pedantic, the hypothesis tested is "does adding zoneout improve things given we perform model selection on a different, although related, task".
Consequently, a hyper parameter search performed on zoneout might even lead to worse results.
I would like to ask why the authors chose not to just perform an extensive hyper parameter search and report those results instead. Was it out of convenience for not having to do it, or was it because the authors want to stress this behaviour as a feature of zoneout? If so, what is the authors advice for a reader (e.g. "we conjecture that zoneout will also lead to superior results if hyper parameter search is performed with a zoneout model.")
We chose our experiments in order to demonstrate that adding zoneout to existing models improves performance without requiring additional tuning. We believe that using these references architectures without any hyper-parameter provides a strong and fair comparison of zoneout with previous approaches (on the same task).
We see no reason to suspect that a hyper-parameter search over model architectures would lead to worse results for zoneout (except in the under-fitting regime).
Rather, tuning the architecture in conjunction with the zoneout probabilities could only improve our results.
Could you please clarify what you mean by this sentence:
“Was it out of convenience for not having to do it, or was it because the authors want to stress this behaviour as a feature of zoneout?”
Based on our experiments, our advice to readers who want to reduce overfitting in their RNN models is to use zoneout probability ~0.5 on cells and/or ~0.1 on states.
"We see no reason to suspect that a hyper-parameter search over model architectures would lead to worse results for zoneout; we presume that tuning the architecture in conjunction with the zoneout probabilities could only improve our results. Could you clarify why you think this would lead to worse results?"
Zoneout has arguably a much less regularizing effect than just doing dropout: information is not completely forgotten, instead the transition to the next step is perturbed. Table 2 also indicates that the validation error is not a good estimate of the testing error.
Consequently, there is a chance that a certain setting of hyper parameters will lead to lower validation loss but higher testing loss. Maybe one of the other reviewers is experienced enough with the data sets to comment.
Nevertheless, the hypothesis "zoneout leads to lower test error when model selection is done with zoneout models" is not tested, although this is arguably of the highest interest to the reader.
“Was it out of convenience for not having to do it, or was it because the authors want to stress this behaviour as a feature of zoneout?”
Put more simply: why did the authors not run a full model selection for zoneout?
This paper tests zoneout against a variety of datasets - character level, word level, and pMNIST classification - showing applicability in a wide range of scenarios.
While zoneout acts as a regularizer to prevent overfitting, it also has similarities to residual connections. The continued analysis of this aspect, including analyzing how the gradient flow improves the given tasks, is of great interest and helps show it as an inherent property of zoneout.
This is a well written paper with a variety of experiments that support the claims. I have also previously used this technique in a recurrent setting and am confident on the positive impact it can have upon tasks. This is likely to become a standard technique used within RNNs across various frameworks.
Thank you for your review, and great to hear that zoneout has helped in your experiments!
Paper Summary
This paper proposes a variant of dropout, applicable to RNNs, in which the state
of a unit is randomly retained, as opposed to being set to zero. This provides
noise which gives the regularization effect, but also prevents loss of
information over time, in fact making it easier to send gradients back because
they can flow right through the identity connections without attenuation.
Experiments show that this model works quite well. It is still worse that
variational dropout on Penn Tree bank language modeling task, but given the
simplicity of the idea it is likely to become widely useful.
Strengths
- Simple idea that works well.
- Detailed experiments help understand the effects of the zoneout probabilities
and validate its applicability to different tasks/domains.
Weaknesses
- Does not beat variational dropout (but maybe better hyper-parameter tuning
will help).
Quality
The experimental design and writeup is high quality.
Clarity
The paper clear and well written, experimental details seem adequate.
Originality
The proposed idea is novel.
Significance
This paper will be of interest to anyone working with RNNs (which is a large
group of people!).
Minor suggestion-
- As the authors mention - Zoneout has two things working for it - the noise and
the ability to pass gradients back without decay. It might help to tease apart
the contribution from these two factors.
For example, if we use a fixed
mask over the unrolled network (different at each time step) instead of resampling
it again for every training case, it would tell us how much help comes from the
identity connections alone.
Thank you for your detailed review and thoughtful suggestions.
Zoneout has two things working for it - the noise and
the ability to pass gradients back without decay. It might help to tease apart
the contribution from these two factors.
For example, if we use a fixed
mask over the unrolled network (different at each time step) instead of resampling
it again for every training case, it would tell us how much help comes from the
identity connections alone.
This seemed like a pretty cool experiment; we have run it and added it in the Appendix, Figure 7. We're grateful for this interesting contribution to the paper!
The authors propose a conceptually simple method for regularisation of recurrent neural networks. The idea is related to dropout, but instead of zeroing out units, they are instead set to their respective values at the preceding time step element-wise with a certain probability.
Overall, the paper is well written. The method is clearly represented up to issues raised by reviewers during the pre-review question phase. The related work is complete and probably the best currently available on the matter of regularising RNNs.
The experimental section focuses on comparing the method with the current SOTA on a set of NLP benchmarks and a synthetic problem. All of the experiments focus on sequences over discrete values. An additional experiment also shows that the sequential Jacobian is far higher for long-term dependencies than in the dropout case.
Overall, the paper bears great potential.
However, I do see some points.
1) As raised during the pre-review questions, I would like to see the results of experiments that feature a complete hyper parameter search. I.e. a proper model selection process,as it should be standard in the community. I do not see why this was not done, especially as the author count seems to indicate that the necessary resources are available.
I want to repeat at this point that Table 2 of the paper shows that validation error is not a reliable estimator for testing error in the respective data set.
Thus, overfitting the model selection process is a serious concern here.
Zoneout does not seem to improve that much in the other tasks.
2) Zoneout is not investigated well mathematically. E.g. an analysis of the of the form of gradients from unit K at time step T to unit K’ at time step T-R would have been interesting, especially as these are not necessarily non-zero for dropout.
Also, the question whether zoneout has a variational interpretation in the spirit of Yarin Gal’s work is an obvious one. I can see that it is if we treat zoneout in a resnet framework and dropout on the incremental parts.
Overall, little effort is done answering the question *why* zoneout works well, even though the literature bears plenty of starting points for such analysis.
3) The data sets used are only symbolic. It would have been great if more ground was covered, i.e. continuous data such as from dynamical systems. To me it is not obvious whether it will transfer right away.
An extreme amount of “tricks” is being published currently for improved RNN training. How does zoneout stand out? It is a nice idea, and simple to implement.
However, the paper under delivers: the experiments do not convince me (see 1) and 3)). There authors do not provide convincing theoretical insights either. (2)
Consequently, the paper reduces to a “epsilon improvement, great text, mediocre experimental evaluation, little theoretical insight”.
Thank you for your thoughtful and detailed review.  Your comments raise some important concerns which we address individually below.
Q:
I want to repeat at this point that Table 2 of the paper shows that validation error is not a reliable estimator for testing error in the respective data set.
Thus, overfitting the model selection process is a serious concern here.
A:
Thank you for pointing out this issue. Our initial table was not clear or representative of our results. We have reformatted the table to clarify both that our test-validation gaps are very consistent within our experiments, and that zoneout does achieve the best validation performance within our experiments. These experiments include a search over regularization techniques and strengths, as detailed below.  We believe this addresses your concerns about model selection.
(To fit everything in one table (Table 1), we had included a mix of results from other papers and from our own experiments. Semeniuta et al.’s recurrent dropout got worse validation performance than zoneout in our experiments, but we then reported results from their paper. This was confusing, and has been corrected.)
We have also corrected the validation and test values of recurrent dropout on Text8. They were incorrectly reported, and we apologize for any confusion caused by this error.
For the purposes of early stopping and model selection, we measured validation BPC on sequences of length 100 as in training (as opposed to computing it over the entire sequence, which is what we do at test time). This is why we have a larger gap between validation and test BPC than previous works.
In order to update Table 1, we reran the best performing recurrent dropout (p=0.25) and zoneout (p_cells=0.5 and p_hiddens = 0.05) models *with* sequence overlap.  In doing so, we also noted that neither of these best-performing models had entirely overfit, so we ran both for longer.  Our implementation of recurrent dropout achieves better performance (1.286 BPC test) than reported by Semeniuta et al. (2016), and zoneout achieves near state of the art 1.252 BPC.
Q:
1) As raised during the pre-review questions, I would like to see the results of experiments that feature a complete hyper parameter search. I.e. a proper model selection process,as it should be standard in the community. I do not see why this was not done, especially as the author count seems to indicate that the necessary resources are available.
A:
We had previously run a suite of experiments searching over regularization strength for zoneout, recurrent dropout, and recurrent stochastic depth, but without sequence overlap. We show this search for zoneout in Figure 3, and report the best results achieved with all models in Table 1, and as noted, zoneout achieves the lowest validation score.
Given our updates to Table 1, if you feel that additional hyperparameter search is necessary, please clarify which hyperparameters you think are most relevant to compare.  We can probably run 20-60 experiments within a few days given our available computing resources.
Q:
Zoneout does not seem to improve that much in the other tasks.
A:
We respectfully disagree. Zoneout achieves a consistent improvement of more than 4%, relative to a vanilla LSTM  baseline, on all datasets (cf Table 1 and Table 2),  just by plug-and-play with existing models.
Q:
3) The data sets used are only symbolic. It would have been great if more ground was covered, i.e. continuous data such as from dynamical systems. To me it is not obvious whether it will transfer right away.
A:
These datasets are standard benchmarks for RNNs and allow us to compare to previously published results. Permuted sequential MNIST does take continuous pixel values as input, and we don’t see any reason to believe that zoneout would not improve performance on tasks with continuous outputs.
Q:
2) Zoneout is not investigated well mathematically.
A:
We propose a novel technique and perform a thorough empirical evaluation to demonstrate the benefit of our proposal. We investigate gradient propagation and relate zoneout to other techniques and theory (pseudoensembles, stochastic depth, residual networks, etc.).
While it would be great to see further work and theoretically grounded analysis, we think that our technique and results are already of interest to the community.
Thanks for the improvements. I will adapt my rating.

Meta-review:
role: PC
data: comment: Very nice paper, with simple, intuitive idea that works quite well, solving the problem of how to do recurrent dropout.
 
 Pros:
 - Improved results
 - Very simple method
 
 Cons:
 - Almost the best results (aside from Variational Dropout)

Decision: Accept (Poster)