Venue: ICLR 2017 conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: The authors propose a conceptually simple method for regularisation of recurrent neural networks. The idea is related to dropout, but instead of zeroing out units, they are instead set to their respective values at the preceding time step element-wise with a certain probability.

Overall, the paper is well written. The method is clearly represented up to issues raised by reviewers during the pre-review question phase. The related work is complete and probably the best currently available on the matter of regularising RNNs.

The experimental section focuses on comparing the method with the current SOTA on a set of NLP benchmarks and a synthetic problem. All of the experiments focus on sequences over discrete values. An additional experiment also shows that the sequential Jacobian is far higher for long-term dependencies than in the dropout case.

Overall, the paper bears great potential. However, I do see some points.

1) As raised during the pre-review questions, I would like to see the results of experiments that feature a complete hyper parameter search. I.e. a proper model selection process,as it should be standard in the community. I do not see why this was not done, especially as the author count seems to indicate that the necessary resources are available.

I want to repeat at this point that Table 2 of the paper shows that validation error is not a reliable estimator for testing error in the respective data set. Thus, overfitting the model selection process is a serious concern here.
Zoneout does not seem to improve that much in the other tasks.

2) Zoneout is not investigated well mathematically. E.g. an analysis of the of the form of gradients from unit K at time step T to unit K’ at time step T-R would have been interesting, especially as these are not necessarily non-zero for dropout. Also, the question whether zoneout has a variational interpretation in the spirit of Yarin Gal’s work is an obvious one. I can see that it is if we treat zoneout in a resnet framework and dropout on the incremental parts. Overall, little effort is done answering the question *why* zoneout works well, even though the literature bears plenty of starting points for such analysis.

3) The data sets used are only symbolic. It would have been great if more ground was covered, i.e. continuous data such as from dynamical systems. To me it is not obvious whether it will transfer right away.


An extreme amount of “tricks” is being published currently for improved RNN training. How does zoneout stand out? It is a nice idea, and simple to implement. However, the paper under delivers: the experiments do not convince me (see 1) and 3)). There authors do not provide convincing theoretical insights either. (2)

Consequently, the paper reduces to a “epsilon improvement, great text, mediocre experimental evaluation, little theoretical insight”.

review_title: Incremental improvement, not convincing enough

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: This paper tests zoneout against a variety of datasets - character level, word level, and pMNIST classification - showing applicability in a wide range of scenarios. While zoneout acts as a regularizer to prevent overfitting, it also has similarities to residual connections. The continued analysis of this aspect, including analyzing how the gradient flow improves the given tasks, is of great interest and helps show it as an inherent property of zoneout.

This is a well written paper with a variety of experiments that support the claims. I have also previously used this technique in a recurrent setting and am confident on the positive impact it can have upon tasks. This is likely to become a standard technique used within RNNs across various frameworks.
review_title: Review

final_score: rating: null
review_content: Zoneout is both a regularizer (due to the noise) and a better optimizer (due to the ability to pass gradients back through the identity connections). Can any conclusions be drawn about how much each factor is helping ? Is there a way to tease apart the contributions to the improvements from these two factors ?
review_title: Better Regularization or Optimization ?

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: Paper Summary
This paper proposes a variant of dropout, applicable to RNNs, in which the state
of a unit is randomly retained, as opposed to being set to zero. This provides
noise which gives the regularization effect, but also prevents loss of
information over time, in fact making it easier to send gradients back because
they can flow right through the identity connections without attenuation.
Experiments show that this model works quite well. It is still worse that
variational dropout on Penn Tree bank language modeling task, but given the
simplicity of the idea it is likely to become widely useful.

Strengths
- Simple idea that works well.
- Detailed experiments help understand the effects of the zoneout probabilities
  and validate its applicability to different tasks/domains.

Weaknesses
- Does not beat variational dropout (but maybe better hyper-parameter tuning
  will help).

Quality
The experimental design and writeup is high quality.

Clarity
The paper clear and well written, experimental details seem adequate.

Originality
The proposed idea is novel.

Significance
This paper will be of interest to anyone working with RNNs (which is a large
group of people!).

Minor suggestion-
- As the authors mention - Zoneout has two things working for it - the noise and
  the ability to pass gradients back without decay. It might help to tease apart
the contribution from these two factors. For example, if we use a fixed
mask over the unrolled network (different at each time step) instead of resampling
it again for every training case, it would tell us how much help comes from the
identity connections alone.
review_title: Simple idea, well executed.

final_score: rating: null
review_content: Very nice paper. Simple idea, good results and a strong writeup.


From what I understand from the conclusion, no hyper parameter search was conducted. Instead, the hyper parameters from other works are taken over.

I am undecided whether this is the best for readers. Being pedantic, the hypothesis tested is "does adding zoneout improve things given we perform model selection on a different, although related, task". Consequently, a hyper parameter search performed on zoneout might even lead to worse results.

I would like to ask why the authors chose not to just perform an extensive hyper parameter search and report those results instead. Was it out of convenience for not having to do it, or was it because the authors want to stress this behaviour as a feature of zoneout? If so, what is the authors advice for a reader (e.g. "we conjecture that zoneout will also lead to superior results if hyper parameter search is performed with a zoneout model.")
review_title: What hypothesis do the experiments test?

final_score: rating: null
review_content: I'm glad to see that the setup for the word level PTB task is similar to many other related works, making it easy to compare at a glance. Do you have any intuition as to the different between a zoneout LSTM and the variational LSTM when it comes to perplexity? You mention elsewhere (conclusion?) that you don't perform extensive hyperparameter tuning, which Gal et al. 2015 do for the variational LSTM. Intuitively I prefer zoneout to the variational LSTM as the entire hidden state remains active, essentially retaining more "bandwidth", but it's unfortunate the results don't quite line up there.

I am confused over the training of the character level PTB task too. You state that you train LSTMs, GRUs, and tanh-RNNs on non-overlapping sequences of 100 in batches of 32. In the next sentence however you then state that you train on overlapping sequences.

The investigation into the properties of gradient flow are interesting and help justify the claim that gradient propagation benefits are an inherent property of zoneout. I hope to see these types of analyses become standard when investigating new techniques, especially when they may impact gradient propagation.

Overall this is a good paper with many experiments to support the paper's claims. The introduction of zoneout is a positive contribution to the community - I'm looking forward to seeing it used more extensively and being added as a standard component in various frameworks.
review_title: Clarification re: performance of zoneout vs variational LSTM and char-level PTB

Meta-review: Very nice paper, with simple, intuitive idea that works quite well, solving the problem of how to do recurrent dropout.
 
 Pros:
 - Improved results
 - Very simple method
 
 Cons:
 - Almost the best results (aside from Variational Dropout)
Decision: Accept (Poster)