{
  "name" : "394.pdf",
  "metadata" : {
    "source" : "CRF",
    "title" : "ZONEOUT: REGULARIZING RNNS BY RANDOMLY PRESERVING HIDDEN ACTIVATIONS",
    "authors" : [ "David Krueger", "Tegan Maharaj", "János Kramár", "Mohammad Pezeshki", "Nicolas Ballas", "Nan Rosemary Ke", "Anirudh Goyal", "Yoshua Bengio", "Aaron Courville", "Christopher Pal" ],
    "emails" : [ "firstname.lastname@umontreal.ca.", " firstname.lastname@polymtl.ca." ], 
    "sections" : [ {
      "heading" : "1 INTRODUCTION",
      "text" : "Regularizing neural nets can significantly improve performance, as indicated by the widespread use of early stopping, and success of regularization methods such as dropout and its recurrent variants (Hinton et al.,  2012; Srivastava et al.,  2014; Zaremba et al.,  2014; Gal, 2015).  In this paper, we address the issue of regularization in recurrent neural networks (RNNs) with a novel method called zoneout.\nRNNs sequentially construct fixed-length representations of arbitrary-length sequences by folding new observations into their hidden state using an input-dependent transition operator.  The repeated application of the same transition operator at the different time steps of the sequence, however, can make the dynamics of an RNN sensitive to minor perturbations in the hidden state; the transition dynamics can magnify components of these perturbations exponentially.  Zoneout aims to improve RNNs’ robustness to perturbations in the hidden state in order to regularize transition dynamics.\nLike dropout, zoneout injects noise during training.  But instead of setting some units’ activations to 0 as in dropout, zoneout randomly replaces some units’ activations with their activations from the previous timestep.  As in dropout, we use the expectation of the random noise at test time.  This results in a simple regularization approach which can be applied through time for any RNN architecture, and can be conceptually extended to any model whose state varies over time.\nCompared with dropout, zoneout is appealing because it preserves information flow forwards and backwards through the network.  This helps combat the vanishing gradient problem (Hochreiter, 1991; Bengio et al.,  1994), as we observe experimentally.\nWe also empirically evaluate zoneout on classification using the permuted sequential MNIST dataset, and on language modelling using the Penn Treebank and Text8 datasets, demonstrating competitive or state of the art performance across tasks.  In particular, we show that zoneout performs competitively with other proposed regularization methods for RNNs, including recently-proposed dropout variants.  Code for replicating all experiments can be found at: http://github.com/teganmaharaj/zoneout"
    }, {
      "heading" : "2 RELATED WORK",
      "text" : ""
    }, {
      "heading" : "2.1 RELATIONSHIP TO DROPOUT",
      "text" : ""
    }, {
      "heading" : "2.2 DROPOUT IN RNNS",
      "text" : "Initially successful applications of dropout in RNNs (Pham et al.,  2013; Zaremba et al.,  2014) only applied dropout to feed-forward connections (“up the stack”), and not recurrent connections (“forward through time”), but several recent works (Semeniuta et al.,  2016; Moon et al.,  2015; Gal, 2015) propose methods that are not limited in this way.  Bayer et al. ( 2013) successfully apply fast dropout (Wang & Manning, 2013), a deterministic approximation of dropout, to RNNs.\nSemeniuta et al. ( 2016) apply recurrent dropout to the updates to LSTM memory cells (or GRU states), i.e. they drop out the input/update gate in LSTM/GRU.  Like zoneout, their approach prevents the loss of long-term memories built up in the states/cells of GRUs/LSTMS, but zoneout does this by preserving units’ activations exactly.  This difference is most salient when zoning out the hidden states (not the memory cells) of an LSTM, for which there is no analogue in recurrent dropout.  Whereas saturated output gates or output nonlinearities would cause recurrent dropout to suffer from vanishing gradients (Bengio et al.,  1994), zoned-out units still propagate gradients effectively in this situation.  Furthermore, while the recurrent dropout method is specific to LSTMs and GRUs, zoneout generalizes to any model that sequentially builds distributed representations of its input, including vanilla RNNs.\nAlso motivated by preventing memory loss, Moon et al. ( 2015) propose rnnDrop.  This technique amounts to using the same dropout mask at every timestep, which the authors show results in improved performance on speech recognition in their experiments.  Semeniuta et al. ( 2016) show, however, that past states’ influence vanishes exponentially as a function of dropout probability when taking the expectation at test time in rnnDrop; this is problematic for tasks involving longer-term dependencies.\nGal (2015) propose another technique which uses the same mask at each timestep.  Motivated by variational inference, they drop out the rows of weight matrices in the input and output embeddings and LSTM gates, instead of dropping units’ activations.  The proposed variational RNN technique achieves single-model state-of-the-art test perplexity of 73.4 on word-level language modelling of Penn Treebank." 
    }, {
      "heading" : "2.3 RELATIONSHIP TO STOCHASTIC DEPTH",
      "text" : "Zoneout can also be viewed as a per-unit version of stochastic depth (Huang et al.,  2016), which randomly drops entire layers of feed-forward residual networks (ResNets (He et al.,  2015)).  This is\nequivalent to zoning out all of the units of a layer at the same time.  In a typical RNN, there is a new input at each timestep, causing issues for a naive implementation of stochastic depth.  Zoning out an entire layer in an RNN means the input at the corresponding timestep is completely ignored, whereas zoning out individual units allows the RNN to take each element of its input sequence into account.  We also found that using residual connections in recurrent nets led to instability, presumably due to the parameter sharing in RNNs.  Concurrent with our work, Singh et al. ( 2016) propose zoneout for ResNets, calling it SkipForward.  In their experiments, zoneout is outperformed by stochastic depth, dropout, and their proposed Swapout technique, which randomly drops either or both of the identity or residual connections.  Unlike Singh et al. ( 2016), we apply zoneout to RNNs, and find it outperforms stochastic depth and recurrent dropout." 
    }, {
      "heading" : "2.4 SELECTIVELY UPDATING HIDDEN UNITS",
      "text" : "Like zoneout, clockwork RNNs (Koutnik et al.,  2014) and hierarchical RNNs (Hihi & Bengio, 1996) update only some units’ activations at every timestep, but their updates are periodic, whereas zoneout’s are stochastic.  Inspired by clockwork RNNs, we experimented with zoneout variants that target different update rates or schedules for different units, but did not find any performance benefit.  Hierarchical multiscale LSTMs (Chung et al.,  2016) learn update probabilities for different units using the straight-through estimator (Bengio et al.,  2013; Courbariaux et al.,  2015), and combined with recently-proposed Layer Normalization (Ba et al.,  2016), achieve competitive results on a variety of tasks.  As the authors note, their method can be interpreted as an input-dependent form of adaptive zoneout.\nIn recent work, Ha et al. ( 2016) use a hypernetwork to dynamically rescale the row-weights of a primary LSTM network, achieving state-of-the-art 1.21 BPC on character-level Penn Treebank when combined with layer normalization (Ba et al.,  2016) in a two-layer network.  This scaling can be viewed as an adaptive, differentiable version of the variational LSTM (Gal, 2015), and could similarly be used to create an adaptive, differentiable version of zoneout.  Very recent work conditions zoneout probabilities on suprisal (a measure of the discrepancy between the predicted and actual state), and sets a new state of the art on enwik8 (Rocki et al.,  2016)." 
    }, {
      "heading" : "3 ZONEOUT AND PRELIMINARIES",
      "text" : "We now explain zoneout in full detail, and compare with other forms of dropout in RNNs.  We start by reviewing recurrent neural networks (RNNs)." 
    }, {
      "heading" : "3.1 RECURRENT NEURAL NETWORKS",
      "text" : "Recurrent neural networks process data x1, x2, . . . ,  xT sequentially, constructing a corresponding sequence of representations, h1, h2, . . . ,  hT .  Each hidden state is trained (implicitly) to remember and emphasize all task-relevant aspects of the preceding inputs, and to incorporate new inputs via a transition operator, T , which converts the present hidden state and input into a new hidden state: ht = T (ht−1, xt).  Zoneout modifies these dynamics by mixing the original transition operator T̃ with the identity operator (as opposed to the null operator used in dropout), according to a vector of Bernoulli masks, dt:\nZoneout: T = dt T̃ + (1− dt) 1 Dropout: T = dt T̃ + (1− dt) 0"
    }, {
      "heading" : "3.2 LONG SHORT-TERM MEMORY",
      "text" : "In long short-term memory RNNs (LSTMs) (Hochreiter & Schmidhuber, 1997), the hidden state is divided into memory cell ct, intended for internal long-term storage, and hidden state ht, used as a transient representation of state at timestep t. In the most widely used formulation of an LSTM (Gers et al.,  2000), ct and ht are computed via a set of four “gates”, including the forget gate, ft, which directly connects ct to the memories of the previous timestep ct−1, via an element-wise multiplication.  Large values of the forget gate cause the cell to remember most (not all) of its previous value.  The other gates control the flow of information in (it, gt) and out (ot) of the cell.  Each gate has a weight matrix and bias vector; for example the forget gate has Wxf , Whf , and bf .  For brevity, we will write these as Wx,Wh, b.\nAn LSTM is defined as follows:\nit, ft, ot = σ(Wxxt +Whht−1 + b)\ngt = tanh(Wxgxt +Whght−1 + bg)\nct = ft ct−1 + it gt ht = ot tanh(ct)\nA naive application of dropout in LSTMs would zero-mask either or both of the memory cells and hidden states, without changing the computation of the gates (i, f, o, g).  Dropping memory cells, for example, changes the computation of ct as follows:\nct = dt (ft ct−1 + it gt)\nAlternatives abound, however; masks can be applied to any subset of the gates, cells, and states.  Semeniuta et al. ( 2016), for instance, zero-mask the input gate:\nct = (ft ct−1 + dt it gt)\nWhen the input gate is masked like this, there is no additive contribution from the input or hidden state, and the value of the memory cell simply decays according to the forget gate.\nIn zoneout, the values of the hidden state and memory cell randomly either maintain their previous value or are updated as usual.  This introduces stochastic identity connections between subsequent time steps:\nct = d c t ct−1 + (1− dct) ( ft ct−1 + it gt ) ht = d h t ht−1 + (1− dht ) ( ot tanh ( ft ct−1 + it gt\n)) We usually use different zoneout masks for cells and hiddens.  We also experiment with a variant of recurrent dropout that reuses the input dropout mask to zoneout the corresponding output gates:\nct = (ft ct−1 + dt it gt) ht = ((1− dt) ot + dt ot−1) tanh(ct)\nThe motivation for this variant is to prevent the network from being forced (by the output gate) to expose a memory cell which has not been updated, and hence may contain misleading information." 
    }, {
      "heading" : "4 EXPERIMENTS AND DISCUSSION",
      "text" : "We evaluate zoneout’s performance on the following tasks: (1) Character-level language modelling on the Penn Treebank corpus (Marcus et al.,  1993); (2) Word-level language modelling on the Penn Treebank corpus (Marcus et al.,  1993); (3) Character-level language modelling on the Text8 corpus (Mahoney, 2011); (4) Classification of hand-written digits on permuted sequential MNIST (pMNIST) (Le et al.,  2015).  We also investigate the gradient flow to past hidden states, using pMNIST." 
    }, {
      "heading" : "4.1 PENN TREEBANK LANGUAGE MODELLING DATASET",
      "text" : "The Penn Treebank language model corpus contains 1 million words.  The model is trained to predict the next word (evaluated on perplexity) or character (evaluated on BPC: bits per character) in a sequence.  1"
    }, {
      "heading" : "4.1.1 CHARACTER-LEVEL",
      "text" : "For the character-level task, we train networks with one layer of 1000 hidden units.  We train LSTMs with a learning rate of 0.002 on overlapping sequences of 100 in batches of 32, optimize using Adam, and clip gradients with threshold 1.  These settings match those used in Cooijmans et al. ( 2016).  We also train GRUs and tanh-RNNs with the same parameters as above, except sequences are nonoverlapping and we use learning rates of 0.001, and 0.0003 for GRUs and tanh-RNNs respectively.  Small values (0.1, 0.05) of zoneout significantly improve generalization performance for all three models.  Intriguingly, we find zoneout increases training time for GRU and tanh-RNN, but decreases training time for LSTMs.\nWe focus our investigation on LSTM units, where the dynamics of zoning out states, cells, or both provide interesting insight into zoneout’s behaviour.  Figure 3 shows our exploration of zoneout in LSTMs, for various zoneout probabilities of cells and/or hiddens.  Zoneout on cells with probability 0.5 or zoneout on states with probability 0.05 both outperform the best-performing recurrent dropout (p = 0.25).  Combining zc = 0.5 and zh = 0.05 leads to our best-performing model, which achieves 1.27 BPC, competitive with recent state-of-the-art set by (Ha et al.,  2016).  We compare zoneout to recurrent dropout (for p ∈ {0.05, 0.2, 0.25, 0.5, 0.7}), weight noise (σ = 0.075), norm stabilizer (β = 50) (Krueger & Memisevic, 2015), and explore stochastic depth (Huang et al.,  2016) in a recurrent setting (analagous to zoning out an entire timestep).  We also tried a shared-mask variant of zoneout as used in pMNIST experiments, where the same mask is used for both cells and hiddens.  Neither stochastic depth or shared-mask zoneout performed as well as separate masks, sampled per unit.  Figure 3 shows the best performance achieved with each regularizer, as well as an unregularized LSTM baseline.  Results are reported in Table 1, and learning curves shown in Figure 4.\nLow zoneout probabilities (0.05-0.25) also improve over baseline in GRUs and tanh-RNNs, reducing BPC from 1.53 to 1.41 for GRU and 1.67 to 1.52 for tanh-RNN.  Similarly, low zoneout probabilities work best on the hidden states of LSTMs.  For memory cells in LSTMs, however, higher probabilities (around 0.5) work well, perhaps because large forget-gate values approximate the effect of cells zoning out.  We conjecture that best performance is achieved with zoneout LSTMs because of the stability of having both state and cell.  The probability that both will be zoned out is very low, but having one or the other zoned out carries information from the previous timestep forward, while having the other react ’normally’ to new information." 
    }, {
      "heading" : "4.1.2 WORD-LEVEL",
      "text" : "For the word-level task, we replicate settings from Zaremba et al. ( 2014)’s best single-model performance.  This network has 2 layers of 1500 units, with weights initialized uniformly [-0.04, +0.04].  The model is trained for 14 epochs with learning rate 1, after which the learning rate is reduced by a factor of 1.15 after each epoch.  Gradient norms are clipped at 10.\nWith no dropout on the non-recurrent connections (i.e. zoneout as the only regularization), we do not achieve competitive results.  We did not perform any search over models, and conjecture that the large model size requires regularization of the feed-forward connections.  Adding zoneout (zc = 0.25 and zh = 0.025) on the recurrent connections to the model optimized for dropout on the non-recurrent connections however, we are able to improve test perplexity from 78.4 to 77.4.  We report the best performance achieved with a given technique in Table 1.\n1 These metrics are deterministic functions of negative log-likelihood (NLL).  Specifically, perplexity is exponentiated NLL, and BPC (entropy) is NLL divided by the natural logarithm of 2." 
    }, {
      "heading" : "4.2 TEXT8",
      "text" : "Enwik8 is a corpus made from the first 109 bytes of Wikipedia dumped on Mar. 3, 2006.  Text8 is a \"clean text\" version of this corpus; with html tags removed, numbers spelled out, symbols converted to spaces, all lower-cased.  Both datasets were created and are hosted by Mahoney (2011).\nWe use a single-layer network of 2000 units, initialized orthogonally, with batch size 128, learning rate 0.001, and sequence length 180.  We optimize with Adam (Kingma & Ba, 2014), clip gradients to a maximum norm of 1 (Pascanu et al.,  2012), and use early stopping, again matching the settings of Cooijmans et al. ( 2016).  Results are reported in Table 1, and Figure 4 shows training and validation learning curves for zoneout (zc = 0.5, zh = 0.05) compared to an unregularized LSTM and to recurrent dropout." 
    }, {
      "heading" : "4.3 PERMUTED SEQUENTIAL MNIST",
      "text" : "In sequential MNIST, pixels of an image representing a number [0-9] are presented one at a time, left to right, top to bottom.  The task is to classify the number shown in the image.  In pMNIST , the pixels are presented in a (fixed) random order.\nWe compare recurrent dropout and zoneout to an unregularized LSTM baseline.  All models have a single layer of 100 units, and are trained for 150 epochs using RMSProp (Tieleman & Hinton, 2012) with a decay rate of 0.5 for the moving average of gradient norms.  The learning rate is set to 0.001 and the gradients are clipped to a maximum norm of 1 (Pascanu et al.,  2012).\nAs shown in Figure 5 and Table 2, zoneout gives a significant performance boost compared to the LSTM baseline and outperforms recurrent dropout (Semeniuta et al.,  2016), although recurrent batch normalization (Cooijmans et al.,  2016) outperforms all three.  However, by adding zoneout to the recurrent batch normalized LSTM, we achieve state of the art performance.  For this setting, the zoneout mask is shared between cells and states, and the recurrent dropout probability and zoneout probabilities are both set to 0.15." 
    }, {
      "heading" : "4.4 GRADIENT FLOW",
      "text" : "We investigate the hypothesis that identity connections introduced by zoneout facilitate gradient flow to earlier timesteps.  Vanishing gradients are a perennial issue in RNNs.  As effective as many techniques are for mitigating vanishing gradients (notably the LSTM architecture Hochreiter & Schmidhuber (1997)), we can always imagine a longer sequence to train on, or a longer-term dependence we want to capture.\nWe compare gradient flow in an unregularized LSTM to zoning out (stochastic identity-mapping) and dropping out (stochastic zero-mapping) the recurrent connections after one epoch of training on pMNIST.  We compute the average gradient norms ‖ ∂L∂ct ‖ of loss L with respect to cell activations ct at each timestep t, and for each method, normalize the average gradient norms by the sum of average gradient norms for all timesteps.\nFigure 6 shows that zoneout propagates gradient information to early timesteps much more effectively than dropout on the recurrent connections, and even more effectively than an unregularized LSTM.  The same effect was observed for hidden states ht.\nzoneout (zc = 0.5), dropout (zc = 0.5), and an unregularized LSTM on one epoch of pMNIST ." 
    }, {
      "heading" : "5 CONCLUSION",
      "text" : "We have introduced zoneout, a novel and simple regularizer for RNNs, which stochastically preserves hidden units’ activations.  Zoneout improves performance across tasks, outperforming many alternative regularizers to achieve results competitive with state of the art on the Penn Treebank and Text8 datasets, and state of the art results on pMNIST.  While searching over zoneout probabilites allows us to tune zoneout to each task, low zoneout probabilities (0.05 - 0.2) on states reliably improve performance of existing models.\nWe perform no hyperparameter search to achieve these results, simply using settings from the previous state of the art.  Results on pMNIST and word-level Penn Treebank suggest that Zoneout works well in combination with other regularizers, such as recurrent batch normalization, and dropout on feedforward/embedding layers.  We conjecture that the benefits of zoneout arise from two main factors: (1) Introducing stochasticity makes the network more robust to changes in the hidden state; (2) The identity connections improve the flow of information forward and backward through the network." 
    }, {
      "heading" : "ACKNOWLEDGMENTS",
      "text" : "We are grateful to Hugo Larochelle, Jan Chorowski, and students at MILA, especially Çağlar Gülçehre, Marcin Moczulski, Chiheb Trabelsi, and Christopher Beckham, for helpful feedback and discussions.  We thank the developers of Theano (Theano Development Team, 2016), Fuel, and Blocks (van Merriënboer et al.,  2015).  We acknowledge the computing resources provided by ComputeCanada and CalculQuebec.  We also thank IBM and Samsung for their support.  We would also like to acknowledge the work of Pranav Shyam on learning RNN hierarchies.  This research was developed with funding from the Defense Advanced Research Projects Agency (DARPA) and the Air\nForce Research Laboratory (AFRL).  The views, opinions and/or findings expressed are those of the authors and should not be interpreted as representing the official views or policies of the Department of Defense or the U.S. Government." 
    }, {
      "heading" : "6 APPENDIX",
      "text" : ""
    }, {
      "heading" : "6.1 STATIC IDENTITY CONNECTIONS EXPERIMENT",
      "text" : "This experiment was suggested by AnonReviewer2 during the ICLR review process with the goal of disentangling the effects zoneout has (1) through noise injection in the training process and (2) through identity connections.  Based on these results, we observe that noise injection is essential for obtaining the regularization benefits of zoneout.\nIn this experiment, one zoneout mask is sampled at the beginning of training, and used for all examples.  This means the identity connections introduced are static across training examples (but still different for each timestep).  Using static identity connections resulted in slightly lower training (but not validation) error than zoneout, but worse performance than an unregularized LSTM on both train and validation sets, as shown in Figure 7." 
    } ],
    "references" : [ {
      "title" : "Learning with pseudo-ensembles",
      "author" : [ "Philip Bachman", "Ouais Alsharif", "Doina Precup" ],
      "venue" : "In Advances in Neural Information Processing Systems,",
      "citeRegEx" : "Bachman et al\\.,?  \\Q2014\\E",
      "shortCiteRegEx" : "Bachman et al\\.",
      "year" : 2014
    }, {
      "title" : "On Fast Dropout and its Applicability to Recurrent Networks",
      "author" : [ "J. Bayer", "C. Osendorfer", "D. Korhammer", "N. Chen", "S. Urban", "P. van der Smagt" ],
      "venue" : null,
      "citeRegEx" : "Bayer et al\\.,?  \\Q2013\\E",
      "shortCiteRegEx" : "Bayer et al\\.",
      "year" : 2013
    }, {
      "title" : "Learning long-term dependencies with gradient descent is difficult",
      "author" : [ "Yoshua Bengio", "Patrice Simard", "Paolo Frasconi" ],
      "venue" : "Neural Networks, IEEE Transactions on,",
      "citeRegEx" : "Bengio et al\\.,?  \\Q1994\\E",
      "shortCiteRegEx" : "Bengio et al\\.",
      "year" : 1994
    }, {
      "title" : "Estimating or propagating gradients through stochastic neurons for conditional computation",
      "author" : [ "Yoshua Bengio", "Nicholas Léonard", "Aaron C. Courville" ],
      "venue" : "CoRR, abs/1308.3432,",
      "citeRegEx" : "Bengio et al\\.,?  \\Q2013\\E",
      "shortCiteRegEx" : "Bengio et al\\.",
      "year" : 2013
    }, {
      "title" : "Hierarchical multiscale recurrent neural networks",
      "author" : [ "Junyoung Chung", "Sungjin Ahn", "Yoshua Bengio" ],
      "venue" : "CoRR, abs/1609.01704,",
      "citeRegEx" : "Chung et al\\.,?  \\Q2016\\E",
      "shortCiteRegEx" : "Chung et al\\.",
      "year" : 2016
    }, {
      "title" : "Recurrent batch normalization",
      "author" : [ "Tim Cooijmans", "Nicolas Ballas", "César Laurent", "Caglar Gulcehre", "Aaron Courville" ],
      "venue" : "arXiv preprint arXiv:1603.09025,",
      "citeRegEx" : "Cooijmans et al\\.,?  \\Q2016\\E",
      "shortCiteRegEx" : "Cooijmans et al\\.",
      "year" : 2016
    }, {
      "title" : "Binaryconnect: Training deep neural networks with binary weights during propagations",
      "author" : [ "Matthieu Courbariaux", "Yoshua Bengio", "Jean-Pierre David" ],
      "venue" : "In NIPS,",
      "citeRegEx" : "Courbariaux et al\\.,?  \\Q2015\\E",
      "shortCiteRegEx" : "Courbariaux et al\\.",
      "year" : 2015
    }, {
      "title" : "A Theoretically Grounded Application of Dropout in Recurrent Neural Networks",
      "author" : [ "Yarin Gal" ],
      "venue" : "ArXiv e-prints,",
      "citeRegEx" : "Gal.,?  \\Q2015\\E",
      "shortCiteRegEx" : "Gal.", 
      "year" : 2015
    }, {
      "title" : "Learning to forget: Continual prediction with LSTM",
      "author" : [ "Felix A. Gers", "Jürgen Schmidhuber", "Fred A. Cummins" ],
      "venue" : "Neural Computation,",
      "citeRegEx" : "Gers et al\\.,?  \\Q2000\\E",
      "shortCiteRegEx" : "Gers et al\\.",
      "year" : 2000
    }, {
      "title" : "Deep residual learning for image recognition",
      "author" : [ "Kaiming He", "Xiangyu Zhang", "Shaoqing Ren", "Jian Sun" ],
      "venue" : "arXiv preprint arXiv:1512.03385,",
      "citeRegEx" : "He et al\\.,?  \\Q2015\\E",
      "shortCiteRegEx" : "He et al\\.",
      "year" : 2015
    }, {
      "title" : "Hierarchical recurrent neural networks for long-term dependencies",
      "author" : [ "Salah El Hihi", "Yoshua Bengio" ],
      "venue" : "In Advances in Neural Information Processing Systems",
      "citeRegEx" : "Hihi and Bengio.,?  \\Q1996\\E",
      "shortCiteRegEx" : "Hihi and Bengio.", 
      "year" : 1996
    }, {
      "title" : "Improving neural networks by preventing co-adaptation of feature detectors",
      "author" : [ "Geoffrey E Hinton", "Nitish Srivastava", "Alex Krizhevsky", "Ilya Sutskever", "Ruslan R Salakhutdinov" ],
      "venue" : "arXiv preprint arXiv:1207.0580,",
      "citeRegEx" : "Hinton et al\\.,?  \\Q2012\\E",
      "shortCiteRegEx" : "Hinton et al\\.",
      "year" : 2012
    }, {
      "title" : "Untersuchungen zu dynamischen neuronalen netzen",
      "author" : [ "Sepp Hochreiter" ],
      "venue" : "Master’s thesis, Institut fur Informatik, Technische Universitat,",
      "citeRegEx" : "Hochreiter.,?  \\Q1991\\E",
      "shortCiteRegEx" : "Hochreiter.", 
      "year" : 1991
    }, {
      "title" : "Long short-term memory",
      "author" : [ "Sepp Hochreiter", "Jürgen Schmidhuber" ],
      "venue" : "Neural computation,",
      "citeRegEx" : "Hochreiter and Schmidhuber.,?  \\Q1997\\E",
      "shortCiteRegEx" : "Hochreiter and Schmidhuber.", 
      "year" : 1997
    }, {
      "title" : "Deep networks with stochastic depth",
      "author" : [ "Gao Huang", "Yu Sun", "Zhuang Liu", "Daniel Sedra", "Kilian Weinberger" ],
      "venue" : "arXiv preprint arXiv:1603.09382,",
      "citeRegEx" : "Huang et al\\.,?  \\Q2016\\E",
      "shortCiteRegEx" : "Huang et al\\.",
      "year" : 2016
    }, {
      "title" : "Adam: A method for stochastic optimization",
      "author" : [ "Diederik Kingma", "Jimmy Ba" ],
      "venue" : "arXiv preprint arXiv:1412.6980,",
      "citeRegEx" : "Kingma and Ba.,?  \\Q2014\\E",
      "shortCiteRegEx" : "Kingma and Ba.", 
      "year" : 2014
    }, {
      "title" : "A clockwork rnn",
      "author" : [ "Jan Koutnik", "Klaus Greff", "Faustino Gomez", "Juergen Schmidhuber" ],
      "venue" : "arXiv preprint arXiv:1402.3511,",
      "citeRegEx" : "Koutnik et al\\.,?  \\Q2014\\E",
      "shortCiteRegEx" : "Koutnik et al\\.",
      "year" : 2014
    }, {
      "title" : "Regularizing rnns by stabilizing activations",
      "author" : [ "David Krueger", "Roland Memisevic" ],
      "venue" : "arXiv preprint arXiv:1511.08400,",
      "citeRegEx" : "Krueger and Memisevic.,?  \\Q2015\\E",
      "shortCiteRegEx" : "Krueger and Memisevic.", 
      "year" : 2015
    }, {
      "title" : "A simple way to initialize recurrent networks of rectified linear units",
      "author" : [ "Quoc V Le", "Navdeep Jaitly", "Geoffrey E Hinton" ],
      "venue" : "arXiv preprint arXiv:1504.00941,",
      "citeRegEx" : "Le et al\\.,?  \\Q2015\\E",
      "shortCiteRegEx" : "Le et al\\.",
      "year" : 2015
    }, {
      "title" : "Building a large annotated corpus of english: The penn treebank",
      "author" : [ "Mitchell P Marcus", "Mary Ann Marcinkiewicz", "Beatrice Santorini" ],
      "venue" : "Computational linguistics,",
      "citeRegEx" : "Marcus et al\\.,?  \\Q1993\\E",
      "shortCiteRegEx" : "Marcus et al\\.",
      "year" : 1993
    }, {
      "title" : "Rnndrop: A novel dropout for rnns in asr",
      "author" : [ "Taesup Moon", "Heeyoul Choi", "Hoshik Lee", "Inchul Song" ],
      "venue" : "Automatic Speech Recognition and Understanding (ASRU),",
      "citeRegEx" : "Moon et al\\.,?  \\Q2015\\E",
      "shortCiteRegEx" : "Moon et al\\.",
      "year" : 2015
    }, {
      "title" : "Understanding the exploding gradient problem",
      "author" : [ "Razvan Pascanu", "Tomas Mikolov", "Yoshua Bengio" ],
      "venue" : "CoRR, abs/1211.5063,",
      "citeRegEx" : "Pascanu et al\\.,?  \\Q2012\\E",
      "shortCiteRegEx" : "Pascanu et al\\.",
      "year" : 2012
    }, {
      "title" : "Dropout improves Recurrent Neural Networks for Handwriting Recognition",
      "author" : [ "V. Pham", "T. Bluche", "C. Kermorvant", "J. Louradour" ],
      "venue" : "ArXiv e-prints,",
      "citeRegEx" : "Pham et al\\.,?  \\Q2013\\E",
      "shortCiteRegEx" : "Pham et al\\.",
      "year" : 2013
    }, {
      "title" : "Recurrent dropout without memory loss",
      "author" : [ "Stanislau Semeniuta", "Aliaksei Severyn", "Erhardt Barth" ],
      "venue" : "arXiv preprint arXiv:1603.05118,",
      "citeRegEx" : "Semeniuta et al\\.,?  \\Q2016\\E",
      "shortCiteRegEx" : "Semeniuta et al\\.",
      "year" : 2016
    }, {
      "title" : "Swapout: Learning an ensemble of deep architectures",
      "author" : [ "S. Singh", "D. Hoiem", "D. Forsyth" ],
      "venue" : "ArXiv e-prints,",
      "citeRegEx" : "Singh et al\\.,?  \\Q2016\\E",
      "shortCiteRegEx" : "Singh et al\\.",
      "year" : 2016
    }, {
      "title" : "Dropout: A simple way to prevent neural networks from overfitting",
      "author" : [ "Nitish Srivastava", "Geoffrey Hinton", "Alex Krizhevsky", "Ilya Sutskever", "Ruslan Salakhutdinov" ],
      "venue" : "The Journal of Machine Learning Research,",
      "citeRegEx" : "Srivastava et al\\.,?  \\Q1929\\E",
      "shortCiteRegEx" : "Srivastava et al\\.",
      "year" : 1929
    }, {
      "title" : "Lecture 6.5-rmsprop: Divide the gradient by a running average of its recent magnitude",
      "author" : [ "Tijmen Tieleman", "Geoffrey Hinton" ],
      "venue" : "COURSERA: Neural Networks for Machine Learning,",
      "citeRegEx" : "Tieleman and Hinton.,?  \\Q2012\\E",
      "shortCiteRegEx" : "Tieleman and Hinton.", 
      "year" : 2012
    }, {
      "title" : "Fast dropout training",
      "author" : [ "Sida Wang", "Christopher Manning" ],
      "venue" : "In Proceedings of the 30th International Conference on Machine Learning,",
      "citeRegEx" : "Wang and Manning.,?  \\Q2013\\E",
      "shortCiteRegEx" : "Wang and Manning.", 
      "year" : 2013
    }, {
      "title" : "Recurrent neural network regularization",
      "author" : [ "Wojciech Zaremba", "Ilya Sutskever", "Oriol Vinyals" ],
      "venue" : "arXiv preprint arXiv:1409.2329,",
      "citeRegEx" : "Zaremba et al\\.,?  \\Q2014\\E",
      "shortCiteRegEx" : "Zaremba et al\\.",
      "year" : 2014
    } ],
    "referenceMentions" : [ {
      "referenceID" : 5,
      "context" : "We achieve competitive results with relatively simple models in character- and word-level language modelling on the Penn Treebank and Text8 datasets, and combining with recurrent batch normalization (Cooijmans et al.,  2016) yields state-of-the-art results on permuted sequential MNIST.", 
      "startOffset" : 199,
      "endOffset" : 223
    }, {
      "referenceID" : 11,
      "context" : "Regularizing neural nets can significantly improve performance, as indicated by the widespread use of early stopping, and success of regularization methods such as dropout and its recurrent variants (Hinton et al.,  2012; Srivastava et al.,  2014; Zaremba et al.,  2014; Gal, 2015).", 
      "startOffset" : 199,
      "endOffset" : 278
    }, {
      "referenceID" : 28,
      "context" : "Regularizing neural nets can significantly improve performance, as indicated by the widespread use of early stopping, and success of regularization methods such as dropout and its recurrent variants (Hinton et al.,  2012; Srivastava et al.,  2014; Zaremba et al.,  2014; Gal, 2015).", 
      "startOffset" : 199,
      "endOffset" : 278
    }, {
      "referenceID" : 7,
      "context" : "Regularizing neural nets can significantly improve performance, as indicated by the widespread use of early stopping, and success of regularization methods such as dropout and its recurrent variants (Hinton et al.,  2012; Srivastava et al.,  2014; Zaremba et al.,  2014; Gal, 2015).", 
      "startOffset" : 199,
      "endOffset" : 278
    }, {
      "referenceID" : 12,
      "context" : "This helps combat the vanishing gradient problem (Hochreiter, 1991; Bengio et al.,  1994), as we observe experimentally.", 
      "startOffset" : 49,
      "endOffset" : 88
    }, {
      "referenceID" : 2,
      "context" : "This helps combat the vanishing gradient problem (Hochreiter, 1991; Bengio et al.,  1994), as we observe experimentally.", 
      "startOffset" : 49,
      "endOffset" : 88
    }, {
      "referenceID" : 0,
      "context" : "Zoneout, like dropout, can be viewed as a way to train a pseudo-ensemble (Bachman et al.,  2014), injecting noise using a stochastic “identity-mask” rather than a zero-mask.", 
      "startOffset" : 73,
      "endOffset" : 95
    }, {
      "referenceID" : 22,
      "context" : "Initially successful applications of dropout in RNNs (Pham et al.,  2013; Zaremba et al.,  2014) only applied dropout to feed-forward connections (“up the stack”), and not recurrent connections (“forward through time”), but several recent works (Semeniuta et al.", 
      "startOffset" : 53,
      "endOffset" : 94
    }, {
      "referenceID" : 28,
      "context" : "Initially successful applications of dropout in RNNs (Pham et al.,  2013; Zaremba et al.,  2014) only applied dropout to feed-forward connections (“up the stack”), and not recurrent connections (“forward through time”), but several recent works (Semeniuta et al.", 
      "startOffset" : 53,
      "endOffset" : 94
    }, {
      "referenceID" : 23,
      "context" : ", 2014) only applied dropout to feed-forward connections (“up the stack”), and not recurrent connections (“forward through time”), but several recent works (Semeniuta et al.,  2016; Moon et al.,  2015; Gal, 2015) propose methods that are not limited in this way.", 
      "startOffset" : 156,
      "endOffset" : 210
    }, {
      "referenceID" : 20,
      "context" : ", 2014) only applied dropout to feed-forward connections (“up the stack”), and not recurrent connections (“forward through time”), but several recent works (Semeniuta et al.,  2016; Moon et al.,  2015; Gal, 2015) propose methods that are not limited in this way.", 
      "startOffset" : 156,
      "endOffset" : 210
    }, {
      "referenceID" : 7,
      "context" : ", 2014) only applied dropout to feed-forward connections (“up the stack”), and not recurrent connections (“forward through time”), but several recent works (Semeniuta et al.,  2016; Moon et al.,  2015; Gal, 2015) propose methods that are not limited in this way.", 
      "startOffset" : 156,
      "endOffset" : 210
    }, {
      "referenceID" : 2,
      "context" : "Whereas saturated output gates or output nonlinearities would cause recurrent dropout to suffer from vanishing gradients (Bengio et al.,  1994), zoned-out units still propagate gradients effectively in this situation.", 
      "startOffset" : 121,
      "endOffset" : 142
    }, {
      "referenceID" : 1,
      "context" : "Bayer et al. ( 2013) successfully apply fast dropout (Wang & Manning, 2013), a deterministic approximation of dropout, to RNNs.", 
      "startOffset" : 0,
      "endOffset" : 20
    }, {
      "referenceID" : 1,
      "context" : "Bayer et al. ( 2013) successfully apply fast dropout (Wang & Manning, 2013), a deterministic approximation of dropout, to RNNs.  Semeniuta et al. ( 2016) apply recurrent dropout to the updates to LSTM memory cells (or GRU states), i.",
      "startOffset" : 0,
      "endOffset" : 151
    }, {
      "referenceID" : 1,
      "context" : "Bayer et al. ( 2013) successfully apply fast dropout (Wang & Manning, 2013), a deterministic approximation of dropout, to RNNs.  Semeniuta et al. ( 2016) apply recurrent dropout to the updates to LSTM memory cells (or GRU states), i.e. they drop out the input/update gate in LSTM/GRU.  Like zoneout, their approach prevents the loss of long-term memories built up in the states/cells of GRUs/LSTMS, but zoneout does this by preserving units’ activations exactly.  This difference is most salient when zoning out the hidden states (not the memory cells) of an LSTM, for which there is no analogue in recurrent dropout.  Whereas saturated output gates or output nonlinearities would cause recurrent dropout to suffer from vanishing gradients (Bengio et al.,  1994), zoned-out units still propagate gradients effectively in this situation.  Furthermore, while the recurrent dropout method is specific to LSTMs and GRUs, zoneout generalizes to any model that sequentially builds distributed representations of its input, including vanilla RNNs.  Also motivated by preventing memory loss, Moon et al. ( 2015) propose rnnDrop.", 
      "startOffset" : 0,
      "endOffset" : 1094
    }, {
      "referenceID" : 1,
      "context" : "Bayer et al. ( 2013) successfully apply fast dropout (Wang & Manning, 2013), a deterministic approximation of dropout, to RNNs.  Semeniuta et al. ( 2016) apply recurrent dropout to the updates to LSTM memory cells (or GRU states), i.e. they drop out the input/update gate in LSTM/GRU.  Like zoneout, their approach prevents the loss of long-term memories built up in the states/cells of GRUs/LSTMS, but zoneout does this by preserving units’ activations exactly.  This difference is most salient when zoning out the hidden states (not the memory cells) of an LSTM, for which there is no analogue in recurrent dropout.  Whereas saturated output gates or output nonlinearities would cause recurrent dropout to suffer from vanishing gradients (Bengio et al.,  1994), zoned-out units still propagate gradients effectively in this situation.  Furthermore, while the recurrent dropout method is specific to LSTMs and GRUs, zoneout generalizes to any model that sequentially builds distributed representations of its input, including vanilla RNNs.  Also motivated by preventing memory loss, Moon et al. ( 2015) propose rnnDrop.  This technique amounts to using the same dropout mask at every timestep, which the authors show results in improved performance on speech recognition in their experiments.  Semeniuta et al. ( 2016) show, however, that past states’ influence vanishes exponentially as a function of dropout probability when taking the expectation at test time in rnnDrop; this is problematic for tasks involving longer-term dependencies.", 
      "startOffset" : 0,
      "endOffset" : 1307
    }, {
      "referenceID" : 1,
      "context" : "Bayer et al. ( 2013) successfully apply fast dropout (Wang & Manning, 2013), a deterministic approximation of dropout, to RNNs.  Semeniuta et al. ( 2016) apply recurrent dropout to the updates to LSTM memory cells (or GRU states), i.e. they drop out the input/update gate in LSTM/GRU.  Like zoneout, their approach prevents the loss of long-term memories built up in the states/cells of GRUs/LSTMS, but zoneout does this by preserving units’ activations exactly.  This difference is most salient when zoning out the hidden states (not the memory cells) of an LSTM, for which there is no analogue in recurrent dropout.  Whereas saturated output gates or output nonlinearities would cause recurrent dropout to suffer from vanishing gradients (Bengio et al.,  1994), zoned-out units still propagate gradients effectively in this situation.  Furthermore, while the recurrent dropout method is specific to LSTMs and GRUs, zoneout generalizes to any model that sequentially builds distributed representations of its input, including vanilla RNNs.  Also motivated by preventing memory loss, Moon et al. ( 2015) propose rnnDrop.  This technique amounts to using the same dropout mask at every timestep, which the authors show results in improved performance on speech recognition in their experiments.  Semeniuta et al. ( 2016) show, however, that past states’ influence vanishes exponentially as a function of dropout probability when taking the expectation at test time in rnnDrop; this is problematic for tasks involving longer-term dependencies.  Gal (2015) propose another technique which uses the same mask at each timestep.", 
      "startOffset" : 0,
      "endOffset" : 1540
    }, {
      "referenceID" : 14,
      "context" : "Zoneout can also be viewed as a per-unit version of stochastic depth (Huang et al.,  2016), which randomly drops entire layers of feed-forward residual networks (ResNets (He et al.", 
      "startOffset" : 69,
      "endOffset" : 89
    }, {
      "referenceID" : 9,
      "context" : ", 2016), which randomly drops entire layers of feed-forward residual networks (ResNets (He et al.,  2015)).", 
      "startOffset" : 87,
      "endOffset" : 104
    }, {
      "referenceID" : 24,
      "context" : "Concurrent with our work, Singh et al. ( 2016) propose zoneout for ResNets, calling it SkipForward.", 
      "startOffset" : 26,
      "endOffset" : 46
    }, {
      "referenceID" : 24,
      "context" : "Concurrent with our work, Singh et al. ( 2016) propose zoneout for ResNets, calling it SkipForward.  In their experiments, zoneout is outperformed by stochastic depth, dropout, and their proposed Swapout technique, which randomly drops either or both of the identity or residual connections.  Unlike Singh et al. ( 2016), we apply zoneout to RNNs, and find it outperforms stochastic depth and recurrent dropout.", 
      "startOffset" : 26,
      "endOffset" : 317
    }, {
      "referenceID" : 16,
      "context" : "Like zoneout, clockwork RNNs (Koutnik et al.,  2014) and hierarchical RNNs (Hihi & Bengio, 1996) update only some units’ activations at every timestep, but their updates are periodic, whereas zoneout’s are stochastic.", 
      "startOffset" : 29,
      "endOffset" : 51
    }, {
      "referenceID" : 4,
      "context" : "Hierarchical multiscale LSTMs (Chung et al.,  2016) learn update probabilities for different units using the straight-through estimator (Bengio et al.", 
      "startOffset" : 30,
      "endOffset" : 50
    }, {
      "referenceID" : 3,
      "context" : ", 2016) learn update probabilities for different units using the straight-through estimator (Bengio et al.,  2013; Courbariaux et al.,  2015), and combined with recently-proposed Layer Normalization (Ba et al.", 
      "startOffset" : 92,
      "endOffset" : 139
    }, {
      "referenceID" : 6,
      "context" : ", 2016) learn update probabilities for different units using the straight-through estimator (Bengio et al.,  2013; Courbariaux et al.,  2015), and combined with recently-proposed Layer Normalization (Ba et al.", 
      "startOffset" : 92,
      "endOffset" : 139
    }, {
      "referenceID" : 7,
      "context" : "This scaling can be viewed as an adaptive, differentiable version of the variational LSTM (Gal, 2015), and could similarly be used to create an adaptive, differentiable version of zoneout.", 
      "startOffset" : 90,
      "endOffset" : 101
    }, {
      "referenceID" : 2,
      "context" : ", 2016) learn update probabilities for different units using the straight-through estimator (Bengio et al.,  2013; Courbariaux et al.,  2015), and combined with recently-proposed Layer Normalization (Ba et al.,  2016), achieve competitive results on a variety of tasks.  As the authors note, their method can be interpreted as an input-dependent form of adaptive zoneout.  In recent work, Ha et al. ( 2016) use a hypernetwork to dynamically rescale the row-weights of a primary LSTM network, achieving state-of-the-art 1.", 
      "startOffset" : 93,
      "endOffset" : 401
    }, {
      "referenceID" : 8,
      "context" : "In the most widely used formulation of an LSTM (Gers et al.,  2000), ct and ht are computed via a set of four “gates”, including the forget gate, ft, which directly connects ct to the memories of the previous timestep ct−1, via an element-wise multiplication.", 
      "startOffset" : 47,
      "endOffset" : 66
    }, {
      "referenceID" : 23,
      "context" : "Semeniuta et al. ( 2016), for instance, zero-mask the input gate: ct = (ft ct−1 + dt it gt)",
      "startOffset" : 0,
      "endOffset" : 24
    }, {
      "referenceID" : 23,
      "context" : "Figure 2: (a) Zoneout, vs (b) the recurrent dropout strategy of (Semeniuta et al.,  2016) in an LSTM.", 
      "startOffset" : 64,
      "endOffset" : 88
    }, {
      "referenceID" : 19,
      "context" : "We evaluate zoneout’s performance on the following tasks: (1) Character-level language modelling on the Penn Treebank corpus (Marcus et al.,  1993); (2) Word-level language modelling on the Penn Treebank corpus (Marcus et al.", 
      "startOffset" : 125,
      "endOffset" : 146
    }, {
      "referenceID" : 19,
      "context" : ", 1993); (2) Word-level language modelling on the Penn Treebank corpus (Marcus et al.,  1993); (3) Character-level language modelling on the Text8 corpus (Mahoney, 2011); (4) Classification of hand-written digits on permuted sequential MNIST (pMNIST) (Le et al.", 
      "startOffset" : 71,
      "endOffset" : 92
    }, {
      "referenceID" : 18,
      "context" : ", 1993); (3) Character-level language modelling on the Text8 corpus (Mahoney, 2011); (4) Classification of hand-written digits on permuted sequential MNIST (pMNIST) (Le et al.,  2015).", 
      "startOffset" : 165,
      "endOffset" : 182
    }, {
      "referenceID" : 14,
      "context" : "075), norm stabilizer (β = 50) (Krueger & Memisevic, 2015), and explore stochastic depth (Huang et al.,  2016) in a recurrent setting (analagous to zoning out an entire timestep).", 
      "startOffset" : 89,
      "endOffset" : 109
    }, {
      "referenceID" : 5,
      "context" : "These settings match those used in Cooijmans et al. ( 2016).  We also train GRUs and tanh-RNNs with the same parameters as above, except sequences are nonoverlapping and we use learning rates of 0.", 
      "startOffset" : 35,
      "endOffset" : 59
    }, {
      "referenceID" : 28,
      "context" : "For the word-level task, we replicate settings from Zaremba et al. ( 2014)’s best single-model performance.", 
      "startOffset" : 52,
      "endOffset" : 74
    }, {
      "referenceID" : 21,
      "context" : "We optimize with Adam (Kingma & Ba, 2014), clip gradients to a maximum norm of 1 (Pascanu et al.,  2012), and use early stopping, again matching the settings of Cooijmans et al.", 
      "startOffset" : 81,
      "endOffset" : 103
    }, {
      "referenceID" : 5,
      "context" : ", 2012), and use early stopping, again matching the settings of Cooijmans et al. ( 2016).  Results are reported in Table 1, and Figure 4 shows training and validation learning curves for zoneout (zc = 0.", 
      "startOffset" : 64,
      "endOffset" : 88
    }, {
      "referenceID" : 21,
      "context" : "001 and the gradients are clipped to a maximum norm of 1 (Pascanu et al.,  2012).", 
      "startOffset" : 57,
      "endOffset" : 79
    }, {
      "referenceID" : 23,
      "context" : "As shown in Figure 5 and Table 2, zoneout gives a significant performance boost compared to the LSTM baseline and outperforms recurrent dropout (Semeniuta et al.,  2016), although recurrent batch normalization (Cooijmans et al.", 
      "startOffset" : 144,
      "endOffset" : 168
    }, {
      "referenceID" : 5,
      "context" : ", 2016), although recurrent batch normalization (Cooijmans et al.,  2016) outperforms all three.", 
      "startOffset" : 48,
      "endOffset" : 72
    }, {
      "referenceID" : 28,
      "context" : "336 NR-dropout (Zaremba et al.,  2014) – – 82.", 
      "startOffset" : 15,
      "endOffset" : 37
    }, {
      "referenceID" : 7,
      "context" : "4 – – V-dropout (Gal, 2015) – – – 73.", 
      "startOffset" : 16,
      "endOffset" : 27
    }, {
      "referenceID" : 5,
      "context" : "4 – – RBN (Cooijmans et al.,  2016) – 1.", 
      "startOffset" : 10,
      "endOffset" : 34
    }, {
      "referenceID" : 4,
      "context" : "250 – – – – 3-HM-LSTM + LN (Chung et al.,  2016) – 1.", 
      "startOffset" : 27,
      "endOffset" : 47
    }, {
      "referenceID" : 12,
      "context" : "As effective as many techniques are for mitigating vanishing gradients (notably the LSTM architecture Hochreiter & Schmidhuber (1997)), we can always imagine a longer sequence to train on, or a longer-term dependence we want to capture.", 
      "startOffset" : 102,
      "endOffset" : 134
    } ],
    "year" : 2017,
    "abstractText" : "We propose zoneout, a novel method for regularizing RNNs.  At each timestep, zoneout stochastically forces some hidden units to maintain their previous values.  Like dropout, zoneout uses random noise to train a pseudo-ensemble, improving generalization.  But by preserving instead of dropping hidden units, gradient information and state information are more readily propagated through time, as in feedforward stochastic depth networks.  We perform an empirical investigation of various RNN regularizers, and find that zoneout gives significant performance improvements across tasks.  We achieve competitive results with relatively simple models in characterand word-level language modelling on the Penn Treebank and Text8 datasets, and combining with recurrent batch normalization (Cooijmans et al.,  2016) yields state-of-the-art results on permuted sequential MNIST.", 
    "creator" : "LaTeX with hyperref package"
  }
}