Playing the lottery with rewards and multiple languages: lottery tickets in RL and NLP
Authors: Haonan Yu
Sergey Edunov
Yuandong Tian
Ari S. Morcos
Venue:  (2020)
Keywords: lottery tickets
nlp
transformer
rl
reinforcement learning

Abstract: The lottery ticket hypothesis proposes that over-parameterization of deep neural networks (DNNs) aids training by increasing the probability of a “lucky” sub-network initialization being present rather than by helping the optimization process (Frankle& Carbin, 2019). Intriguingly, this phenomenon suggests that initialization strategies for DNNs can be improved substantially, but the lottery ticket hypothesis has only previously been tested in the context of supervised learning for natural image tasks. Here, we evaluate whether “winning ticket” initializations exist in two different domains: natural language processing (NLP) and reinforcement learning (RL).For NLP, we examined both recurrent LSTM models and large-scale Transformer models (Vaswani et al., 2017). For RL, we analyzed a number of discrete-action space tasks, including both classic control and pixel control. Consistent with workin supervised image classification, we confirm that winning ticket initializations generally outperform parameter-matched random initializations, even at extreme pruning rates for both NLP and RL. Notably, we are able to find winning ticket initializations for Transformers which enable models one-third the size to achieve nearly equivalent performance. Together, these results suggest that the lottery ticket hypothesis is not restricted to supervised learning of natural images, but rather represents a broader phenomenon in DNNs.

Published as a conference paper at ICLR 2020
PLAYING THE LOTTERY WITH REWARDS AND MULTIPLE LANGUAGES: LOTTERY TICKETS IN RL AND NLP
Haonan Yu; Sergey Edunov, Yuandong Tian, and Ari S. Morcos
Facebook AI Research
haonanu@gmail.com com, (edunov, yuandong, arimorcos ) ffb. com
ABSTRACT
The lottery ticket hypothesis proposes that over areeeaaramttiinntooonn of deep neural net- works (DNNs) aids training by increasing the probability of a "lucky" sub-network initialization being present rather than by helping the optimization process (Frankle) & Carbin 2019). Intriguingly, this phenomenon suggests that initialization strate- gies for DNNs can be improved substantially, but the lottery ticket hypothesis has only previously been tested in the context of supervised learning for natural image tasks. Here, we evaluate whether "winning ticket" initializations cxist in two differ- ent domains: natural language processing (NLP) and reinforcement learning (RL). For NLP, we examined both recurrent LSTM models and large-scale Transformer models (Vaswani al., 2017). For RL, we analyzed number of discrete-action space tasks, including both classic control and pixel control. Consistent with work in supervised image classification, we confirm that winning ticket initializations generally outperform parameter- parameter-matched matched random initializations, even at extreme pruning rates for both NLP and RL. Notably, we are able to find winning ticket initializations for Transformers which enable models one-third the size to achieve nearly equivalent performance. Together, these results suggest that the lottery ticket hypothesis is not restricted to supervised learning of natural images, but rather represents a broader phenomenon in DNNs.
INTRODUCTION
The lottery ticket phenomenon (Frankle Carbin 2019 Frankle et al. 2019; Zhou et al. 2019)
occurs when small, sparse sub-networks can be found in over-parameterized deep neural networks (DNNs) which, when trained in isolation, can achieve similar or even greater performance than the original, highly peratttreettri network. This phenomenon suggests that over-parameetrrization in DNN training is beneficial primarily due to proper initialization rather than regularization during
the training process itself (Allen-Zhu al. 2019 2018 Du & Lee 2018
Du et
2019
Neyshabur
ct al. 2014 2019).
However, despite extensive experiments in Frankle & Carbin (019)) and Frankle al. (2019)
remains unclear whether the lottery ticket phenomenon is an intrinsic feature of DNN behavior or whether it is dependent on other factors such as supervised learning, network architecture, specific tasks (e.g., image classification), the bias in the dataset, or artifacts from the optimization algorithm itself. As discussed in Frankle & Carbin (2019) and Liu et al. (2019), large learning rates severely damage the lottery ticket effect, and for larger models (such as VGG and ResNets) and datasets (c.g., ImageNet), heuristics like learning rate warmup (Frankle Carbin 2019) and late rewinding (Frankle al. 2019) are needed to induce high performance and reliable winning tickets. Recent work has also questioned the effectiveness of the lottery ticket hypothesis, raising concerns about the generality of this phenomenon (Liu et al. 2019 Gale et al. 2019).
In this work, we address the question of whether the lottery ticket phenomenon is merely an artifact of supervised image classification with feed-forward convolutional neural networks, whether
Work done while at Facebook Al Research; currently at Horizon Robotics
To whom correspondence should be addressed.

---

Published as a conference paper at ICLR 2020
this phenomenon generalizes to other domains, architectural paradigms, and learning regimes (e.g.. environments with reward signals). Many natural language processing (NLP) models feature complex gating mechanies paired with recurrent dynamics, either of which may significantly impact the optimization process and, consequently, the lottery ticket phenomenon. Furthermore, prior work has suggested that this phenomenon is not present in Transformer models (Gale et al.. 2019), calling the broad applicability of lottery tickets into question. In reinforcement learning (RL), the data distribution shifts the agent learns from often reward signals, significantly modifying the optimization process and the resultant networks. Pre-trained feature extractors have proven successful in computer vision (Korbllth et al. 2019 Razavian et al., [0144] Yosinski et al. 2014), but in RL, agents often fail to generalize even to extremely similar situations (Raghu et al. 2018 Lanctot et al. 2017 Cobbe et al. 2018 Ruderman et al. 2019).
To answer this question, we evaluate whether the lottery ticket hypothesis holds for NLP and RI, both of which are drastically different from traditional supervised image classification. To demonstrate the lottery ticket phenomenon, we ask whether sparsified subnetworks initialized winning tickets outperform randomly initialized subnetworks at convergence. We note that, though desirable, we do not require that subnetworks match the performance of the full network, originally stated in Frankle et al. (2019). We exclude this requirement because we are primarily interested in whether appropriate initialization impacts the performance of subnetwork training, consistent with the revised definition of the lottery ticket hypothesis in Frankle & Carbin (2019). For NLP, we evaluate language modelling on Wikitext-2 with LSTMs (Merity al. 2017) and machine translation on the WMT' 14 English-German translation task with Transformers (Vaswani et al. (017)). For RL, we evaluate both classic control problems and Atari games (Bellemare et 2013).
Perhaps surprisingly, we found that lottery tickets are present in both NLP and RL tasks. In NLP, winning tickets were present both in recurrent LSTMs trained on language modeling and in Transformers (Vaswani et al., (017) trained on a machine translation task while in RL, we observed winning tickets both classic control problems and Atari games (though with high variance). Notably, we are able to find masks and initializations which enable a Transformer Big model to train from scratch to achieve 99% the BLEU score of the unpruned model on the Newstest' 14 machine translation task while using only third of the parameters. Together, these results demonstrate that the lottery ticket phenomenon is a general property of deep neural networks, and highlight their potential for practical applications.
2 RELATED WORK
Our work is primarily inspired by the lottery ticket hypothesis, first introduced in Franklee & Carbin (2119), which argues that over-parameterized neural networks contain small, sparse sub-networks (with as few as 0.1% 1% of the original network's parameters) which can achieve high performance when trained in isolation. Frankle et al. (209)) revised the lottery ticket hypothesis to include the notion of late rewinding, which was found significantly improve performance for large-scale models and large-scale image classification datasets. In addition, the revised lottery ticket hypothesis relaxed the need for subnetworks match the performance of the full network to simply exceeding the performance a randomly initialized subnetwork. For brevity, we will refer exclusively to the revised definition throughout the paper. However, both of these works solely focused on supervised image classification, leaving it unclear whether the lottery ticket phenomenon is present in other domains and learning paradigms.
Recent work (Liu et al. 2019) challenged the lottery ticket hypothesis, demonstrated that for structured pruning settings, random sub-networks were able to match winning ticket performance, Gale et al.) (2019) also explored the lottery ticket hypothesis in the context of ResNets and Transformers. Notably, they found that random sub-networks could achieve similar performance to that of winning ticket networks for both model classes. However, they did not use iterative pruning or late rewinding, both of which have been found to significantly improve winning ticket performance Frankle et al. (019);
Frankle & Carbin (019).
More broadly. pruning methods for deep neural networks have been explored extensively (Han et al. 2015). Following Frankle et al. (2019) and Frankle & Carbin (2019), ) this work, in which the smallest magnitude weights are pruned first (Han et al., 2015). To determine optimal pruning performance, Molchanov al. (2017b) greedily prune weights to determine an
magnitude pruning in
2

---

Published as a conference paper at ICLR 2020
oracle ranking. Also, Ayinde et al. (2019) and Qin al. (2019) have attempted rank channels by redundancy and preferentially prune redundant filters, and Molchanov et al. (2017a) used variational methods to prune models. However, all of these works only evaluated these approaches for supervised image classification.
3
APPROACH
3.1 GENERATING LOTTERY TICKETS
Pruning methods In our experiments, we use both one-shot and iterative pruning to find winning ticket initializations. one-shot pruning, the full network is trained convergence, and then a given fraction of parameters are pruned, with lower magnitude parameters pruned first. To evaluate winning ticket performance, the remaining weights are reset their initial values, and the sparsified model is retrained to convergence. However, one-shot pruning is very susceptible to noise in the pruning process, and as result, has widely been observed that one-shot pruning ander-performs iterative pruning methods (Frankle & Carbin 2019
Han et al. 2015 Liu ct al. 2019).
In iterative pruning Franklle Carbin 2019 Han et al. 2015), alternating cycles of training models from scratch and pruning are performed. At each pruning iteration, fixed, small fraction of the remaining weights are pruned, followed by re-initialization to a winning ticket and another cycle of training and pruning. More formally, the pruning at iteration k + 1 is performed on the trained weights the winning ticket found at iteration k. At iteration with an iterative pruning rate p, the fraction of weights pruned is:
Th -1 p)*, Isks S 20
We therefore perform iterative pruning for all our experiments unless otherwise noted, with an iterative pruning rate p 0.2. For our RL experiments, we perform 20 pruming iterations. Pruning was always performed globally, such that all weights (including biases) of different layers were pooled and their magnitudes ordered for pruning. As result, the fraction of parameters pruned across layers may vary given a total pruning fraction.
Late rewinding In the original incarnation of the lottery ticket hypothesis (Frankle & Carbin 2019), winning tickets reset to their values at initialization. However, Frankle al. (2019) found that resetting winning tickets to their values to an iteration early in training resulted in dramatic improvements winning ticket performance on large-scale image datasets, even when weights were reset to their values only few iterations into training. late rewinding can therefore be defined as resetting winning tickets to their weights iteration j training, with the original lottery ticket paradigm taking j O. Unless otherwise noted, we use late rewinding throughout, with a late rewinding value of epoch used for all RL experiments. We also compare late rewinding with normal resetting for NL. in section 4.1 and on several representative RI games in section 4.2
3.2 NATURAL LANGUAGE PROCESSING
To test. the lottery ticket hypothesis in NLP, we use two broad model and task paradigms: two- layer LSTMs for language modeling on Wikitext-2 (Merity et al. 2017) and Transformer Base and Transformer Big models (Vaswani al. 2017) on the WMT' 14 En2De News Translation task.
Language modeling using LSTMs For the language modeling task, we trained an LSTM model with a hidden state size of 650. contained dropout layer between the two RNN layers with dropout probability of 0.5. The LSTM received word embeddings of size 650. For training, used truncated Backpropagation Through Time (truncated BPTT) with a sequence length of 50. The training batch size was 10-3 and 31 = 0.9, 32 = 0.999, c 10-3
30, and models were optimized using Adam with learning rate of
As in the RL experiments, 20 total pruning iterations. We also employ late rewinding where the initial weights of a winning ticket were set to the weights after first epoch of training the full network. For ticket evaluation,
global iterative pruning with an iterative pruning rate of 0.2 and
3

---

Published as a conference paper at ICLR 2020
trained the model for 10 epochs the training set, after which we computed the log perplexity the test set. We also perform two ablation studies without late rewinding and using one-shot pruning, respectively.
Machine translation using transformers For the machine translation task, we use the FAIRSEQ framework (Ott et al. 2019). following the setup described in (Ott al. 2018 train Transformer- base model on the pre-processed dataset from (Vaswani et al.) 2017). We train models for 50,000 updates and apply checkpoint averaging. We report case-sensitive tokenized BLEU with multi-bleu.pp@ the Newstest 2014
3.3
REINFORCEMENT LEARNING
3.3.1 SIMULATED ENVIRONMENTS
For our RL experiments, we use two types of discrete-action games: classic control and pixel control. For classic control, we evaluated 3 OpenAl Gymfeenviromments that have vectors of real numbers as network inputs. These simple experiments mainly serve to verify whether winning tickets exist in networks that solely consist of cully-comnecte
(FC) layers for RL problems. For pixel control, we evaluated 9 Atari (Bellemare et al. 2013) games. A summary of all the games along with the corresponding networks is provided in Table AT, Classic control games were trained using the FLARE framework
and and Atari games were trained using the ELF framework (Tian et al. 2017
3.3.2 TICKET EVALUATION
To evaluate a ticket (pruned sub-network), we train the ticket to play a game with its corresponding initial weights for N epochs. Here, epoch is defined as every M game episodes or every M training batches depending on the game type. At the end of training, we compute the averaged episodic reward over the last T. game episodes. This average reward, defined as tickel reward, indicates the final performance of the ticket playing a game by sampling actions from its trained policy. For each game, we plot ticket reward curves for both winning and random tickets the fraction of weights pruned increases. To evaluate the impact of random seed on our results, we repeated the iterative pruning process three times on every game, and plot (mean standard deviation) for all results.
Classic control All three games were trained in the FLARE framework with 32 game threads running in parallel, and each thread gets blocked every 4 time steps for training. Thus training batch contains 32 x 4 = 128 time steps. Immediate rewards are divided by 100. For optimization, we use RMSprop with a learning rate of 10-4 and a 0.99, c 10-8
Pixel control All 9 Atari games are trained using a common ELF configuration with all RL hyperparameters being shared across games (see AAT for our choices of N and M). Specifically, each game has 1024 game threads running in parallel, and each thread gets blocked every 6 time steps for training. For each training batch, the trainer samples 128 time steps from the common pool. The policy entropy cost for exploration is weighted by 0.01. We clip both immediate rewards and advantages to [1]. Because the training is asynchronous and off-policy, we impose an importance factor which is the ratio of action probability given by the current policy to that from the old policy. This ratio clamped at 1.5 to stabilize training. For optimization, we use Adam with a
learning rate of 10-3 and 31 0.9, 82 0.999, 6 10-3
4

---

Published as a conference paper at ICLR 2020
Fondom Bokee
8.0
Log perplexity
7.5
Wirking Sood, noll
wireng toos,
7.0-
6.5
6.0
5.0
0.
0.36
0.488
0.9
0.78-
0.79
0.89,
0.066
0.93
0334
0383
0355
0:972
0.97
0.962
0.900
0.968
Fraction of weights pruned
Figure 1: Performance of winning ticket initializations for LSTT models trained Wikitext-2.
RESULTS
4.1 NATURAL LANGUAGE PROCESSING
In this section, we investigate whether winning ticket outperform random tickets in the context of NLP. In particular, we focus on language modeling with a recurrent LSTT and machine translation using two variants of the Transformer model (Vaswani al. 2017).
4.1.1 LANGUAGE MODELING WITH LSTMS
We first investigate whether winning tickets exist in two-layer LSTM model trained perform language modeling task on the Wikitext-2 dataset. Encouragingly, we found that winning tickets with late rewinding significantly outperformed random tickets this task for all pruning levels and demonstrated high performance even at high pruning rates, with as many as 90% of parameters capable of being removed without a noticeable increase in log perplexity (Figure
To measure the impact of iterative pruning and late rewinding, we performed ablation studies for these two properties. Interestingly, removing late rewinding only slightly damaged performance and primarily impacted intermediate pruning levels, suggesting that it is only partially necessary for language modeling with LSTMs. Iterative pruning, however, was essential, as performance plummeted, reaching values worse than random tickets once 80% of parameters had been pruned. Together, these results both validate the lottery ticket hypothesis language modeling with LSTMs and demonstrate the impact of iterative pruning and late rewinding in this setting.
4.1.2 2 MACHINE TRANSLATION WITH TRANSFORMERS
We next evaluate whether winning tickets present in Transformer models trained machine translation. Our baseline machine translation model Transformer Base achieves a BLEU score of 27.6 on the Newstest' 14 test set (compared 27.3 in Vaswani al. (017)). We perform global iterative pruning with and without late rewinding to the parameters of the baseline model after 1000 updates. Consistent with our results on language modeling with LSTMs, winning tickets outperform random tickets in Transformer models (Figure 2 left). Additionally, pruning and late rewinding significantly improved performance, with iterative pruning again having a larger impact than late rewinding. The necessity of these modifications explain why our results differ from Gale et al. 2019). which only used one-shot pruning without late rewinding.
again found that iterative
iti-bleu.perl
https://aym.openai..
https://githbbbcom/pptttr//////
5

---

We also evaluated version of the Transformer Base model in which only Transformer layer weights (attention and fully connected layers) were pruned, but embeddings were left intact (Figure ( right). Published as a conference paper at ICLR 2020
28
BLEU Newstest
24
20
22
15-
18
BLEU
10
ticket
Random ticket
Winning ticket In
Winning ticket, Ir
ticket,
Winning
Winning ticket, na It, ane-shot
12
Winning ticket, no it, one-sho shot
0.59
P-067
0.74
0.79
0.3
os
0.5
0,08
0.9
055
0.74
6.79
oo
0.8
0.99
Fraction of weights pruned
Fraction of weights pruned, excluding embeddings
Figure 2: Winning ticket initialization performance for Transformer Base models trained machine translation.
30
BLEU BLEU Newstest144
Random ticket
Winning ticket,
5
Winning ticket,
Winning ticket. no It, ohe
0.36
0,49
0.59
0.67
0,74
0,79
0.8
0,55
0,98
0,99
Fraction of weights pruned
Figure Winning ticket initialization performance for Transformer Big models trained on machine translation.
Results in this setting were noticeably different from when we pruned all weights. First, the random ticket performance drops off at much faster rate than in the full pruning setting. This suggests that, for random initializations, a large fraction of embedding weights can prumed without damaging network performance, but very few transformer layer weights can be pruned. Second, and in stark contrast to the random ticket case, we observed that winning ticket performance was remarkably robust pruning of only the transformer layer weights, with roughly linear drop in BLEU score.
To determine whether the lottery phenomenon scales even further, we trained a Transformer Big model with 210M parameters, which reaches BLEU score of 29.3 (compared 28.4 in Vaswani ct al. 2017 ). Here, we again observe that winning tickets significantly outperform random tickets (Figure 2). Notably, we found that with iterative pruning and late rewinding model performance we can train models from scratch 99% of the unpruned model's performance with only a third of the weights (28.9 BLEU 67% pruned vs. 29.2 BLEU unpruned). This result demonstrates the practical implications for the lottery ticket phenomenon as current state of the art Transformer-baseee systems are often too large to deploy and are highly expensive to train, though sparse matrix multiplication libraries would be necessary to realize this gain.
4.2
REINFORCEMENT LEARNING
4.2.1
CLASSIC CONTROL
To begin our investigation of the lottery ticket hypothesis in reinforcement learning, we evaluated three simple classic control games: Cartpole-vO, Acrobot-vl, and LunarLander-v2. As discussed in Section 3.3.1 and Table A1 used simple fully-connected models with three hidden layers. Encouragingly. and consistent with supervised image classification results, we found that winning tickets successfully outperformed random tickets in classic control environments (Figure 4). This

---

Published as a conference paper at ICLR 2020
CarPPolevVO
Acrobot-vl
Lunarl aaneer23
200
275
-200
150
Ticket rewerrd
-250
Ticket comsun
:25
-20
200
-250
tinse
malousk
-20
Fraction weights pruned
Frection
Fraction
weights
Figure 4: Winning ticket performance classic control games. Fach curve plots the + standard deviation three independent iterative pruning pracesses for each game.
Assault
Berzerk
Breakout
700
Ticket rewerd
Ticket reward
230
Eco
402
500
400
100
200
Fraction weights pruned
Fraction weights pruned
Fraction of weights pruned
Centipede
Kangaroo
Space Invaders
142-
242
2000
Ticket reward
Ticket reward
boce-
2010
202-
302
2000
82
182
Fraction weights pruned
Fraction of weights pruned
Fraction of weights pruned
Krull
Scaquest
Qbert
6010
2500
6010
4000
Ticket reward
Ticket roward
aace
5000
:500
Tscket
20ce
1000
4020
1020
Randvn tock
Wenning
Tut retwarr
traction
weights pruned
Fraction pruned
Fraction weights pruned
Figure Reward of WTs (blue) and RTs (red) on Atari. Shaded bars represent standard deviation across runs the gray curve represents performance unprumed network.
result suggests that the lottery ticket phenomenon not merely an artifact of supervised image classification, but occurs in RL paradigms as well.
4.2.2 ATARI GAMES
However, in the original lottery ticket study (Frankle Carbin 2019). winning tickets were sub- stantially casier to find in simple, fully-connected models trained on simple datasets (e.g., MNIST) than in more complex models trained on larger datasets (c.g. ResNets on CIFAR and ImageNet). We therefore asked whether winning tickets exist in convolutional networks trained Atari games as
7

---

Published as a conference paper at ICLR 2020
CarPPolevOO
Lunarl.ander-a
Acrobot-vl
200
275
150
Ticket rewerd
50
Ticket ronsun
-220
:25
200
Raroom
-420
-150
Wmm tisse
unacled
-150
Fraction
weights pruned
Fraction
Fraction weights
Figure Ablation studies of several classic control games on the effects of late rewinding and iterative pruning. Shaded error hars represent mean standard deviation across runs and the gray curve represents performance of the unpruned network.
well. We found that the impact of winning tickets varied substantially across Atari games (Figure 5 with some games, such as Assault, Seaquest, and Berzerk benefiting significantly from winning ticket initializations, while other games, such as Breakout and Centipede only benefitted slightly. Notably, winning ticket initializations increased reward for both Berzerk and Qbert. Interestingly, one game, Krull, such benefit, and both winning and random tickets performed well even at the most extreme pruning fractions, suggesting that Krull may be so perr-parameterized that we were unable to get into the regime in which winning ticket differences emerge.
One particularly interesting case is that of Kangaroo. Because we used the same hyperparameter settings for all games, the initial, unpruned Kangaroo models failed converge to high rewards (typical reward on Kangaroo for converged models is in the several thousands). Surprisingly. however, winning ticket initializations substantially improved performance (though these models were still very far from optimal performance on this task) over random tickets, enabling some learning where no learning at all was previously possible. Together, these results suggest that while beneficial winning ticket initializations can be found for some Atari games, winning ticket initializations for other games may not exist or be more difficult to find.
We also observed that the shape of pruning curves for random tickets Atari games also varied substantially based on the game. For example, some games, such as Breakout and Space Invaders, were extremely sensitive to pruning, with performance dropping almost immediately, while other games, such as Berzerk, Centipede, and Krull actually saw performance steadily increase in early pruning iterations. This result suggests that the level of over-paramettrrzation varies dramatically across Atari games and that "one size fits all" models may have subtle impacts on performance based on their level of preeettramttti..i.nnn.
To measure the impact of late rewinding and iterative pruning on the performance of winning ticket initializations in RL, we performed ablation studies on six representative games both from classic control and Atari: CartPole-vO, Acrobot-v1, LunarLander-v2, Assault, Breakout, and Seaquest. For all ablation experiments, we leave all training parameters fixed (configuration, hyperparameter, optimizer, etc.) except for those specified. For both classic control (Figure (6) and Atari (Figure (7). we observed that, consistent with previous results supervised image classification (Frankle et al.. 2019; Frankle & Carbin, 2019), both late rewinding and iterative pruning improve winning ticket performance, though interestingly, the degree to which these modifications improved performance varied significantly across games.
5 CONCLUSION
In this study, we investigated whether the lottery ticket hypothesis holds in regimes beyond simple supervised image classification by analyzing both NLP and RL domains. For NLP, we found that winning ticket initializations beat random tickets both for recurrent LSTM models trained on language modeling and Transformer models trained on machine translation. Notably, we found high performing Transformer Big models even at high pruning rates (2 67%). For RL, we found that winning ticket initializations substantially outperformed random tickets on classic control problems and for many, but not all, Atari games. Together, these results suggest that the lottery ticket phenomenon is not
8

---

Published as a conference paper at ICLR 2020
Assault
Seaquest
Breakout
500
2M00
700
Ticket reward
Ticket reward
500
2.00
193
202-
Randan doket
Wornnn
200
1.00
102
800
0
Fraction weights pruned
Fraction
pruned
Figure Ablation studies of several pixel control games the effects of iterative pruning. Shaded hars represent mean standard deviation across runs and the gray curve represents performance of the unpruned network. "Ir" means late-resetting.
restricted to supervised image classification, but rather represents a general feature of deep neural network training.
REFERENCES
Zeyuan Allen Allen-Zhu, Yuanzhi Li, and Yingyu Liang. Learning and generalization in overparameterized neural networks, going beyond two layers. November 2018. httphtttp///axiivorrttt abb/
1811.00 04918]
Zeyuan Allen-Zhu, Yuanzhi Li, and Zhao Song. A convergence theory for deep learning via over- parameterization. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Reseurch, pp. 252, Long Beach, California, USA, 09-15 Jun 2019. PMLR. URI. ngs m press / v a een- zhu 1 9a ht.l
Babajide O Ayinde, Tamer Inanc, and Jacek M Zurada. Redundant feature pruning for accelerated inference in deep neural networks. Neural networks: the official journal of the International Neural
Network Society, May 2019. ISSN 0883-6080. doi: 10. m0.u0016 04.022. URL http:
//www. www.scciencei.rec.om /ttp:///sc.eeeea......//////////
com/ swwenceeeeeeeeeecceeee.eeeeccese http:////...ace////6333381111110
M. G. Bellemare, Y. Naddaf, J. Veness, and M. Bowling. The arcade learning environment: An evaluation platform for general agents. Journal of Artificial Intelligncce Research, 47:253-279, jun 2013.
Karl Cobbe, Oleg Klimov, Chris Hesse, Tachoon Kim, and John Schulman Quantifying generalization in reinforcement learning. December 2018. URL httpp:://acx.v.../////////////
02341
Simon S Du and Jason Lee. On the power of over over-vveaaeeee parametrizaaio neural networks with quadratic activation. In International Conference in Machine Learning (ICML), 2018.
Simon S Du, Xiyu Zhai, Barnabas Poczos, and Aarti Singh. Gradient descent provably optimizes pwer-parameterized neural networks. International Conference in Learning Representations (ICLR), 2019.
Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable
neural networks. In International Conference on Learning Representations, 2019. URLhttp:
//arxiv. org hwwpgg/a/a//aaa /atba///803.00 03635
Jonathan Frankle, Gintare Karolina Dziugaite, Daniel Roy, and Michael Carbin. The lottery ticket hypothesis at scale. March 2019. URL http: /ttp:////xxiiv/ arxiv org/ /190/.193...0 01611
Trevor Gale, Erich Elsen, and Sara Hooker. The state of sparsity in deep neural networks. February 2019. URL http::ttp://tttt org/ abs 1902. C9574
Song Han, Jeff Pool, John Tran, and William Dally. Learning both weights and connections for efficient neural network. In Advances in neural information processing systems, pp. 1135-1143, 2015.

---

Published as a conference paper at ICLR 2020
Simon Kornblith, Jonathon Shlens, and Quoc V Le. Do better ImageNet models transfer better? In Computer Vision and Pattern Recogntion (CVPR), 2019.
Mare Lanctot, Vinicius Zambaldi, Audrûnas Gruslys, Angeliki Lazaridou, Karl Tuyls, Julien Pérolat, David Silver, and Thore Graepel. A unified game-theoretic approach to multiagent reinforcement learning. Proceedings of the s1 International Conference on Neural Information Processing Systems, NIPS pp. 4193 4206, USA, 2017. Curran Associates Inc. ISBN 978-1-5108-6096- URL httphttt://dd cm org/ citattttttt ion. cffm:ld=329999...mmmm 3295174
Zhuang Liu, Mingjie Sun, Tinghui Zhou, Gao Huang, and Trevor Darrell. Rethinking the value of network pruning. In International Conference Learning Representations, 2019. URL
http://arxi: (trg/abb/18100 00117027000000000
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. In International Conference in Learning Representations (ICLR), 2017.
Dmitry Molchanov, Arsenii Ashukha, and Dmitry Vetrov. Variational dropout sparsifies deep neural networks. In Proceedings of the 34th International Conference on Machine Learning- Volume 70, pp. 2498-2507. JMLR. org, 2017a.
Pavlo Molchanov, Stephen Tyree, Tero Karras, Timo Aila, and Jan Kautz. Pruning convolutional neural networks for resource efficient inference. In International Conference Learning Repre- sentations (ICLR), 2017b.
Behnam Neyshabur, Ryota Tomioka, and Nathan Srebro. In search of the real inductive bias: On the role of implicit regularization in deep learning. In International Conference in Learning Representations (ICLR) Workshop Track, 2014.
Behnam Neyshabur, Zhiyuan Li, Srinadh Bhojanapalli. Yann LeCun, and Nathan' Srebro. The role of over-parametrizzatoon generalization of neural networks. In International Conference on Learn- ing Representationss 2019. URL BRLLttts:///oeeeeeee noteId=BygfghAc1x
net. wtt.:eteeooommmee.agggggggee
Myle Ott, Sergey Edunov, David Grangier, and Michael Auli. Scaling neural machine translation. In Proc. of WMT, 2018.
Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli, fairseq: A fast, extensible toolkit for sequence modeling. Proceedings of NAACL-HLT 2019: Demonstrations, 2019.
Zhuwei Qin, Fuxun Yu, Chenchen Liu, and Xiang Chen. Interpretable convolutional filter pruning, 2019. http:////pppeerreeeeeeee//rorroa
Maithra Raghu, Alex Irpan, Jacob Andreas, Robert Kleinberg, Quoc V Le, and Jon Kleinberg. Can deep reinforcement learning solve Erdos-Selfie Seefridge-Spennen games? In International Conference in Machine Learning (ICML), 2018.
Ali Sharif Razavian, Hossein Azizpour, Josephine Sullivan, and Stefan Carlsson. Cnn features off- the-shelf: An astounding baseline for recognition. In Proceedings of the 2014 IEEE Conference on Computer Vision and Pattern Recognition Workshops, CVPRW '14 pp. 512-519, Washington, DC, USA, 2014. IEEE Computer Society. ISBN 978- 4799-44008-1 doi: 10. 19110099//RW...11.111111
URL http: /ttp://dxz doi. wrw.i.oor....... wtt./////......///....
1ww9/111EEEE http::://x..co.or//11100 2014.0111111 131
Avraham Ruderman, Richard Everett, Bristy Sikder, Hubert Soyer, Jonathan Uesato, Ananya Kumar, Charlie Beattie, and Pushmeet Kohli. Uncovering surprising behaviors reinforcement learning via worst-case analysis, 2019. URL
Yuandong Tian, Qucheng Gong, Wenling Shang, Yuxin Wu, and C. Lawrence Zitick.. Elf: An extensive, lightweight and flexible research platform for real-time strategy games. Advances in Neural Information Processing Systems (NIPS), 2017.
10

---

Published as a conference paper at ICLR 2020
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ukasz Kaiser, and Illia Polosukhin. Attention is all you need. I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neural Information Processing Systems 30, pp. 1998-6008 Curran Associates, Inc., 2017. URL http: 7/papers n1 ps. wttpss/ppppppppcapcccaaeceepa wtt.://tttaattttttniccmm on- a- w-n-----ll--o------a -ouu--e--ueeed pdf
Jason Yosinski, Jeff Clune, Yoshua Bengio, and Hod Lipson. How transferable are features in deep neural networks? In Z. Ghahramani, M. Welling, C. Cortes, N. D. Lawrence, and Q. Weinberger (eds.), Advances in Neural Information Processing Systems 27, pp. Curran Associates, Inc., 2014. URL
cctpppscccppprrrreee 5347 trw.sfraaeerablleee are features- in deep--eereeeee neerall-nttooe pdf
Hattie Zhou, Janice Lan, Rosanne Liu, and Jason Yosinski. Deconstructing lottery tickets: Zeros, signs, and the supermask. In Neural Information Processing Symposium (NeurIPS), 2019.

---

Published as a conference paper at ICLR 2020
A
APPENDIX
Type
Name
Network spees
Algorithm
NN
CarPPole-VO
MM.P(128-128112--u
A2C
20
160 (games)
100
Classic
Acrohot-vl LunarLander-v2
MM.P(256-256-566-uu)
A2C
20
320 (games)
100
MILPI256-256-25--0u))
A2C
20
640 (games)
100
Assault, Berzerk, Breakout, Centipede, Kangaroo, Krull, 1onv(3,64,1,1)Maazz
Conv(5.6.1.2--Mxxxoolll
A2C
Conv(5,64.1.2)-Maxx))
(with
1000 (batches)
Pixel
importance
25
1024
Qbert. Seaquest,
Conv(3.64.1.))MMxxxPo
factor
Space Invaders
-MLPP11200-51--512-00)
correction)
Table Al: A summary of the games in our RL experiments. Conv(w, x.z1.2) represents a convolution layer of filter size w, channel number z, stride y, and padding 2, respectively. All the layer activations are ReLUs. See Sec. 3.3.2 for the mcaning M, N and
12


Reviews:
[{"review_id": "HyxTjvVaKH", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper studies the existence of “winning ticket” initialization in natural language processing (NLP) and reinforcement learning (RL). The lottery ticket hypothesis has been found effective in over-parameterized deep neural networks, which provides better sub-network initialization if not outperforming the original full network. Experiments show that winning ticket initializations generally outperform parameter-matched random initializations on recurrent LSTM models, large-scale Transformer, and discrete-action space RL tasks, including both classic control and pixel control.  A substantial number of parameters can be saved.\n\nThis paper is clearly motivated and easy to follow.  The results are interesting.  However, my major concern is its intellectual merit.  The paper does not seem to propose any new algorithm, and the application of lottery ticket (late rewinding) to LSTM, Transformer, and RL looks quite straightforward. There are a few insights drawn from the experiment, such as when only the Transformer weights are pruned.  However they does not appear substantial. Overall, I do not find the paper innovative enough for publication at top conferences like ICLR.  I understand this is subjective, so I leave it to the AC for further evaluation.", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "Hkx3RhBR5r", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper used the lottery ticket hypothesis to study the over-parameterization of deep neural networks (DNNs). The main idea is that overparametrization increases the probability of a “lucky” sub-network initialization being present rather than by helping the optimization process. \n\nThe paper conducted experiments to evaluate whether “winning ticket” initializations exist in two different domains: natural language processing (NLP) and reinforcement learning (RL). The authors confirm that winning ticket initializations\ngenerally outperform parameter-matched random initializations, even at extreme pruning rates for both NLP and RL. The results suggest that the lottery ticket\nhypothesis is not restricted to supervised learning\n\nThe similarity between supervised learning and RL and NLP problem is obvious from a function approximation and optimization point of view. The paper is empirical in nature, and do not offer any additional insight. \nThe experiments are not very conclusive. ", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper."}, "extra_text": {}}, {"review_id": "BkxEUdm9FB", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper describes an application of lottery ticket hypothesis to NLP and RL problems. An extensive set of experiments on very strong baseline models on language modeling, machine translation and atari games demonstrates that lottery ticket hypothesis is not only present in feed-forward and convolutional nets on image classification tasks as demonstrated originally in papers by Frankle & Carbin, 2019 and Frankle et al 2019. I don't have any major complaints regarding this work and I believe it is well executed.\n\nFor NLP problems it would be nice to have additional ablation studies comparing lottery ticket hypothesis pruning with methods like distillation (Hinton et al 2014) and attention pruning (Michel et al 2019 Are Sixteen Heads Really Better than One?). \n\nFor RL it is quite interesting that pruning weights sometimes improves the final scores on Atari games like Berzerk, Kangaroo, Krull and Centipede perhaps due to exploration. As a future work it would be interesting to see a way to using lottery ticket hypothesis to guide exploration in RL.\n\nOverall it is well executed paper, although it is mainly an application of existing lottery ticket hypothesis techniques to NLP and RL.", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper explores the application of the lottery ticket hypothesis to NLP and RL problems for better initialisations of deep networks and reduced model sizes. This is evaluated in a variety of settings, including continuous control and ATARI games for RL, and LSTMs and Transformers for NLP, showing very positive results.

The main issue raised by the reviewers was the lack of algorithmic novelty in the paper. Despite this, I believe the paper to present an important contribution that could stimulate much additional research. The paper is well written and the results are rigorous and interesting. For these reasons I recommend acceptance.
Decision: Accept (Poster)