How many weights are enough : can tensor factorization learn efficient policies ?
Authors: Pierre H. Richemond
Arinbjorn Kolbeinsson
Yike Guo
Venue:  (2020)
Keywords: reinforcement learning
Q-learning
tensor factorization
low-rank approximation
data efficiency
second-order optimization
scattering

Abstract: Deep reinforcement learning requires a heavy price in terms of sample efficiency and overparameterization in the neural networks used for function approximation. In this work, we employ tensor factorization in order to learn more compact representations for reinforcement learning policies. We show empirically that in the low-data regime, it is possible to learn online policies with 2 to 10 times less total coefficients, with little to no loss of performance. We also leverage progress in second order optimization, and use the theory of wavelet scattering to further reduce the number of learned coefficients, by foregoing learning the topmost convolutional layer filters altogether. We evaluate our results on the Atari suite against recent baseline algorithms that represent the state-of-the-art in data efficiency, and get comparable results with an order of magnitude gain in weight parsimony.

Under review as a conference paper at ICLR 2020
HOW MANY WEIGHTS ARE ENOUGH : CAN TENSOR FACTORIZATION LEARN EFFICIENT POLICIES ?
Anonymous authors Paper under double-blind review
ABSTRACT
Deep reinforcement learning requires a heavy price in terms of sample efficiency and overparameterization in the neural networks used for function approximation. In this work, we employ tensor factorization in order to learn more compact rep- resentations for reinforcement learning policies. We show empirically that in the low-data regime, it is possible to learn online policies with 2 to 10 times less to- tal coefficients, with little to no loss of performance. We also leverage progress in second order optimization, and use the theory of waveler scattering to further reduce the number of learned coefficients, by foregoing learning the topmost con- volutional layer filters altogether. We evaluate our results on the Atari suite against recent baseline algorithms that represent the state-of-the-an aftt in data efficiency, and get comparable results with an order of magnitude gain in weight parsimony.
INTRODUCTION
The successes of reinforcement learning (thereafter 'RL', and specifically deep RL., come at heavy computational price. It is well known that achieving human-level performance in domains such as Atari (Sutton & Barto 2018 Mnih et al., 2013 Hessel et al. 2017)) requires hundreds of millions of frames of environment interaction. As such, the problem of sample efficiency in RL is of critical importance. Several tracks of concurrent research are being investigated, and have reduced by orders of magnitude the number of environment interactions required for good performance beyond the previous benchmark of polloicaall-innsieee episodic control methods (Blundell et al., 2016 Pritzel al. 2017 to a couple hours of human gameplay time (van Hasselt et al. 2019 Kaiser et al. 2019).
However, while the data-efficiency of RL methods has seen recent drastic performance gains, the function approximators they use still require millions of learned weights, potentially still leaving them heavily overparameterized. Independently motivated by biological facts like the behavioural readiness of newborn animals, several authors (Gaier Ha 2019: Cuceu et al.) 2018 Wang et al.) 2019) have recently looked at doing away with learning
many weights for RL tasks. Smaller networks not only train faster, but may yet offer another avenue for gains in the form of better gener- alization (Zhang et al. 2016). Recent work from Gaier & Ha 2019) studies the effect of inductive bias of neural architectures in RL, they forego training altogether, but transfer networks that only obtain "better than chance performance on MNIST'. In similar fashion, WWang al. (2019) investi- gate the effect of random projections in the restricted setting of imitation learning. Finally, Cuccu al. (2018) manage human-level performance on the Atari suite using separate dictionary learning procedure for their features, bypassing the usual end-to-end learning paradigm. The perspective of neural architecture search applied to RL appears difficult, if not computationally inextricable.
Concurrently, the study of lelooiccall-innspiee models of learning has exhibited two mathematical characterizations that might be critical in explaining how biological learning takes place so effi- ciently. First, the low-rank properties of learned perceptual manifolds (Chung et al., 2018 2016) are giving rise to a rich theory borrowing from statistical physics. Second, another well known line of work has identified Gabor filters, and more generally wavelet filter-like structures, in the actual visual cortex animals (Jones & Palmer 1987). and linked those to sparsity-promotin promoting methods and dictionary learning (Oshhassen & Field 1996 1997, Hyvärinen & Hoyer 2001). But these breakthroughs have not, so far, been reflected as inductive priors in the shape of modifications in deep RI. neural networks architectures, which remain fairly fixed the Atari domain.

---

Under review as a conference paper at ICLR 2020
Therefore the following questions remain: how parsimonious do function approximators in RL need to be, in order to maintain good performance? And can we be at once efficient ? In this work, we turn to the mathematical theories of tensor factorization (Cichocki et al.) 2009), second-order optimization (Amari) 1998, Martens & Grosse 2015) and wavelet scattering (Mallat 2011) to answer this question positively and empirically, in model-free setting.
and weight-
We propose to use these methods in order to save weights and therefore favour convergence of policies:
We replace dense, fully-connected layers with tensor regression layers.
Optionally, we replace the topmost layer in the convolutional architecture with a scattering layer; the deeper convolutional layers are left untouched.
The (positive) impact of second-order optimization also evaluated.
To the best of our knowledge, this the first time those fields have been combined together this context, and that tensor factorization is applied to deep RI..
BACKGROUND & RELATED WORK
2.1 DEEP REINFORCEMENT LEARNING
We consider the standard Markov Decision Process framework as in Sutton & Barto (2018). This setting is characterised by a tuple (S, A, T, 7) where S is a set of states, A set of actions, R reward function that is the immediate, intrinsic desirability of certain state, T transition dynamics and y E [0, 1] discount factor. The purpose of the RL problem is to to find policy x, which represents mapping from states to probability distribution over actions, that is optimal, i.e. that maximizes the expected cumulative discounted return o R++k+1 at each state 8t E S. In Q-learning, the policy is given implicitly by acting greedily or é-greedily with respect to learned action-value functions q" (s, a), that are learned following the Bellman equation. In deep Q-learning, ge becomes parameterized by the weights 0 of neural network and one minimizes the expected Bellman loss:
E [Ro+1 + 1 max go (S++1, a') - (S., A.)
a'
In practice, this is implemented stochastically via uniform sampling of transitions in experience replay buffer, as is done in the seminal paper Mnih et al. (2013). Several algorithmic refinements to that approach exist. First, Double Q-learning (van Hasselt al.] 2015) proposes decouple learning between two networks in order to alleviate the Q-value overestimation problem. Second, dueling Q-networks (Wang et al., 2015) explicitly decompose the learning of an action-value func- tion (s, a) as the sum of an action- indienndedeeendentt state-value, much like what is traditionally done policy gradient methods (Sutton & Barto, 2018), implemented via two-headed neural network architecture. Finally, prioritized RI. (Schaul et al. 2015) proposes to replace the uniform sampling of transitions in the experience replay buffer with importance sampling, by prioritizing those transi- tions that present the most Bellman error (those transitions that are deemed the most 'surprising' by the agent). Fortunato et al. (2017) use extra weights to learn the variance of the exploration noise in granular fashion, while Bellemare et al. (201)) propose learn full distribution of action-values for each action and state. Combined, those methods form the basis of the Rainbow algorithm in Hessel et al. (201)).
2.2 TENSOR FACTORIZATION
Here introduce notations and concepts from the tensor factorization literature. An intuition is that the two main decompositions below, CP and Tucker decompositions, can be understood as multilinear algebra analogues of SVD eigendecomposition.
CP decomposition. A tensor X x can be decomposed into sum of R rank-1 tensors, known as the Canonical-Polyadii decomposition, where It is the rank of the decomposition.
(1) (2) (N) for k [1... ... R). well vector of weights
Its purpose is to find vectors u up ... ,u
2

---

Under review as a conference paper at ICLR 2020
À IR such that:
R
X =
u
o k o ..... uk
Krm
rank-1 componenis
Tucker decomposition. A tensor &' E R can be decomposed into a low rank approx-
imation including core G E R x xRxx-XRR and set of projection factors (U("), ,... U with (*a) RR k E (0,.... N - 1) that, when projected along the corresponding dimension of the core, reconstruct the full tensor X. The tensor in its decomposed form can written:
X' 9 X1 U X2 U x XN U ( .... [
Tensor regression layer. For two tensors X' R xxxxxxxxx x xx"xx,, denote by (X, V)N X by y along their N last modes; their generalized inner product
R and y
R x the contraction of
(2,))N
Σ 4=11===
Σ
This makes possible to define a lensor regression layer (Kossafi al. 2017b) that differentiable and learnable end-to-end by gradient descent. Let us denote by X xxx.xxxxxxxx the input activation tensor for sample and y E R the label vector. A tensor regression layer estimates the regression weight tensor W E R4x under low-rank decomposition. In the case of Tucker decomposition (as per our experiments) with ranks (Ri,.., Rx), we have
y (X, +b
with W 9 U U xN U(N)
as 9 E RR Utk) E RRIxXR for each l: in NJ and U(N) e RIXRN
2.3 WAVELET SCATTERING
The wavelet scattering transform was originally introduced by Mallat 2011 and Bruna & Mallat (012) as a non-linear extension to the classical wavelet filter bank decomposition. Its principle as follows. Denoting
y/n] the 2-dimensional, circular convolution of two signals x[n] and ylu]. let us assume that we have pre-defined two wavelet filter banks available {x)) A.EA,
( [m]] AgEA, with \ and X2 two frequency indices. These wavelet filters correspond to high frequencies, so we also give ourselves the data of a lowpass filter Φ pn]n.. Finally, and by opposition to traditional linear wavelet transforms, we also assume given nonlinearity p(t). Then the scattering transform is given by coefficients order 0,1, and respectively:
Saz(n] = & 6s|n
Six (n, p
Sax in, \1, A2) = p
x
d d E
This can effectively be understood and implemented as a two-layer convolutional neural network whose weights are not learned but rather frozen and given by the coefficients of wavelets & and (with Gabor filters as a special case (Mallat) 1998)). The difference with traditional filter banks comes from the iterated modulus/nonlinear activation function applied at each stage, much like traditional deep learning convolutional neural networks. The generic mathematical definition involves order iterated scatterings, in the vein of S,r above, but sometimes restricts nonlinearity to be a modulus function I. practice, the potential of scattering transforms to accelerate learning by providing ready-made convolutional layers has been investigated in Oyallon al.) (2013) and Oyallon et al. (2018) and a subject of active ongoing research. Scattering will be the second of weight-saving methods.
3

---

Under review as a conference paper at ICLR 2020
2.4 SECOND ORDER OPTIMIZATION WITH K-FAC
While stochastic gradient descent is usually performed purely from gradient observations derived from auto-differentiation, faster, second order optimization methods first multiply the weights' gradient vector Ve by a preconditioning matrix, yielding the weight update 0n+1 + On 1/ Ve. with 7) a step size. In the case of second order methods, the matrix G, is chosen act as a tractable iterative approximation to the inverse Hessian or Empirical Fisher Information Matrix (Amari 1998) of the neural network model in question.
sronecker-factores approximate curvature or K-FAC (Martens & Grosse 2015) enforces Kronecker decomposition of the type G = A & B, with A and B being smaller, architecture-dependen matrices. Unlike the above methods, K-FAC has been applied a plug-in in the deep RL literature and shown to promote both anytime convergence properties as well as terminal accuracies (Wu et al. 2017).
3 OBSERVATIONS AND METHODS
3.1 EXPLORING TRAINED AGENTS
Stability of trained dense layers eigenvalues. In order to assess experimentally if tensor factor- ization can make sense in RI,, we investigate the eigenvalues of the dense layers of a deep RL agent. Unlike the traditional supervised learning setting, the input data distribution RL function approximators shifts as the agent explores its environment; as such, concentration properties of the eigenvalues of the linear layers cannot be guaranteed all the way throughout training. Since condi- tioning techniques such as batch normalization (loffe & Szegedy (015) are rarely used in deep RL, this is all the more important. Our experiments (see figure 1 show that the distribution of eigen- values does not seem to widen significantly, at least during the initial phases of training we care about.
Frame #
Frame :0,003
Froms
Fram 30
Trame 40,000
125
8.35
825
Frrme e 50,300
Frome 60,003
Frome 70 603
Fame 80 ace
Fraee # $6,000
355
325
850
025
100
2
Egenvatus
Figure 1: Eigenvalue histograms of the value-based linear layer during training. 50 agent runs data- efficient Rainbow (van Hasselt et al. 2019)) of 100,000 steps the Atari game Road Runner.
Furthermore and interestingly, it does not seem to deviate significantly from the one observed at initialization. All together, this suggests there might be some merit in learning low-rank policies.
3.2 ARCHITECTURAL MODIFICATIONS
Baseline. We then proceed to build upon the well-known Rainbow algorithm (Hessel et al. 2017). Rainbow uses fairly standard shallow convolutional architecture like the seminal DQN paper of Mnih et al. (2013), and is an oft-cited baseline on the Atari suite. In spite of its performance, Rain- bow often requires dozens of millions of a single game's frames in order to perform well. Very re- cently, 'dttaefffccient" efficient version of Rainbow has been proposed by van Hasselt et al. (2019), with a view to match or beat the latest of-te-of-the-arr results achieved by model-basee based RL methods. This is achieved with major change in network architecture, but via selection of mildly hand- tuned hyperparameters favouring efficiency against wall-clock running time (see appendix). We do take this baseline.
4

---

Under review as a conference paper at ICLR 2020
Input
Convolutions
Tensor regression
Q values
Scattering
Input
Convolutions
values
Scattering
Tensor regression
Input
Convolutions
Q values
Figure 2:
Our architectural approach consists in replacing hidden layers in deep RL agents with tensor regression (top). Optionally we substitute the topmost convolutional layer with scattering (middle), and combine both methods (bottom).
Changes. We modify the architecture of the neural network function approximators used, in accor- dance with the principles described above, combining them to reflect inductive biases promoting fewer learnable parameters:
We replace the fully- colly-connected, linear layers used in Rainbow and data-efficient Rainbow with tensor regression layers (Kossaifi et al. 2017b) in order to learn low-rank policies (ranks in appendix).
We either the K-FAC second order stochastic optimizer, or the standard ADAM opti- mizer (Kingma Ba 2014). Optimization with K-FAC yields better results ceteris paribus and therefore works to counter performance loss due to using fewer weights.
We combine the two methods with various rank (and therefore weight compression) ratios, targetting sensible compression ratios guided by deep learning intuition; and evaluate those on the same subset of Atari games as both van Hasselt et al. 2019): Kaiser al. (2019).
When possible, we replace the first convolutional layer in the approximating neural network with a scattering layer for further gains in terms of learnable weights. In that way, we inves- tigate the impact of not actually learning one of the convolutional layer weights. Deep RL, paradoxically, tends to shallow network architectures with two three convolutional layers only, which makes it very fit for scattering methods. But since learned convolutional features discriminate well between high and low rewards, the promise of fully unsupervised scattering layers seems remote, without resorting to further ad-hoc methods like dictionary learning. Therefore limit ourselves one-step (one single layer) scatterings.
This is illustrated in figure 2 We then proceed to evaluate the merit of these changes.
EXPERIMENTAL RESULTS
4.1 PRIORITIZED TENSORIZED DQN
Proof of concept. We begin with showing proof of concept on the simple Pong Atari game. Our experimental setup consists in our own implementation of prioritized double DQN as a baseline (Schaul al. 2015: van Hasselt al. 2015). We replaced the densely connected layer of the original DQN architecture with a tensor regression layer implementing Tucker decomposition for different Tucker ranks, yielding different network compression factors.
5

---

Under review as a conference paper at ICLR 2020
Qualitative behaviour. First results, both in terms of learning performance and compression factor, can be seen in figure 3 Our two main findings are that first, the final performance of the agent remains unaffected by the tensor factorization,
with high compression rates with respect to all network weights of five times. In line with intuition, larger compression rates do however cause more delays in learning. Second, tensor factorization negatively affects stability during training in tough compression regimes, the plateauing phases of learning curves feature occasional noisy drawdowns, illustrating the increased difficulty of learning, as seen in figure Interestingly, approx- imation errors incurred by tensor regression noise do sometimes have poor consequences illustrated by those drawdowns, but overall seem behave as additional exploration noise.
15
14
Mediu compression)
-15
10 compression
8x compressian)
-28
(5 0x compressian)
Figure 3: Prioritized tensorized DQN on Atari Pong. Original learning curve versus several learning curves for five different Tucker ranks factorizations and therefore parameter compression rates (3 different random seeds each, with 30 episodes moving average for legibility). Best viewed in colour.
Tucker
-10
200
Figure 4: Focus typical single run of the tensorized DQN learning (score number of thou- sand episodes). The overall shape of the typical learning curve is preserved, but drawdowns in the plateauing phase do appear.
DATA-EFFICIENT RAINBOW ON ATARI
Evaluation protocol. For all our Atari experiments, used OpenAl Gym (Brockman et al. 2016), and a combination of PyTorch (Paszke et al.) (017). TensorLy (Kossaifi et al. (0166) and Kymatio (Andreux et al., 2018) for auto-differentiation. We evaluated our agents in the low-data regime of

---

amidar Under review as a conference paper at ICLR 2020
100,000 steps, half the games, with different random seeds for reproducibility (Henderson et al. 2017), taking the data-efficient Rainbow agent (van Hasselt et al. 2019) as our bascline. Our specific hyperparameters described in appendix. We report our results in tables and
Game alien
SimPLe
Rainbow
Denoised
TRL 2.5x
TRL
TRL 10x
405
740
684
688
454
566
88
189
154
118
86
84
assault
369
431
321
543
521
513
asterix
1090
471
500
459
554
363
benk_heist
51
77
59
134
42
battle zone
5184
10125
9378
14466
13466
5744
boxing
0.2
1
-2
-2
-5
breakout
13
2
3
chopper.commant command
1247
862
1293
1255
1243
1106
39828
16185
9977
3928
4225
2340
demon_attack
170
508
450
362
263
175
freeway
20
28
28
26
25
24
frostbite
255
867
1101
659
912
231
gopher
771
349
391
278
255
396
hero
1295
6857
3013
5351
3732
3321
jamesbond
125
302
295
215
213
218
kangaroo
323
779
1002
804
715
400
krull
4540
2852
2656
2333
2275
2308
kung fu master
17257
14346
4037
9392
4764
4031
ms pacman
763
1204
1053
818
838
517
pong
-19
-20
-20
-19
-21
private_eye
58
98
100
51
100
1128
qbert
560
1153
672
697
581
733
road runner
5169.4
9600
5426
6965
3914
1319
seaquest
371
354
387
345
350
287
up up_n_down
2153
2877
5123
2197
2302
2179
Average (ys. Rainbow)
100%
118%
96
90%
71%
Table 1: Mean episode returns reported in baselines SimPLe (Kaiser et al.
2019) and data-efficient
Rainbow (van Hasselt et al. 2019), versus our agents, on 26 Atari games. "Denoised" is the NoisyNet ablation of Rainbow; 'TRL' shows the performance of the data-efficient Rainbow with tensor regres- sion layers substituted for linear ones.
Table shows proof of concept the online learning low-rank policies, with loss of final performance varying in proportion to the compression in the low-rank linear layers. very much like in the deep learning literature (Kossaifi et al. 2017a|b) b). The number of coefficients in the original data-efficient Rainbow is of the order of magnitude of MM and varies depending on the environment and its action-space size. The corresponding tensor regression layer ranks are appendix, and chosen target 400k, 200k and 100k coefficients respectively. While individual game results tend to decrease monotonously with increasing compression, we observe that they are noisy due to the nature of exploration in RL, and average scores reported correspond the intuition that performance seems to decrease fast after a certain overparameteriiztion threshold is crossed. To take this noisy character into account, we take care to be conservative and report the average of the final three episodes of the learned policy after 80k, 90k and 100k steps, respectively.
Denoised baseline. So as to not muddy the discussion and provide fair baselines, we do report on the NoisyNet (Fortunato al. (017)) ablation of Rainbow ("Denoised" columns), as the NoisyLinear layer doubles up the number coefficients required and actually performs worse our experiments. Its principle is that the variance of exploration noise represents a criticial tradeoff for performance (too little and one stalls, too much and one risks catastrophic updates), so is sensible to treat parameter learn. However, in order to decouple both factors of overparametriization and exploration in the discussion of deep RL performance, we simply use fixed exploration schedule. This denoised exploration baseline is ablation
then compare tensorized methods

---

Under review as a conference paper at ICLR 2020
Second-order optimization. We then proceed assess the impact of second-order optimization in architecture by substituting ADAM optimization for K-FAC, and introducing scattering, in table 2] In spite of our conservative reporting. the efficiency boost from using second order scheme more than makes up for low-rank approximation error (109% performance) with five times less co- efficients than van Hasselt et al. (2019), and learning with full order of magnitude less coefficients (98% performance) is made possible by the combination of K-FAC and TRL. The results however do show sharp drop in average performance when scattering added. Interestingly enough some games perform relatively very well with that method, simultaneously showing proof of viability and of additional work required.
Game alien
KFACCDenoised
KFAC+TRL5x
KFAC+TRLIOX Scattering
996
734
643
441
amidar assault asterix
163
101
98
84
501
491
496
434
537
549
526
502
bank heist
100
73
57
29
battle_zone
8622
15178
6156
4311
boxing
-1
-9
breakout
3
chopper_commane command
692
611
1302
441
crazy_climber
14242
12377
3546
740
demon attack
582
434
318
692
freeway
26
26
24
19
frostbite
1760
718
1483
654
gopher
363
341
265
172
hero
4188
6284
4206
4127
jamesbond
263
327
217
48
kangaroo
2085
613
588
391
krull
2855
3441
3392
772
kung fu master
8481
10738
7357
233
ms-pacman
1137
920
867
613
pong
-9.33
-19
-19
-20
private eye
56
100
100
0
qbert
731
520
538
475
road_runner
4516
8493
7224
1278
seaquest
349
317
520
213
up_n.down
2557
2291
2108
993
Average (vs. Rainbow)
114%
109%
98%
55%
Table 2: Mean episode returns of our low-rank agents with second-order optimization and scattering. The Scattering column also includes KFAC optimization and TRL 5x, resulting in around 10x total weights efficiency gains.
CONCLUSION
We have demonstrated that in the low-data regime, possible leverage biologically plausible characterizations of experience data (namely low-rank properties and wavelet scattering separability) exhibit architectures that learn policies with an order of magnitude less weights than current state-of-the theth-rrrt baselines, essentially without loss of performance, and in an online fashion. In particular, this provides a compelling alternative to methods like policy distillation (Rusu et al.) 2015 Czarnecki al., 2019). We do hope that this will lead to even further progress towards sample efficiency and speedy exploration methods. Further work will, first, focus on thorough evaluation and research of scattering architectures in order to achieve further gains, and second investigate additional, orthogonal biologically-friendl research directions such as promoting sparsity via, for instance, L regularization. Finally, we excited by the potential of tensor factorization to offer shared core tensors for policies in multi-tale task and meta-learning.
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
Shunichi Amari. Natural gradient works efficiently in learning. Neural Computation, 10:251-276. 1998.
M. Andreux, T. Angles, G. Exarchakis, R. Leonarduzzi, G. Rochette, L. Thiry, J. Zarka, S. Mallat, J. andén, E. Belilovsky, Bruna, V. Lostanlen, M. J. Him, E. Oyallon, S. Zhang, C. Cella, and M. Eickenberg. Kymatio: Scattering Transforms Python. arXiv e-prints, December 2018.
M. G. Bellemare, W. Dabney, and R. Munos. A Distributional Perspective on Reinforcement Learn-
ing. arXiv e-prints, July 2017.
C. Blundell, B. Uria, A. Pritzel, Li, A. Ruderman, J. Z. Leibo, Rae, D. Wierstra, and D. Hassabis. Model-Free Episodic Control. arXiv e-prints, June 2016.
G. Brockman, V. Cheung, L. Pettersson, J. Schneider, Schulman, J. Tang, and W. Zaremba. Ope- nAI Gym. arXiv e-prints, June 2016.
Bruna and S. Mallat. Invariant Scattering Convolution Networks. arXiv e-prints, March 2012.
S. Chung, D. D. Lee, and H. Sompolinsky. Linear readout of object manifolds. 93(6):060301, June 2016. doi: 10. 101/PhyyseeeE..5.0000000
Chung, D. Lee, and H. Sompolinsky. Classification and Geometry of General Perceptual Manifolds. Physical Review X, 8(3):031003, July 2018. doi: 10.1 11100//hyseexx...0310000
Andrzej Cichocki, Rafal Zdunek, Anh Huy Phan, and Sh. Amari. Nonnegative Matrix and Tensor Factorizations Applications to Exploratory Multi-way Data Analysis and Blind Source Separa- tion. 2009.
G. Cuccu, J. Togelius, and P. Cudre-Maurous Playing Atari with Six Neurons. arXiv e-prints, June 2018.
W. M. Czarnecki, R. Pascanu, Osindero, M. Jayakumar, Swirszez, and M. Jaderberg. Dis- tilling Policy Distillation. arXiv e-prints, February 2019.
M. Eickenberg, G. Exarchakis, M. Hirn, S. Mallat, and L. Thiry. Solid harmonic wavelet scattering for predictions of molecule properties. 148(24))241732, June 2018. doi: 10. 10.1063/1 00.1063/1.5023999
M. Fortunato, M. Gheshlaghi Azar, B. Piot, J. Menick, I. Osband, A. Graves, V. Mnih, R. Munos, D. Hassabis, O. Pietquin, C. Blundell, and S. Legg. Noisy Networks for Exploration. arXiv e-prints, June 2017.
A. Gaier and D. Ha. Weight Agnostic Neural Networks. arXiv e-prints, June 2019.
Henderson, R. Islam, P. Bachman, Pineau, D. Precup, and D. Meger. Deep Reinforcement Learning that Matters. arXiv e-prints, September 2017.
M. Hessel, Modayil, van Hasselt, Schaul, G. Ostrovski, W. Dabney, Horgan, B. Piot, M. Azar, and D. Silver. Rainbow: Combining Improvements in Deep Reinforcement Learning. arXiv e-prints, October 2017.
Aapo Hyvärinen and Patrik O. Hoyer. two-layer sparse coding model learns simple and complex cell receptive fields and topography from natural images. Vision Research, 41:2413-2423, 2001.
loffe and C. Szegedy. Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift. arXiv e-prints, February 2015.
Jeffrey P. Jones and Larry A. Palmer. An evaluation of the two-dimensional gabor filter model of simple receptive fields in cat striate cortex. Journal of neurophysiology. 58 6:1233-58, 1987.
L. Kaiser, M. Babaeizadeh, P. Milos, B. Osinski, R. H Campbell, K. Czechowski, D. Erhan, C. Finn, P. Kozakowski, S. Levine, A. Mohiuddin, R. Sepassi, G. Tucker, and H. Michalewski. Model- Based Reinforcement Learning for Atari. arXiv e-prints, March 2019.

---

Under review as a conference paper at ICLR 2020
D. P. Kingma and J. Ba. Adam: A Method for Stochastic Optimization. arXiv e-prints, December 2014.
J. Kossaifi, Y. Panagakis, A. Anandkumar, and M. Pantic. TensorLy: Tensor Learning in Python. arXiv e-prints, October 2016.
J. Kossaifi, A. Khanna, Z. C. Lipton, T. Furlanello, and A. Anandkumar. Tensor Contraction Layers for Parsimonious Deep Nets. arXiv e-prints, June 2017a.
J. Kossaifi, Z. C. Lipton, A. Khanna, T. Furlanello, and A. Anandkumar. Tensor Regression Net- works. arXiv e-prints, July 2017b.
Mallat, Group Invariant Scattering, arXiv e-prints, January 2011.
Stéphane Mallat. A Wavelet Tour of Signal Processing. 1998.
James Martens and Roger B. Grosse. Optimizing neural networks with kronecker-factored approxi- mate curvature. ArXiv, abs/ 100.0566,,,,,,, 2015.
V. Mnih, K. Kavukcuoglu, D. Silver, A. Graves, I. Antonoglou, D. Wierstra, and M. Riedmiller. Playing Atari with Deep Reinforcement Learning. arXiv e-prints, December 2013.
Bruno A. Olshausen and David Field. Emergence of simple-cell receptive field properties by learning sparse code for natural images. Nature, 381:607-609, 1996.
Bruno A. Olshausen and David J. Field. Sparse coding with an overcomplete basis set: strategy employed by v1? Vision Research, 97:3311-3325, 1997.
E. Oyallon, S. Mallat, and L. Sifre. Generie Deep Networks with Wavelet Scattering. arXiv e-prints, December 2013.
E. Oyallon, Zagoruyko, G. Huang, N. Komodakis, S. Lacoste-Julien, Blaschko, and E. Belilovsky. Scattering Networks for Hybrid Representation Learning. arXiv e-prints, Septem- ber 2018.
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang. Zachary Vito,, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. 2017.
A. Pritzel, B. Uria, S. Srinivasan, A. Puigdomënech, O. Vinyals, D. Hassabis, D. Wierstra, and C. Blundell. Neural Episodic Control. arXiv e-prints, March 2017.
A. A. Rusu, S. Gomez Colmenarejo, C. Gulcehre, G. Desjardins, J. Kirkpatrick, R. Pascanu, V. Mnih, K. Kavukcuoglu, and R. Hadsell. Policy Distillation. arXiv e-prints, November 2015.
T. Schaul, J. Quan, I. Antonoglou, and D. Silver. Prioritized Experience Replay. arXiv e-prints, November 2015.
Richard S. Sutton and Andrew G. Barto. Reinforcemment Learnnn An Introduction. The MIT Press, second edition, 2018. URL http::///eooooeeeeeeeess//////ss
html
H. van Hasselt, A. Guez, and D. Silver. Deep Reinforcement Learning with Double Q-learning. arXiv e-prints, September 2015.
H. van Hasselt, M. Hessel, and Aslanides. When to use parametric models in reinforcement learning? arXiv e-prints, June 2019.
R. Wang, C. Ciliberto, P. Amadori, and Y. Demiris. Random Expert Distillation: Imitation Learning via Expert Policy Support Estimation. arXiv e-prints, May 2019.
Z. Wang, T. Schaul, M. Hessel, H. van Hasselt, M. Lanctot, and N. de Freitas. Ducling Network Architectures for Deep Reinforcement Learning. arXiv e-prints, November 2015.
Y. Wu, E. Mansimov, S. Liao, R. Grosse, and J. Ba. Scalable trust-region method for deep reinforce- ment learning using fronecker-factorce approximation. arXiv e-prints, August 2017.
C. Zhang, S. Bengio, M. Hardt, B. Recht, and O. Vinyals. Understanding deep learning requires rethinking generalization. arXiv e-prints, November 2016.
10

---

Under review as a conference paper at ICLR 2020
A
APPENDIX
HYPERPARAMETERS AND REPRODUCIBILITY
Hyperparameters are follows. First, our specific architecture-modifeer hyperparameters:
Specific architecture hyperparameters
Value
Scattering maximum log-scale Scattering volume width M Scattering tensor input shape Scattering tensor output shape Scattering type
(1,4,84,84)
(1,16,11,1)
Harmonic 3D, Andreux et al. 2018
Eickenberg (2018)
Hidden linear layer rank constraint, 2.5x compression Final linear layer rank constraint, 2.5x compression Hidden linear layer rank constraint, 5x compression Final linear layer rank constraint, 5x compression Hidden linear layer rank constraint, 10x compression Final lincar layer rank constraint, 10x compression
128
48
32
48
16
10
KFAC Tikhonov regularization parameter
0.1
KFAC Update frequency inverses
Table 3: Our additional, architecture-specific hyperparameters.
Furthermore, we mirror the Data-Efficient Rainbow van Hasselt al. 2019 baseline:
Data-efficient Rainbow hyperparameters
Value
Grey-scaling
True
Observation down-sampling
(84, 84)
Frames stacked
Action repetitions
Reward clipping
[-1, I]
Terminal on loss of life
True
Max frames per episode
108K
Update
Distributional Double
Target network update period"
every 2000 updates bins
Support Q-distribution
Discount factor
0.99
Minibatch size
32
Optimizer
Adam
Optimizer: first moment decay
Optimizor: second moment decay
0.999
Optimizer:
0.00015
Max gradient norm
10
Priority exponent
0.5
Priority correction"
0.4 +
Hardware
NVidia 1080Ti GPU
Noisy nets parameter
0.1
Training frames
400,000
Min replay size for sampling
1600
Memory size
unbounded steps
Replay period every
Multi-step return length
20
network: channels
32, 64
network: filter size
5 5,
Q network: stride
5,
Q network: hidden units
256
Optimizer: learning rate
0.0001
Table 4: Data-efficient Rainbow agent hyperparameters, as per van Hasselt et al. 2019
11

---

Under review as a conference paper at ICLR 2020
Our codebase is available request.
STANDARD DEVIATIONS FOR SCORE RUNS
Game
Denoised
TRL 2.5x
TRL 10x
alien
684 + 7
688 + 123
5664 38
amidar
154 = 21
118 12
84 15
assault
321 + 224
543 = 94
513 64
asterix
500 124
459 91
363 66
bank heist
77 + 23
59 + 22
42 4
battle zone
9378 + 2042
14466 2845
5744 + 575
boxing
1 1
-2 +
-5
breakout
3 + 1.5
2 1
4 0.3
chopper.commant
1293 445
1255 215
1106
124
crazy climber
9977 + 3744
3928 + 221
2340 - 595
demon attack
450 +
362 +
175 +
freeway
28 + 0.6
26 0
24 - 0.5
frostbite
1101 355
659 + 523
231 +
gopher
391 + 46
278 = 39
396 + 24
hero
3013 90
5351 = 1948
3321 598
jamesbond
57
215 42
218
kangaroo
587
804 + 289
400 278
krull
2656 180
2333 + 309
2308 + 268
Lung_fu_mastr
4037 + 2962
9392 6289
4031 + 3068
ms- mss-pacman
1053 + 193
818 94
517 + 38
pong
-20 - 0.4
-0
-21 = 0.1
private.ey eeye
100 0
51
1128 + 1067
qbert
672 + 144
697 78
733 291
road runner
5426 + 2830
6965 6569
1319 216
seaquest
387 +: 24
345 + 40
287 + 87
up_n_down
5123 + 3146
2197 + 231
2179 + 178
Table 5: Standard deviations across seeds for presented Table
12

---

Under review as a conference paper at ICLR 2020
Game
KFAC+Denoised
KFAC+TRLIOx
Seattering
alien
996 180
643 51
441 90
amidar
163 + 15
26
84
assault
501 + 85
496 129
434 = 304
asterix
537 = 96
526 + 64
502 91
bank heist
100 + 14
57 + 36
29 = 13
battle zone
8622 + 5358
6156 + 1951
4311
1517
boxing
0 0
-1 + 3
9 12
breakout
3
2
2
phopper_commany
692 + 81
1302 = 328
4
crazy climber
14242 + 2936
3546 + 1231
740 291
demon attack
582 + 130
318 + 168
92 + 232
freeway
26 0
24
19
frostbite
1760 + 448
1483 + 466
654 + 709
gopher
363 4
265 67
1
hero
4188 = 1635
4206 + 1862
4127 = 1074
jamesbond
263 + 22
217 +
40
kangaroo
2085 + 2055
588 +
krull
2855 156
3392 2205
772+ 560
kung_fu,nng fu._tattrrrrrr
8481 + 8270
7357 + 9200
233 205
ms-pacman
1137 + 180
867 + 128
613 + 159
pong
-19 1 0.6
-19 +
-20 + 0
private.eye
56 + 42
100 + 0 538 114
0 0
qbert
731 256
475 + 161
road runner
4516 + 2869
7224 + 4598
1278 1-463
seaquest
349 +
520 +
213
up_n_down
2557 + 641
2108 + 298
993 + 244
Table 6: Standard deviations across seeds for runs presented Table 2.
13


Reviews:
[{"review_id": "ryxByGUaKS", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper aims at parsimonious reinforcement learning by employing 3\ndifferent techniques: using tensor regression layers (Kossaifi et al.,\n2017b), wavelet scattering (Mallat 2011) and using K-FAC (Kingma & Ba,\n2014) as the optimization method. Learning models with fewer\nweights is important not only in reinforcement learning but also\nin all other machine learning areas. With the combination of tensor\nregression layers and K-FAC, the proposed methods give comparable\nperformance on several Atari games against 2 other methods, SimpPLe\nand data-efficient Rainbow, while using 2 to 10 times fewer\ncoefficients than data-efficient Rainbow. The use of wavelet\nscattering provides improvement on 1 out of 26 Atari games. The paper\nalso points out an interesting concentration of eigenvalues of dense\nlayer of a deep RL agent which provides motivation for low-rank\npresentations.\n\nWhile the savings in terms of coefficients is positive, the obtained\nresults are of little surprise. Tensor regression layers and K-FAC are\nused as is without any modification while space savings and efficiency\nhave been reported in corresponding references. The performances of\nwavelet scattering for the reported tasks are weak (better in only one\ngame) and the space saving is not clear. The proposed improvements\nseem to be tailored to tasks with image inputs and hence reported\nresults are only on Atari games (possibly with sparse and low-rank\nimages). It is not clear if the proposed techniques can be applied to\na wider set of reinforcement learning tasks\n(e.g. https://gym.openai.com/envs/#mujoco).\n\n\nIt would be interesting to see if we can apply the proposed methods to\nother more diverse RL tasks. The performance of wavelet scattering does indeed need\nmore investigation and improvement. It would also be interesting to\ncompare the distributions of the eigenvalues of the tensor layers\nversus the dense layers in deep RL which may provide insights on the\nachieved savings and the compression trade-off.\n\n\nI have read  the authors' rebuttals. The reviews point to a number of directions \nwhere the contributions could be made more significant.\n\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "HyeUam4aKB", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper investigates a list of methods to reduce the number of weights for deep RL architecture under the low-data regime. These methods include tensor regression, wavelet scattering, as well as second-order optimization (K-FAC). The experiments on the Atari games shows that by using tensor regression to replace the dense layer of the neural nets and using K-FAC for the optimization, one can reduce around 10 times of parameters without losing too much of performance. \n\nHowever, I have some concerns on the novelty of this work and therefore I’m giving this paper a weak reject. Here are the reasons:\n\nTo begin with, leveraging tensor structure of the neural nets to reduce number of parameters while maintaining similar level or getting even better results have been done before, for example: Tensorizing Neural Networks (Novikov et al, 2015), Learning compact recurrent neural networks with block-term tensor decomposition (Ye et al., 2018) etc. Although the use of tensor regression might be new, the core idea is still to leverage the low rank property of the tensor and obtain a compression of the weight tensors. Moreover, why use Tucker decomposition specifically for the tensor regression? It has been proposed that using tensor train (TT) decomposition can also get very good results (see Garipov et. al. Ultimate tensorization: compressing convolutional and FC layers alike). Is it possible to investigate the use of TT decomposition for the dense layer of the deep RL architecture? Therefore the novelty for this aspect seems a bit weak for me.\n\nThe second method the authors have attempted is to swap the convolution layer of the deep RL architecture with wavelet scattering. For one particular game (demon_attack), this approach seems to outperform every other methods by a large margin. However the experiment shows that for the rest of the Atari games, there is a huge drop (45%) of performance. Therefore the significance of this approach is rather thin for me. Maybe some further investigation of the game demon_attack is needed to understand why using scattering in this game in particular gives such a huge performance boost. \n\nThirdly, as an approximation of the second order optimization, K-FAC does not really concern with the main theme of the paper, which is an investigation of potential weights reduction methods. It is great that the authors applied this techniques and seems to have great results. However, as the authors pointed out, K-FAC has been wildly applied in the deep RL literature, and the authors did not propose new extension for the K-FAC method, therefore the contribution of this matter is also quite thin. \n\nLast but not least, the writing of the paper is a bit clumsy, and I was having a hard time to figure out what exactly is the proposed method. I think this paper might need some rework on the writing to describe the idea of the authors in a more clear way for the publication. Due to these reasons, I’m giving this paper a weak reject. \n\nSome writing comments and potential writing errors (did not affect the decision):\nPage 3, first line of “Tensor regression layer”, the shape of the tensor X seems to be a typo. \nAlso here, the definition of <X, Y>_N in the paper is to sum over the dimension of I_1…I_N, then the shape of <X, Y>_N should be K_1*…*K_x*L_1*…*L_y, without the I_N in the middle. \nAlso in this section, the authors mentioned Tucker decomposition for the tensor regression. However the phrasing of this sentence needs a bit rework. The usage of “For instance here”, gives the readers a feeling that Tucker is just one possible way of doing this decomposition, but not necessarily the actual decomposition for the reported experiments. \nIn 2.3, there is a lack of definition for  \\Lambda_1 and \\Lambda_2. In addition, it would be better for the general readers to add a few definitions for the terminologies in this section. For example, “circular convolution”, “wavelet filter banks” etc. I guess people with corresponding background will understand it with no problem, however I do find myself a bit lost in this section with these terminologies. \n2.4 line 6, “with A and. B smaller, architecture-dependent matrices”. I think it should be “with A and B being…“\n3.1, line 5, “This is all the more pressing that….”, I did not understand this sentence. \nIn page 6, line 3, there is a lack of definition for “compression rate”. Is it the compression rate w.r.t only the last dense layer, or w.r.t the whole network?\nFigure 4 is lacking y-axis and x-axis labels. \n4.2, last bullet point, “However, one must not forget that the conv layers one learns must be somehow be well adapted…”, I get what you are saying, but the sentence is a bit clumsy. \nTable 1 and 2, the row name “Average” is lacking definition. \n\n Overall it is a good attempt to reduce the number of weights in the deep RL architecture, but I do think the novelty of this work is a bit thin and the three contributions were not tied together with the main theme of the paper. Therefore, I’m giving this work a weak reject. \n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "BkxhBqBBFB", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper suggests three different disconnected ideas for improving the number of parameters of deep vision models for playing ATARI and to improve the training speed.\n- Tensor-regression layer to replace fully connected layers\n- Wavelet-scattering layer to replace the first convolutional layer\n- Second order optimization (K-FAC)\nAll the ideas mentioned in this paper are existing ones (although properly attributed), so the novelty of this work is relatively low. \nThe paper mentions that this particular combination is \"novel\", but it is not clear is there is any significant synergy between these methods and why it should be considered interesting in this particular setup.\nAlso the paper conflates sample-efficiency with parameter-efficiency. However, there is no indication that any of these methods address sample-efficiency which would be an interesting and useful contribution.\nAlso the experiments are neither very conclusive nor are they easy to interpret. For example in the pong case, there is no discernable effect of the compression ratio as the highest and lowest compression give the best (and comparable) results. Also the results come without confidence intervals.\n\nSo, in general, I would consider this paper to be an uninspired combination of pre-existing ideas with weak and inconclusive experimental results: a clear reject.", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}]

Author rebuttal:

Meta-review: In this paper dense layers in deep neural networks representing policies are replaced by tensor regression layers, also by a scattering layer, and second-order optimization is considered. The paper does not have a single consistent message, and combines different techniques for unclear reason. Important related work is not cited. The presentation was found unclear by the reviewers. 
Decision: Reject