Title: HOW MANY WEIGHTS ARE ENOUGH : CAN TENSOR FACTORIZATION LEARN EFFICIENT POLICIES ?

Abstract: Deep reinforcement learning requires a heavy price in terms of sample efficiency and overparameterization in the neural networks used for function approximation. In this work, we employ tensor factorization in order to learn more compact representations for reinforcement learning policies. We show empirically that in the low-data regime, it is possible to learn online policies with 2 to 10 times less total coefficients, with little to no loss of performance. We also leverage progress in second order optimization, and use the theory of wavelet scattering to further reduce the number of learned coefficients, by foregoing learning the topmost convolutional layer filters altogether. We evaluate our results on the Atari suite against recent baseline algorithms that represent the state-of-the-art in data efficiency, and get comparable results with an order of magnitude gain in weight parsimony.

1. INTRODUCTION:

The successes of reinforcement learning (thereafter 'RL'), and specifically deep RL, come at a heavy computational price. It is well known that achieving human-level performance in domains such as Atari (Sutton & Barto, 2018; Mnih et al., 2013; Hessel et al., 2017) requires hundreds of millions of frames of environment interaction. As such, the problem of sample efficiency in RL is of critical importance. Several tracks of concurrent research are being investigated, and have reduced by orders of magnitude the number of environment interactions required for good performance beyond the previous benchmark of biologically-inspired episodic control methods (Blundell et al., 2016; Pritzel et al., 2017) to a couple hours of human gameplay time (van Hasselt et al., 2019; Kaiser et al., 2019) . However, while the data-efficiency of RL methods has seen recent drastic performance gains, the function approximators they use still require millions of learned weights, potentially still leaving them heavily overparameterized. Independently motivated by biological facts like the behavioural readiness of newborn animals, several authors (Gaier & Ha, 2019; Cuccu et al., 2018; Wang et al., 2019) have recently looked at doing away with learning so many weights for RL tasks. Smaller networks not only train faster, but may yet offer another avenue for gains in the form of better generalization (Zhang et al., 2016) . Recent work from Gaier & Ha (2019) studies the effect of inductive bias of neural architectures in RL ; they forego training altogether, but transfer networks that only obtain 'better than chance performance on MNIST'. In similar fashion, Wang et al. (2019) investigate the effect of random projections in the restricted setting of imitation learning. Finally, Cuccu et al. (2018) manage human-level performance on the Atari suite using a separate dictionary learning procedure for their features, bypassing the usual end-to-end learning paradigm. The perspective of neural architecture search applied to RL appears difficult, if not computationally inextricable.

Concurrently, the study of biologically-inspired models of learning has exhibited two mathematical characterizations that might be critical in explaining how biological learning takes place so efficiently. First, the low-rank properties of learned perceptual manifolds (Chung et al., 2018; 2016) are giving rise to a rich theory borrowing from statistical physics. Second, another well known line of work has identified Gabor filters, and more generally wavelet filter-like structures, in the actual visual cortex of animals (Jones & Palmer, 1987) , and linked those to sparsity-promoting methods and dictionary learning (Olshausen & Field, 1996; 1997; Hyvärinen & Hoyer, 2001) . But these breakthroughs have not, so far, been reflected as inductive priors in the shape of modifications in deep RL neural networks architectures, which remain fairly fixed on the Atari domain.

Therefore the following questions remain: how parsimonious do function approximators in RL need to be, in order to maintain good performance? And can we be at once sample-efficient and weightefficient ? In this work, we turn to the mathematical theories of tensor factorization (Cichocki et al., 2009) , second-order optimization (Amari, 1998; Martens & Grosse, 2015) and wavelet scattering (Mallat, 2011) to answer this question positively and empirically, in a model-free setting.

We propose to use these methods in order to save weights and therefore favour convergence of policies:

• We replace dense, fully-connected layers with tensor regression layers.

• Optionally, we replace the topmost layer in the convolutional architecture with a scattering layer; the deeper convolutional layers are left untouched.

• The (positive) impact of second-order optimization is also evaluated.

To the best of our knowledge, this is the first time those fields have been combined together in this context, and that tensor factorization is applied to deep RL.

2. BACKGROUND & RELATED WORK:



2.1. DEEP REINFORCEMENT LEARNING:

We consider the standard Markov Decision Process framework as in Sutton & Barto (2018) . This setting is characterised by a tuple S, A, T, R, γ , where S is a set of states, A a set of actions, R a reward function that is the immediate, intrinsic desirability of a certain state, T a transition dynamics and γ ∈ [0, 1] a discount factor. The purpose of the RL problem is to to find a policy π, which represents a mapping from states to a probability distribution over actions, that is optimal, i.e., that maximizes the expected cumulative discounted return ∞ k=0 γ k R t+k+1 at each state s t ∈ S. In Q-learning, the policy is given implicitly by acting greedily or -greedily with respect to learned action-value functions q π (s, a), that are learned following the Bellman equation. In deep Q-learning, q θ becomes parameterized by the weights θ of a neural network and one minimizes the expected Bellman loss :

E R t+1 + γ t+1 max a q θ (S t+1 , a ) -q θ (S t , A t ) 2

In practice, this is implemented stochastically via uniform sampling of transitions in an experience replay buffer, as is done in the seminal paper Mnih et al. (2013) . Several algorithmic refinements to that approach exist. First, Double Q-learning (van Hasselt et al., 2015) proposes to decouple learning between two networks in order to alleviate the Q-value overestimation problem. Second, dueling Q-networks (Wang et al., 2015) explicitly decompose the learning of an action-value function q θ (s, a) as the sum of an action-independent state-value, much like what is traditionally done in policy gradient methods (Sutton & Barto, 2018) , implemented via a two-headed neural network architecture. Finally, prioritized RL (Schaul et al., 2015) proposes to replace the uniform sampling of transitions in the experience replay buffer with importance sampling, by prioritizing those transitions that present the most Bellman error (those transitions that are deemed the most 'surprising' by the agent). Fortunato et al. (2017) use extra weights to learn the variance of the exploration noise in a granular fashion, while Bellemare et al. (2017) propose to learn a full distribution of action-values for each action and state. Combined, those methods form the basis of the Rainbow algorithm in Hessel et al. (2017) .

2.2. TENSOR FACTORIZATION:

Here we introduce notations and concepts from the tensor factorization literature. An intuition is that the two main decompositions below, CP and Tucker decompositions, can be understood as multilinear algebra analogues of SVD or eigendecomposition. CP decomposition. A tensor X ∈ R I1×I2×•••×I N , can be decomposed into a sum of R rank-1 tensors, known as the Canonical-Polyadic decomposition, where R is the rank of the decomposition. Its purpose is to find vectors u

(1) k , u (2) k , • • • , u (N ) k , for k = [1 . . . R],

as well as a vector of weights

λ ∈ R R such that: X = R k=1 λ k u (1) k • u (2) k • • • • • u (N ) k rank-1 components Tucker decomposition. A tensor X ∈ R I1×I2×•••×I N , can be decomposed into a low rank approx- imation including a core G ∈ R R1×R2×•••×R N and a set of projection factors U (0) , • • • , U (N -1) , with U (k) ∈ R R k , Îk , k ∈ (0, • • • , N -1)

that, when projected along the corresponding dimension of the core, reconstruct the full tensor X . The tensor in its decomposed form can be written: ..,i1,i2,...,in Y i1,i2,...,in,... This makes it possible to define a tensor regression layer (Kossaifi et al., 2017b) that is differentiable and learnable end-to-end by gradient descent. Let us denote by X ∈ R I1×I2×•••×I N the input activation tensor for a sample and y ∈ R I N the label vector. A tensor regression layer estimates the regression weight tensor W ∈ R I1×I2×•••×I N under a low-rank decomposition. In the case of a Tucker decomposition (as per our experiments) with ranks (R 1 , • • • , R N ), we have :

X = G × 1 U (1) × 2 U (2) × • • • × N U (N ) = î G; U (1) , • • • , U (N ) ó Tensor regression layer. For two tensors X ∈ R K1×•••×Kx×I1×•••×I N and Y ∈ R I1×•••×I N ×L1×•••×Ly , we denote by X , Y N ∈ R K1×•••×Kx×L1×•••×Ly the contraction of X by Y along their N last modes; their generalized inner product is X , Y N = I1 i1=1 I2 i2=1 • • • I N in=1 X .

y = X , W N + b with W = G × 1 U (1) × 2 U (2) • • • × N U (N ) as G ∈ R R1×•••×R N , U (k) ∈ R I k ×R k for each k in [1 . . . N ] and U (N ) ∈ R 1×R N .

2.3. WAVELET SCATTERING:

The wavelet scattering transform was originally introduced by Mallat (2011) and Bruna & Mallat (2012) as a non-linear extension to the classical wavelet filter bank decomposition. Its principle is as follows. Denoting by x y[n] the 2-dimensional, circular convolution of two signals x[n] and y[n], let us assume that we have pre-defined two wavelet filter banks available ¶ ψ

(1)

λ1 [n] © λ1∈Λ1 ¶ ψ (2) λ2 [n] © λ2∈Λ2

, with λ 1 and λ 2 two frequency indices. These wavelet filters correspond to high frequencies, so we also give ourselves the data of a lowpass filter φ J [n] . Finally, and by opposition to traditional linear wavelet transforms, we also assume a given nonlinearity ρ(t). Then the scattering transform is given by coefficients of order 0,1, and 2, respectively :

S 0 x[n] = x φ J [n] S 1 x [n, λ 1 ] = ρ Ä x ψ (1) λ1 ä φ J [n] λ 1 ∈ Λ 1 S 2 x [n, λ 1 , λ 2 ] = ρ Ä ρ Ä x ψ (1) λ1 ä ψ (2) λ2 ä φ J [n] λ 1 ∈ Λ 1 , λ 2 ∈ Λ 2 (λ 1 )

This can effectively be understood and implemented as a two-layer convolutional neural network whose weights are not learned but rather frozen and given by the coefficients of wavelets ψ and φ (with Gabor filters as a special case (Mallat, 1998) ). The difference with traditional filter banks comes from the iterated modulus/nonlinear activation function applied at each stage, much like in traditional deep learning convolutional neural networks. The generic mathematical definition involves order n iterated scatterings, in the vein of S i x above, but sometimes restricts nonlinearity ρ to be a modulus function | • |. In practice, the potential of scattering transforms to accelerate learning by providing ready-made convolutional layers has been investigated in Oyallon et al. (2013) and Oyallon et al. (2018) and is a subject of active ongoing research. Scattering will be the second of our weight-saving methods.

2.4. SECOND ORDER OPTIMIZATION WITH K-FAC:

While stochastic gradient descent is usually performed purely from gradient observations derived from auto-differentiation, faster, second order optimization methods first multiply the weights' θ gradient vector ∇ θ by a preconditioning matrix, yielding the weight update θ n+1 ← θ n -ηG -1 n ∇ θ , with η a step size. In the case of second order methods, the matrix G -1 n is chosen to act as a tractable iterative approximation to the inverse Hessian or Empirical Fisher Information Matrix (Amari, 1998) of the neural network model in question. Kronecker-factored approximate curvature or K-FAC (Martens & Grosse, 2015) enforces a Kronecker decomposition of the type G = A ⊗ B, with A and B being smaller, architecture-dependent matrices. Unlike the above methods, K-FAC has been applied as a plug-in in the deep RL literature and shown to promote both anytime convergence properties as well as terminal accuracies (Wu et al., 2017) .

3. OBSERVATIONS AND METHODS:



3.1. EXPLORING TRAINED AGENTS:

Stability of trained dense layers eigenvalues. In order to assess experimentally if tensor factorization can make sense in RL, we investigate the eigenvalues of the dense layers of a deep RL agent. Unlike the traditional supervised learning setting, the input data distribution to RL function approximators shifts as the agent explores its environment; as such, concentration properties of the eigenvalues of the linear layers cannot be guaranteed all the way throughout training. Since conditioning techniques such as batch normalization (Ioffe & Szegedy, 2015) are rarely used in deep RL, this is all the more important. Our experiments (see figure 1 ) show that the distribution of eigenvalues does not seem to widen significantly, at least during the initial phases of training we care about. Furthermore and interestingly, it does not seem to deviate significantly from the one observed at initialization. All together, this suggests there might be some merit in learning low-rank policies.

3.2. ARCHITECTURAL MODIFICATIONS:

Baseline. We then proceed to build upon the well-known Rainbow algorithm (Hessel et al., 2017) . Rainbow uses a fairly standard shallow convolutional architecture like the seminal DQN paper of Mnih et al. (2013) , and is an oft-cited baseline on the Atari suite. In spite of its performance, Rainbow often requires dozens of millions of a single game's frames in order to perform well. Very recently, a 'data-efficient' efficient version of Rainbow has been proposed by van Hasselt et al. (2019) , with a view to match or beat the latest state-of-the-art results achieved by model-based RL methods. This is achieved with no major change in network architecture, but via a selection of mildly handtuned hyperparameters favouring efficiency against wall-clock running time (see appendix). We do take this as a baseline. Changes. We modify the architecture of the neural network function approximators used, in accordance with the principles described above, combining them to reflect inductive biases promoting fewer learnable parameters:

• We replace the fully-connected, linear layers used in Rainbow and data-efficient Rainbow with tensor regression layers (Kossaifi et al., 2017b) in order to learn low-rank policies (ranks in appendix).

• We use either the K-FAC second order stochastic optimizer, or the standard ADAM optimizer (Kingma & Ba, 2014) . Optimization with K-FAC yields better results ceteris paribus and therefore works to counter performance loss due to using fewer weights.

• We combine the two methods with various rank (and therefore weight compression) ratios, targetting sensible compression ratios guided by deep learning intuition; and evaluate those on the same subset of Atari games as both van Hasselt et al. (2019) ; Kaiser et al. (2019) .

• When possible, we replace the first convolutional layer in the approximating neural network with a scattering layer for further gains in terms of learnable weights. In that way, we investigate the impact of not actually learning one of the convolutional layer weights. Deep RL, paradoxically, tends to use shallow network architectures with two or three convolutional layers only, which makes it very fit for scattering methods. But since learned convolutional features discriminate well between high and low rewards, the promise of fully unsupervised scattering layers seems remote, without resorting to further ad-hoc methods like dictionary learning. Therefore we limit ourselves to one-step (one single layer) scatterings. This is illustrated in figure 2 . We then proceed to evaluate the merit of these changes.

4. EXPERIMENTAL RESULTS:



4.1. PRIORITIZED TENSORIZED DQN:

Proof of concept. We begin with showing proof of concept on the simple Pong Atari game. Our experimental setup consists in our own implementation of prioritized double DQN as a baseline (Schaul et al., 2015; van Hasselt et al., 2015) . We replaced the densely connected layer of the original DQN architecture with a tensor regression layer implementing Tucker decomposition for different Tucker ranks, yielding different network compression factors.

Qualitative behaviour. First results, both in terms of learning performance and compression factor, can be seen in figure 3 . Our two main findings are that first, the final performance of the agent remains unaffected by the tensor factorization, even with high compression rates -with respect to all network weights -of five times. In line with intuition, larger compression rates do however cause more delays in learning. Second, tensor factorization negatively affects stability during training -in tough compression regimes, the plateauing phases of learning curves feature occasional noisy drawdowns, illustrating the increased difficulty of learning, as seen in figure 4 . Interestingly, approximation errors incurred by tensor regression noise do sometimes have poor consequences illustrated by those drawdowns, but overall seem to behave as additional exploration noise. Figure 4 : Focus on a typical single run of the tensorized DQN learning (score vs. number of thousand episodes). The overall shape of the typical learning curve is preserved, but drawdowns in the plateauing phase do appear.

4.2. DATA-EFFICIENT RAINBOW ON ATARI:

Evaluation protocol. For all our Atari experiments, we used OpenAI Gym (Brockman et al., 2016) , and a combination of PyTorch (Paszke et al., 2017) , TensorLy (Kossaifi et al., 2016) and Kymatio (Andreux et al., 2018) for auto-differentiation. We evaluated our agents in the low-data regime of 100,000 steps, on half the games, with 3 different random seeds for reproducibility (Henderson et al., 2017) , taking the data-efficient Rainbow agent (van Hasselt et al., 2019) as our baseline. Our specific hyperparameters are described in appendix. We report our results in tables 1 and 2. (Kaiser et al., 2019) and data-efficient Rainbow (van Hasselt et al., 2019) , versus our agents, on 26 Atari games. 'Denoised' is the NoisyNet ablation of Rainbow; 'TRL' shows the performance of the data-efficient Rainbow with tensor regression layers substituted for linear ones.

Game:

Table 1 shows proof of concept of the online learning of low-rank policies, with a loss of final performance varying in proportion to the compression in the low-rank linear layers, very much like in the deep learning literature (Kossaifi et al., 2017a; b) . The number of coefficients in the original data-efficient Rainbow is of the order of magnitude of 1M and varies depending on the environment and its action-space size. The corresponding tensor regression layer ranks are in appendix, and chosen to target 400k, 200k and 100k coefficients respectively. While individual game results tend to decrease monotonously with increasing compression, we observe that they are noisy due to the nature of exploration in RL, and average scores reported correspond to the intuition that performance seems to decrease fast after a certain overparameterization threshold is crossed. To take this noisy character into account, we take care to be conservative and report the average of the final three episodes of the learned policy after 80k, 90k and 100k steps, respectively.

Denoised baseline. So as to not muddy the discussion and provide fair baselines, we do report on the NoisyNet (Fortunato et al., 2017) ablation of Rainbow ('Denoised' columns), as the NoisyLinear layer doubles up the number of coefficients required and actually performs worse in our experiments. Its principle is that the variance of exploration noise represents a criticial tradeoff for performance (too little and one stalls, too much and one risks catastrophic updates), so it is sensible to treat it as a parameter to learn. However, in order to decouple both factors of overparametrization and exploration in the discussion of deep RL performance, we simply use a fixed exploration schedule. This denoised exploration baseline is an ablation we can then compare our tensorized methods to.

Second-order optimization. We then proceed to assess the impact of second-order optimization in architecture by substituting ADAM optimization for K-FAC, and introducing scattering, in table 2 . In spite of our conservative reporting, the efficiency boost from using a second order scheme more than makes up for low-rank approximation error (109% performance) with five times less coefficients than van Hasselt et al. ( 2019), and learning with a full order of magnitude less coefficients (98% performance) is made possible by the combination of K-FAC and TRL. The results however do show a sharp drop in average performance when scattering is added. Interestingly enough some games perform relatively very well with that method, simultaneously showing proof of viability and of additional work required. 

5. CONCLUSION:

We have demonstrated that in the low-data regime, it is possible to leverage biologically plausible characterizations of experience data (namely low-rank properties and wavelet scattering separability) to exhibit architectures that learn policies with an order of magnitude less weights than current state-of-the-art baselines, essentially without loss of performance, and in an online fashion. In particular, this provides a compelling alternative to methods like policy distillation (Rusu et al., 2015; Czarnecki et al., 2019) . We do hope that this will lead to even further progress towards sample efficiency and speedy exploration methods. Further work will, first, focus on thorough evaluation and research of scattering architectures in order to achieve further gains, and second investigate additional, orthogonal biologically-friendly research directions such as promoting sparsity via, for instance, L 1 regularization. Finally, we are excited by the potential of tensor factorization to offer shared core tensors for policies in multi-task and meta-learning.

:



Figures and Tables:

Figure 1: Eigenvalue histograms of the value-based linear layer during training. 50 agent runs dataefficient Rainbow (van Hasselt et al., 2019), of 100,000 steps on the Atari game Road Runner.

Figure 2: Our architectural approach consists in replacing hidden layers in deep RL agents with tensor regression (top). Optionally we substitute the topmost convolutional layer with scattering (middle), and combine both methods (bottom).

Figure 3: Prioritized tensorized DQN on Atari Pong. Original learning curve versus several learning curves for five different Tucker ranks factorizations and therefore parameter compression rates (3 different random seeds each, with a 30 episodes moving average for legibility). Best viewed in colour.

Mean episode returns as reported in baselines SimPLe

Table content: <table><tr><td/><td colspan="6">SimPLe Rainbow Denoised TRL 2.5x TRL 5x TRL 10x</td></tr><tr><td>alien</td><td>405</td><td>740</td><td>684</td><td>688</td><td>454</td><td>566</td></tr><tr><td>amidar</td><td>88</td><td>189</td><td>154</td><td>118</td><td>86</td><td>84</td></tr><tr><td>assault</td><td>369</td><td>431</td><td>321</td><td>543</td><td>521</td><td>513</td></tr><tr><td>asterix</td><td>1090</td><td>471</td><td>500</td><td>459</td><td>554</td><td>363</td></tr><tr><td>bank heist</td><td>8</td><td>51</td><td>77</td><td>59</td><td>134</td><td>42</td></tr><tr><td>battle zone</td><td>5184</td><td>10125</td><td>9378</td><td>14466</td><td>13466</td><td>5744</td></tr><tr><td>boxing</td><td>9</td><td>0.2</td><td>1</td><td>-2</td><td>-2</td><td>-5</td></tr><tr><td>breakout</td><td>13</td><td>2</td><td>3</td><td>2</td><td>2</td><td>4</td></tr><tr><td>chopper command</td><td>1247</td><td>862</td><td>1293</td><td>1255</td><td>1243</td><td>1106</td></tr><tr><td>crazy climber</td><td>39828</td><td>16185</td><td>9977</td><td>3928</td><td>4225</td><td>2340</td></tr><tr><td>demon attack</td><td>170</td><td>508</td><td>450</td><td>362</td><td>263</td><td>175</td></tr><tr><td>freeway</td><td>20</td><td>28</td><td>28</td><td>26</td><td>25</td><td>24</td></tr><tr><td>frostbite</td><td>255</td><td>867</td><td>1101</td><td>659</td><td>912</td><td>231</td></tr><tr><td>gopher</td><td>771</td><td>349</td><td>391</td><td>278</td><td>255</td><td>396</td></tr><tr><td>hero</td><td>1295</td><td>6857</td><td>3013</td><td>5351</td><td>3732</td><td>3321</td></tr><tr><td>jamesbond</td><td>125</td><td>302</td><td>295</td><td>215</td><td>213</td><td>218</td></tr><tr><td>kangaroo</td><td>323</td><td>779</td><td>1002</td><td>804</td><td>715</td><td>400</td></tr><tr><td>krull</td><td>4540</td><td>2852</td><td>2656</td><td>2333</td><td>2275</td><td>2308</td></tr><tr><td>kung fu master</td><td>17257</td><td>14346</td><td>4037</td><td>9392</td><td>4764</td><td>4031</td></tr><tr><td>ms pacman</td><td>763</td><td>1204</td><td>1053</td><td>818</td><td>838</td><td>517</td></tr><tr><td>pong</td><td>5</td><td>-19</td><td>-20</td><td>-20</td><td>-19</td><td>-21</td></tr><tr><td>private eye</td><td>58</td><td>98</td><td>100</td><td>51</td><td>100</td><td>1128</td></tr><tr><td>qbert</td><td>560</td><td>1153</td><td>672</td><td>697</td><td>581</td><td>733</td></tr><tr><td>road runner</td><td>5169.4</td><td>9600</td><td>5426</td><td>6965</td><td>3914</td><td>1319</td></tr><tr><td>seaquest</td><td>371</td><td>354</td><td>387</td><td>345</td><td>350</td><td>287</td></tr><tr><td>up n down</td><td>2153</td><td>2877</td><td>5123</td><td>2197</td><td>2302</td><td>2179</td></tr><tr><td>Average (vs. Rainbow)</td><td/><td>100%</td><td>118%</td><td>96%</td><td>90%</td><td>71%</td></tr></table>

Mean episode returns of our low-rank agents with second-order optimization and scattering. The Scattering column also includes KFAC optimization and TRL 5x, resulting in around 10x total weights efficiency gains.

Table content: <table><tr><td>Game</td><td colspan="4">KFAC+Denoised KFAC+TRL5x KFAC+TRL10x Scattering</td></tr><tr><td>alien</td><td>996</td><td>734</td><td>643</td><td>441</td></tr><tr><td>amidar</td><td>163</td><td>101</td><td>98</td><td>84</td></tr><tr><td>assault</td><td>501</td><td>491</td><td>496</td><td>434</td></tr><tr><td>asterix</td><td>537</td><td>549</td><td>526</td><td>502</td></tr><tr><td>bank heist</td><td>100</td><td>73</td><td>57</td><td>29</td></tr><tr><td>battle zone</td><td>8622</td><td>15178</td><td>6156</td><td>4311</td></tr><tr><td>boxing</td><td>0</td><td>-4</td><td>-1</td><td>-9</td></tr><tr><td>breakout</td><td>3</td><td>3</td><td>2</td><td>2</td></tr><tr><td>chopper command</td><td>692</td><td>611</td><td>1302</td><td>441</td></tr><tr><td>crazy climber</td><td>14242</td><td>12377</td><td>3546</td><td>740</td></tr><tr><td>demon attack</td><td>582</td><td>434</td><td>318</td><td>692</td></tr><tr><td>freeway</td><td>26</td><td>26</td><td>24</td><td>19</td></tr><tr><td>frostbite</td><td>1760</td><td>718</td><td>1483</td><td>654</td></tr><tr><td>gopher</td><td>363</td><td>341</td><td>265</td><td>172</td></tr><tr><td>hero</td><td>4188</td><td>6284</td><td>4206</td><td>4127</td></tr><tr><td>jamesbond</td><td>263</td><td>327</td><td>217</td><td>48</td></tr><tr><td>kangaroo</td><td>2085</td><td>613</td><td>588</td><td>391</td></tr><tr><td>krull</td><td>2855</td><td>3441</td><td>3392</td><td>772</td></tr><tr><td>kung fu master</td><td>8481</td><td>10738</td><td>7357</td><td>233</td></tr><tr><td>ms pacman</td><td>1137</td><td>920</td><td>867</td><td>613</td></tr><tr><td>pong</td><td>-19.3</td><td>-19</td><td>-19</td><td>-20</td></tr><tr><td>private eye</td><td>56</td><td>100</td><td>100</td><td>0</td></tr><tr><td>qbert</td><td>731</td><td>520</td><td>538</td><td>475</td></tr><tr><td>road runner</td><td>4516</td><td>8493</td><td>7224</td><td>1278</td></tr><tr><td>seaquest</td><td>349</td><td>317</td><td>520</td><td>213</td></tr><tr><td>up n down</td><td>2557</td><td>2291</td><td>2108</td><td>993</td></tr><tr><td>Average (vs. Rainbow)</td><td>114%</td><td>109%</td><td>98%</td><td>55%</td></tr></table>

Peer review (iclr2020):
This paper investigates a list of methods to reduce the number of weights for deep RL architecture under the low-data regime. These methods include tensor regression, wavelet scattering, as well as second-order optimization (K-FAC). The experiments on the Atari games shows that by using tensor regression to replace the dense layer of the neural nets and using K-FAC for the optimization, one can reduce around 10 times of parameters without losing too much of performance. However, I have some concerns on the novelty of this work and therefore I’m giving this paper a weak reject. Here are the reasons: To begin with, leveraging tensor structure of the neural nets to reduce number of parameters while maintaining similar level or getting even better results have been done before, for example: Tensorizing Neural Networks (Novikov et al, 2015), Learning compact recurrent neural networks with block-term tensor decomposition (Ye et al., 2018) etc. Although the use of tensor regression might be new, the core idea is still to leverage the low rank property of the tensor and obtain a compression of the weight tensors. Moreover, why use Tucker decomposition specifically for the tensor regression? It has been proposed that using tensor train (TT) decomposition can also get very good results (see Garipov et. al. Ultimate tensorization: compressing convolutional and FC layers alike). Is it possible to investigate the use of TT decomposition for the dense layer of the deep RL architecture? Therefore the novelty for this aspect seems a bit weak for me. The second method the authors have attempted is to swap the convolution layer of the deep RL architecture with wavelet scattering. For one particular game (demon_attack), this approach seems to outperform every other methods by a large margin. However the experiment shows that for the rest of the Atari games, there is a huge drop (45%) of performance. Therefore the significance of this approach is rather thin for me. Maybe some further investigation of the game demon_attack is needed to understand why using scattering in this game in particular gives such a huge performance boost. Thirdly, as an approximation of the second order optimization, K-FAC does not really concern with the main theme of the paper, which is an investigation of potential weights reduction methods. It is great that the authors applied this techniques and seems to have great results. However, as the authors pointed out, K-FAC has been wildly applied in the deep RL literature, and the authors did not propose new extension for the K-FAC method, therefore the contribution of this matter is also quite thin. Last but not least, the writing of the paper is a bit clumsy, and I was having a hard time to figure out what exactly is the proposed method. I think this paper might need some rework on the writing to describe the idea of the authors in a more clear way for the publication. Due to these reasons, I’m giving this paper a weak reject. Some writing comments and potential writing errors (did not affect the decision): Page 3, first line of “Tensor regression layer”, the shape of the tensor X seems to be a typo. Also here, the definition of <X, Y>_N in the paper is to sum over the dimension of I_1…I_N, then the shape of <X, Y>_N should be K_1*…*K_x*L_1*…*L_y, without the I_N in the middle. Also in this section, the authors mentioned Tucker decomposition for the tensor regression. However the phrasing of this sentence needs a bit rework. The usage of “For instance here”, gives the readers a feeling that Tucker is just one possible way of doing this decomposition, but not necessarily the actual decomposition for the reported experiments. In 2.3, there is a lack of definition for Lambda_1 and Lambda_2. In addition, it would be better for the general readers to add a few definitions for the terminologies in this section. For example, “circular convolution”, “wavelet filter banks” etc. I guess people with corresponding background will understand it with no problem, however I do find myself a bit lost in this section with these terminologies. 2.4 line 6, “with A and. B smaller, architecture-dependent matrices”. I think it should be “with A and B being…“ 3.1, line 5, “This is all the more pressing that….”, I did not understand this sentence. In page 6, line 3, there is a lack of definition for “compression rate”. Is it the compression rate w.r.t only the last dense layer, or w.r.t the whole network? Figure 4 is lacking y-axis and x-axis labels. 4.2, last bullet point, “However, one must not forget that the conv layers one learns must be somehow be well adapted…”, I get what you are saying, but the sentence is a bit clumsy. Table 1 and 2, the row name “Average” is lacking definition. Overall it is a good attempt to reduce the number of weights in the deep RL architecture, but I do think the novelty of this work is a bit thin and the three contributions were not tied together with the main theme of the paper. Therefore, I’m giving this work a weak reject.