Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: The paper aims at parsimonious reinforcement learning by employing 3
different techniques: using tensor regression layers (Kossaifi et al.,
2017b), wavelet scattering (Mallat 2011) and using K-FAC (Kingma & Ba,
2014) as the optimization method. Learning models with fewer
weights is important not only in reinforcement learning but also
in all other machine learning areas. With the combination of tensor
regression layers and K-FAC, the proposed methods give comparable
performance on several Atari games against 2 other methods, SimpPLe
and data-efficient Rainbow, while using 2 to 10 times fewer
coefficients than data-efficient Rainbow. The use of wavelet
scattering provides improvement on 1 out of 26 Atari games. The paper
also points out an interesting concentration of eigenvalues of dense
layer of a deep RL agent which provides motivation for low-rank
presentations.

While the savings in terms of coefficients is positive, the obtained
results are of little surprise. Tensor regression layers and K-FAC are
used as is without any modification while space savings and efficiency
have been reported in corresponding references. The performances of
wavelet scattering for the reported tasks are weak (better in only one
game) and the space saving is not clear. The proposed improvements
seem to be tailored to tasks with image inputs and hence reported
results are only on Atari games (possibly with sparse and low-rank
images). It is not clear if the proposed techniques can be applied to
a wider set of reinforcement learning tasks
(e.g. https://gym.openai.com/envs/#mujoco).


It would be interesting to see if we can apply the proposed methods to
other more diverse RL tasks. The performance of wavelet scattering does indeed need
more investigation and improvement. It would also be interesting to
compare the distributions of the eigenvalues of the tensor layers
versus the dense layers in deep RL which may provide insights on the
achieved savings and the compression trade-off.


I have read  the authors' rebuttals. The reviews point to a number of directions 
where the contributions could be made more significant.


review_title: Official Blind Review #2

final_score: rating: 3: Weak Reject
review_content: This paper investigates a list of methods to reduce the number of weights for deep RL architecture under the low-data regime. These methods include tensor regression, wavelet scattering, as well as second-order optimization (K-FAC). The experiments on the Atari games shows that by using tensor regression to replace the dense layer of the neural nets and using K-FAC for the optimization, one can reduce around 10 times of parameters without losing too much of performance. 

However, I have some concerns on the novelty of this work and therefore I’m giving this paper a weak reject. Here are the reasons:

To begin with, leveraging tensor structure of the neural nets to reduce number of parameters while maintaining similar level or getting even better results have been done before, for example: Tensorizing Neural Networks (Novikov et al, 2015), Learning compact recurrent neural networks with block-term tensor decomposition (Ye et al., 2018) etc. Although the use of tensor regression might be new, the core idea is still to leverage the low rank property of the tensor and obtain a compression of the weight tensors. Moreover, why use Tucker decomposition specifically for the tensor regression? It has been proposed that using tensor train (TT) decomposition can also get very good results (see Garipov et. al. Ultimate tensorization: compressing convolutional and FC layers alike). Is it possible to investigate the use of TT decomposition for the dense layer of the deep RL architecture? Therefore the novelty for this aspect seems a bit weak for me.

The second method the authors have attempted is to swap the convolution layer of the deep RL architecture with wavelet scattering. For one particular game (demon_attack), this approach seems to outperform every other methods by a large margin. However the experiment shows that for the rest of the Atari games, there is a huge drop (45%) of performance. Therefore the significance of this approach is rather thin for me. Maybe some further investigation of the game demon_attack is needed to understand why using scattering in this game in particular gives such a huge performance boost. 

Thirdly, as an approximation of the second order optimization, K-FAC does not really concern with the main theme of the paper, which is an investigation of potential weights reduction methods. It is great that the authors applied this techniques and seems to have great results. However, as the authors pointed out, K-FAC has been wildly applied in the deep RL literature, and the authors did not propose new extension for the K-FAC method, therefore the contribution of this matter is also quite thin. 

Last but not least, the writing of the paper is a bit clumsy, and I was having a hard time to figure out what exactly is the proposed method. I think this paper might need some rework on the writing to describe the idea of the authors in a more clear way for the publication. Due to these reasons, I’m giving this paper a weak reject. 

Some writing comments and potential writing errors (did not affect the decision):
Page 3, first line of “Tensor regression layer”, the shape of the tensor X seems to be a typo. 
Also here, the definition of <X, Y>_N in the paper is to sum over the dimension of I_1…I_N, then the shape of <X, Y>_N should be K_1*…*K_x*L_1*…*L_y, without the I_N in the middle. 
Also in this section, the authors mentioned Tucker decomposition for the tensor regression. However the phrasing of this sentence needs a bit rework. The usage of “For instance here”, gives the readers a feeling that Tucker is just one possible way of doing this decomposition, but not necessarily the actual decomposition for the reported experiments. 
In 2.3, there is a lack of definition for  \Lambda_1 and \Lambda_2. In addition, it would be better for the general readers to add a few definitions for the terminologies in this section. For example, “circular convolution”, “wavelet filter banks” etc. I guess people with corresponding background will understand it with no problem, however I do find myself a bit lost in this section with these terminologies. 
2.4 line 6, “with A and. B smaller, architecture-dependent matrices”. I think it should be “with A and B being…“
3.1, line 5, “This is all the more pressing that….”, I did not understand this sentence. 
In page 6, line 3, there is a lack of definition for “compression rate”. Is it the compression rate w.r.t only the last dense layer, or w.r.t the whole network?
Figure 4 is lacking y-axis and x-axis labels. 
4.2, last bullet point, “However, one must not forget that the conv layers one learns must be somehow be well adapted…”, I get what you are saying, but the sentence is a bit clumsy. 
Table 1 and 2, the row name “Average” is lacking definition. 

 Overall it is a good attempt to reduce the number of weights in the deep RL architecture, but I do think the novelty of this work is a bit thin and the three contributions were not tied together with the main theme of the paper. Therefore, I’m giving this work a weak reject. 

review_title: Official Blind Review #1

final_score: rating: 1: Reject
review_content: This paper suggests three different disconnected ideas for improving the number of parameters of deep vision models for playing ATARI and to improve the training speed.
- Tensor-regression layer to replace fully connected layers
- Wavelet-scattering layer to replace the first convolutional layer
- Second order optimization (K-FAC)
All the ideas mentioned in this paper are existing ones (although properly attributed), so the novelty of this work is relatively low. 
The paper mentions that this particular combination is "novel", but it is not clear is there is any significant synergy between these methods and why it should be considered interesting in this particular setup.
Also the paper conflates sample-efficiency with parameter-efficiency. However, there is no indication that any of these methods address sample-efficiency which would be an interesting and useful contribution.
Also the experiments are neither very conclusive nor are they easy to interpret. For example in the pong case, there is no discernable effect of the compression ratio as the highest and lowest compression give the best (and comparable) results. Also the results come without confidence intervals.

So, in general, I would consider this paper to be an uninspired combination of pre-existing ideas with weak and inconclusive experimental results: a clear reject.
review_title: Official Blind Review #3

Meta-review: In this paper dense layers in deep neural networks representing policies are replaced by tensor regression layers, also by a scattering layer, and second-order optimization is considered. The paper does not have a single consistent message, and combines different techniques for unclear reason. Important related work is not cited. The presentation was found unclear by the reviewers. 
Decision: Reject