Venue: ICLR 2018 conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: The authors propose a kind of framework for learning to solve elemental tasks and then learning task switching in a multitask scenario. The individual tasks are inspired by a number of psychological tasks. Specifically, the authors use a pretrained convnet as raw statespace encoding together with previous actions and learn through stochastic optimization to predict future rewards for different actions. These constitute encapsulated modules for individual tasks. The authors test a number of different ways to construct the state representations as inputs to these module and report results from extensive simulations evaluating them. The policy is obtained through a heuristic, selecting actions with highest reward prediction variance across multiple steps of lookahead. Finally, two variants of networks are presented and evaluated, which have the purpose of selecting the appropriate module when a signal is provided to the system that a new task is starting.

I find it particularly difficult to evaluate this manuscript. The presented simulation results are based on the described system, which is very complex and contains several, non-standard components and heuristic algorithms. 

It would be good to motivate the action selection a bit further. E.g., the authors state that actions are sampled proportionally to the reward predictions and assure properties that are not necessarily intuitive, e.g. that a few reward in the future can should be equated to action values. It is also not clear under which conditions the proposed sampling of actions and the voting results in reward maximization. No statements are made on this other that empirically this worked best.  

Is the integral in eq. 1 appropriate or should it be a finite sum?

It seems, that the narrowing of the spatial distribution relative to an \epsilon -greedy policy would highly depend on the actual reward landscape, no? Is the maximum variance as well suited for exploration as for exploitation and reward maximization?

What I find a bit worrisome is the ease with which the manuscript switches between  “inspiration” from psychology and neuroscience to plausibility of proposing algorithms to reinterpreting aimpoints as “salience” and feature extraction as “physical structure”. This necessarily introduces a number of 

Overall, I am not sure what I have learned with this paper. Is this about learning psychological tasks? New exploration policies? Arbitration in mixtures of experts? Or is the goal to engineer a network that can solve tasks that cannot be solved otherwise? I am a bit lost.


Minor points:
“can from”
review_title: Complex network using heuristic structure, state representations, and action selection for solving tasks inspired by psychology

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: The paper comprises several ideas to study the continual learning problem. First, they show an ad-hoc designed environment, namely the Touchstream environment, in which both inputs and actions are represented in a huge space: as it happens with humans – for example when they are using a touch screen – the resolution of the input space, i.e. the images, is at least big as the resolution of the action space, i.e. where you click on the screen. This environment introduces the interesting problem of a direct mapping between input and actions. Second, they introduce an algorithm to solve this mapping problem in the Touchstream space. Specifically, the ReMaP algorithm learns to solve typical neuroscience tasks, by optimizing a computational module that facilitates the mapping in this space. The Early Bottleneck Multiplicative Symmetric (EMS) module extends the types of computation you might need to solve the tasks in the Touchstream space. Third, the authors introduce another module to learn how to switch from task to task in a dynamical way.
The main concern with this paper is about its length. While the conference does not provide any limits in terms of number of pages, the 13 pages for the main text plus other 8 for the supplementary material is probably too much. I am wondering if the paper just contains too much information for a single conference publication. 
As a consequence, either the learning of the single tasks and the task switching experiments could have been addressed with further details. In case of single task learning, the tasks are relatively simple where k_beta = 1 (memory) and k_f = 2 (prediction) are sufficient to solve the tasks. It would have been interesting to evaluate the algorithm on more complex tasks, and how these parameters affect the learning. In case of the task switching a more interesting question would be how the network learns a sequence of tasks (more than one switch). 
Overall, the work is interesting, well described and the results are consistent. 
Some questions:
- the paper starts with clear inspiration from Neuroscience, but nothing has been said on the biological plausibility of the ReMap, EMS and the recurrent neural voting;
- animals usually learn in continuous-time, thus such tasks usually involve a time component, while this work is designed in a time step framework; could the author argument on that? (starting from Doya 2000)
- specifically, the MTS and the Localization tasks involve working memory, thus a comparison with other working memory with reinforcement learning would make more sense that different degree of ablated modules. (for example Bakker 2002)

Minor:
- Fig 6 does not have letters
- TouchStream is sometimes written as Touchstream

Ref.
Doya, K. (2000). Reinforcement learning in continuous time and space. Neural Computation, 12(1), 219–245.
Bakker, B. (2002). Reinforcement Learning with Long Short-Term Memory. In T. G. Dietterich, S. Becker, & Z. Ghahramani (Eds.), (pp. 1475–1482). Presented at the Advances in Neural Information Processing Systems 14, MIT Press.




review_title: A interesting scenario with several implications

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: Reading this paper feels like reading at least two closely-related papers compressed into one, with overflow into the appendix (e.g. one about the EMS module, one about the the recurrent voting, etc).

There were so many aspects/components, that I am not entirely confident I fully understood how they all work together, and in fact I am pretty confident there was at least some part of this that I definitely did not understand. Reading it 5-20 more times would most likely help.

For example, consider the opening example of Section 3. In principle, this kind of example is great, and more of these would be very useful in this paper. This particular one raises a few questions:
-Eq 5 makes it so that $(W \Psi)$ and $(a_x)$ need to be positive or negative together.  Why use ReLu's here at all? Why not just $sign( (W \Psi) a_x) $? Multiplying them will do the same thing, and is much simpler. I am probably missing something here, would like to know what it is... (Or, if the point of the artificial complexity is to give an example of the 3 basic principles, then perhaps point this out, or point out why the simpler version I just suggested would not scale up, etc)
-what exactly, in this example, does $\Psi$ correspond to? In prev discussion, $\Psi$ is always written with subscripts to denote state history (I believe), so this is an opportunity to explain what is different here. 
-Nitpick: why is a vector written as $W$? (or rather, what is the point of bold vs non-bold here?)
-a non-bold version of $Psi$, a few lines below, seems to correspond to the 4096 features of VGG's FC6, so I am still not sure what the bold version represents

-The defs/eqns at the beginning of section 3.1 (Sc, CReLu, etc) were slightly hard to follow and I wonder whether there were any typos, e.g. was CReS meant to refer directly to Sc, but used the notation ${ReLu}^2$ instead? 

Each of these on its own would be easier to overlook, but there is a compounding effect here for me, as a reader, such that by further on in the paper, I am rather confused.

I also wonder whether any of the elements described, have more "standard" interpretations/notations. For example, my slight confusion propagated further: after above point, I then did not have a clear intuition about $l_i$ in the EMS module. I get that symmetry has been built in, e.g. by the definitions of CReS and CReLu, etc, but I still don't see how it all works together, e.g. are late bottleneck architectures *exactly* the same as MLPs, but where inputs have simply been symmetrized, squared, etc? Nor do I have intuition about multiplicative symmetric interactions between visual features and actions, although I do get the sense that if I were to spend several hours implementing/writing out toy examples, it would clarify it significantly (in fact, I wouldn't be too surprised if it turns out to be fairly straightforward, as in my above comment indicating a seeming equivalence to simply multiplying two terms and taking the resulting sign). If the paper didn't need to be quite as dense, then I would suggest providing more elucidation for the reader, either with intuitions or examples or clearer relationships to more familiar formulations.

Later, I did find that some of the info I *needed* in order to understand the results (e.g. exactly what is meant by a "symmetry ablation", how was that implemented?) was in fact in the appendices (of which there are over 8 pages).

I do wonder how sensitive the performance of the overall system is to some of the details, like, e.g. the low-temp Boltzmann sampling rather than identity function, as described at the end of S2.

My confidence in this review is somewhere between 2 and 3.

The problem is an interesting one, the overall approach makes sense, it is clear the authors have done a very substantial  amount of work, and very diligently so (well-done!), some of the ideas are interesting and seem creative, but I am not sure I understand the glue of the details, and that might be very important here in order to assess it effectively.
review_title: An interesting, albeit very dense, paper describing continuous learning over a set of different tasks with a flexible modular architecture and a large state and action space.

Meta-review: Important problem (modular continual RL) and novel contributions. The initial submission was judged to be a little dense and hard to read, but the authors have been responsive in responding and updating the paper. I support accepting this paper. 
Decision: Accept (Poster)