Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 1: Reject
review_content: This paper presents a computational model of motivation for Q learning and relates it to biological models of motivation. Motivation is presented to the agent as a component of its inputs, and is encoded in a vectorised reward function where each component of the reward is weighted. This approach is explored in three domains: a modified four-room domain where each room represents a different reward in the reward vector, a route planning problem, and a pavlovian conditioning example where neuronal activations are compared to mice undergoing a similar conditioning.

Review Summary:
I am uncertain of the neuroscientific contributions of this paper. From a machine learning perspective, this paper has insufficient details to assess both the experimental contributions and proposed formulation of motivation. It is unclear from the discussion of biological forms of motivation, and from the experimental elaboration of these ideas, that the proposed model of motivation is a novel contribution. For these reasons, I suggest a reject.

The Four Rooms Experiment:

In the four-rooms problem, the agent is provided with a one-hot encoding representing which cell it the agent is located in within the grid-world. The reward given to the agent is a combination of the reward signal from the environment (a one-hot vector where the activation is dependent on the room occupied by the agent) and the motivation vector, which is a weighting of the rooms. One agent is given access to the weighting vector mu in its state vector: the motivation is concatenated to the position, encoding the weighting of the rooms at any given time-step. The non-motivated agent does not have access to mu in its state, although its reward is weighted as the motivated agent’s is. The issue with this example is that the non-motivated agent does not have access to the information required to learn a value-function suitable to solve this problem. By not giving the motivation vector to non-motivated agent, the problem has become a partially observable problem, and the comparison is now between a partially observable and fully observable setting, rather than a commentary on the difference between learning with and without motivation.

In places, the claims made go beyond the results presented. How do we know that the non-motivated network is engaging in a "non-motivated delay binge"? We certainly can see that the agent acquires an average reward of 1, but it is not evident from this detail alone that the agent is engaging in the behaviour that the paper claims. 

Moreover, the network was trained 41 times for different values of the motivation parameter theta. Counting out the points in figure 2, it would suggest that the sweep was over 41 values of theta, which leaves me wondering if the results represent a single independent trial, or whether the results are averaged over multiple trials. Looking at the top-right hand corner I see a single yellow dot (non-motivated agent) presented in line with blue (motivated agent) suggesting that the point is possibly an outlier. Given this outlier, I’m led believe that the graph represents a single independent trial. A single trial is insufficient to draw conclusions about the behaviour of an agent. 

The Path Routing Experiment:

In the second experiment, where a population of agents is presented in fig 5, it is claimed that on 82% of the trials, the agent was able to find the shortest path. Looking at the figure itself, at the final depicted iteration, all of the points are presented in a different colour and labelled “shortest path”. The graph suggests that 100% of the agents found the shortest path. The claim is made that for the remaining 18% of the agents, the agents found close to the shortest path—a point not evident in the figures presented.


Pavlovian Conditioning Experiment:

In the third experiment, shouldn’t Q(s) be V(s)? In this setting, the agent is not learning the value of a state action pair, but rather the value of a state. Moreover, the value is described as Q(t), where t is the time-step in the trial; however, elsewhere in the text it is mentioned that the state is not simply t, but contains also the motivation value mu.  

The third experiment does not have enough detail to interpret the results. It is unclear how many trials there were for both of the prediction settings. It is unclear whether the problem described is a continuing problem or a terminating prediction problem—i.e., whether after the conditioned stimulus and unconditioned stimulus are presented to the agent, does the time-step (and thus the state) reset to 0, or does time continue incrementing? If it is a terminating prediction problem, it is unclear whether the conditioned stimulus and unconditioned stimulus were delivered on the same time-steps for each independent trial. If I am interpreting the state-construction correctly, the state is incrementing by one on each time-step; this problem is effectively a Markov Reward Process where the agent transitions from one state to the next until time stops with no ability to transition to previous states.

In both the terminating and continuing cases, the choice of inputs is unusual. What was the motivation for using the time-step as part of the state construction?

How is the conditioned stimulus formulated in this setting? It is mentioned that it is a function of time, but there are no additional details.

From reading the text, it is unclear whether fig 7b/c presents activations over multiple independent trials or a single trial.

General Thoughts on Framing:

This paper introduces non-standard terms without defining them first. For example, TD error is introduced as Reward Prediction Error, or RPE: a term that is not typically used in the Reinforcement Learning literature. To my understanding, there is a hypothesis about RPE in the brain in the cognitive science community; however, the connection between this idea in the cognitive science literature and its relation to RL methods is not immediately clear.

Temporal Difference learning is incorrectly referred to as "Time Difference" learning (pg 2). 

Notes on technical details:

- The discounting function gamma should be 0<= gamma <=1, rather than just <=1.

- discounting not only prevents the sum of future rewards from diverging, but also plays an important role in determining the behaviour of an agent---i.e., the preference for short-term versus long-term rewards.

- pg 2 "the motivation is a slowly changing variable, that is not affected substantially by an average action" -- it is not clear from the context what an average action is. 

- Why is the reward r(s|a), as opposed to r(s,a)?

Notes on paper structure:

- There are some odd choices in the structure of this paper. For instance, the second section---before the mathematical framing of the paper has been presented---is the results section. 

- In some sentences, citations are added where no claim is being made; it is not clear what the relevance of the citation is, or what the citation is supporting. E.g., “We chose to use a recurrent neural network (RNN) as a basis for our model” following with a citation for Sutton & Barto, 1987.

- In some sentences, citations are not added where substantial claims are being made. E.g, “The recurrent network structure in this Pavlovian conditioning is compatible with the conventional models of working memory”. This claim is made, but it is never made clear what the conventional computational models of working memory are, or how they fit into the computational approaches proposed.

- Unfortunately, a number of readers in the machine learning community might be unfamiliar with pavlovian conditioning and classical conditioning. Taking the time to unpack these ideas and contextualise them for the audience might help readers understand the paper and its relevance.

- Figure 7B may benefit from displaying not just the predicted values V(s), but a plot of the prediction over time in comparison to the true expected return.

review_title: Official Blind Review #3

final_score: rating: 3: Weak Reject
review_content: The authors investigate mechanisms underlying action selection in artificial agents and mice. To achieve this goal, they use RL to train neural networks to choose actions that maximize their temporally discounted sum of future rewards. Importantly, these rewards depend on a motivation factor that is itself a function of time and action; this motivation factor is the key difference between the authors' approach and "vanilla" RL. In simple tasks, the RL agent learns effective strategies (i.e., migrating between rooms in Fig. 1, and minimizing path lengths for the vehicle routing problem in Fig. 5). 

The authors then apply their model to a task in which the agent is presented with sound cues. Depending on the trial block, the reward for the given cue is either zero, positive, or negative; the authors suggest that these varying reward values correspond to varying motivational states. In this setting, the model learns to have two populations of units; each selective to either positive or negative rewards. Recurrent excitation within populations and mutual inhibition between populations define the learned dynamics.

Finally, the authors train mice on this same task, and record from neurons in area VP. Those neurons show a similar structure to the RNN: subpopulations of neurons respond to either positive or negative rewards.

First, I'd like to thank the authors for the excellent clarity of this paper. It was very clear, and interesting to read.  

I have some suggestions for how to deepen the connection between the model and the experiment, and some concerns about the necessity of the motivation framework to the Pavlovian task:

1) The authors make the prediction that neurons in VP should show (functional) connectivity matching that learned by their model. This could be tested in their data. If that prediction is true, then one should see positive noise correlations for neuron pairs of the same preference (i.e., within the same pool, defined by spiking more for positive, or for negative rewards), and negative noise correlations for pairs of neurons with different preferences (i.e., one neuron in each pool).

2) A recent preprint by Sederberg and Nemenman (doi: https://doi.org/10.1101/779223) argued against over-interpreting the stimulus selectivity of neurons in recurrent circuits. They showed that, even in randomly connected (untrained) networks: a) neurons show either positive or negative selectivity; and b) neuron pairs with selectivity for the same stimulus (or task) feature tend to excite each other, and neuron pairs with opposite selectivity tend to inhibit each other.  Given that finding, I wonder how compelling is the match between the mouse data and the RL agent (Figs. 6 and 7): could randomly-connected untrained networks show similar phenomena as in the mouse (Fig. 6)?

I'm not asking if the untrained network can duplicate all the details of the trained one in Fig. 7. Just whether the mouse data could be recapitulated by a simpler (no training) model.

3) For the Pavlovian conditioning in the RL agent, I'm not sure I'd describe this as changing motivation. It seems instead that the (external) reward contingency really changes between states. So the fact that the same network can make predictions in both cases seems more like metalearning than motivation-based action selection. For this reason, it's hard for me to connect the two halves of the paper: the first half has nice ideas on motivation-based action selection, while the second one has no apparent action selection, and hence no mechanism for the agent's motivation to matter.

review_title: Official Blind Review #2

final_score: rating: 3: Weak Reject
review_content: This paper builds a model of motivation-dependent learning.  A motivation channel is provided as an additional input to and RL-based learning system (essentially concatenated to state information), similar to goal-conditioned approaches (as the authors mention).  The motivational variables evolve according to their own rules, and are designed/interpreted as biological motivations such as water, food, sleep and work.  While the narrative is interesting, I lean towards reject as I believe it failed to deliver on what it promised.

In the first experiment, the satisfaction of these motivations are mapped onto a 4-room setting, where being in each room satisfies a motivation.  The choice to map the four rooms to biological drives is cute, but possibly confusing/misleading since this navigation problem really has nothing to do with these biological drives. A claim is that by providing the motivation as input to the policy, it is more robustly (across seeds) able to learn the "migration" (i.e. cycling) behavior among the rooms.  In a second example, a similar problem is solved involving navigation on a graph.

The final, most substantial example, is a policy trained to solve a simple, abstract version of a behavioral task. In this setting, a motivation channel was again used.  However, the motivation channel value is now fixed to one of two discrete values, essentially meaning it is simply a task-label variable, a paradigm that has already been applied in the context of simple models of neuroscience tasks, e.g. see Song et al. 2017 "Reward-based training of recurrent neural networks for cognitive and value-based tasks".  

There is a bit of a mixed framing overall as to whether it is being claimed that the "motivation" being passed as an input is a fundamental contribution to AI/RL (I think it is not), versus the computational modeling of biological motivation.  I think the people qualified to judge whether the computational model is a worthwhile model of motivation specifically are probably a narrower set of computational neuroscientists.  I do think there is value in the kind of computational modeling performed, involving establishing a relationship between training a neural network to solve a behavioral task and comparing this with real neural data.  This paradigm already becoming increasingly popular within computational neuroscience.  However, while I find the results slightly interesting, but not very significant, as someone interested in the biology of motivation, I question whether the nature of these contributions would be of broad interest at this venue.  

More fundamentally, I don't believe there is a meaningful ML/AI/RL contribution, and I have some issues with the presentation of the first two examples.  While I do like the narrative inspiring these problems, I find the implementations of the problems too simplified to really be meaningfully related to their inspiration (in terms of motivated behaviors).  Rather than really model motivation as part of the policy architecture, the authors have proposed a solution to modeling motivation that makes motivation a feature of the environment.  Essentially, the reward provided by the environment depends on an extra latent variable and by hiding this (in the cases where the policy does not see motivation inputs), it is quite likely that it becomes too difficult for the value function to predict what is happening (the environment has become partially observed).  This seems less a setting where motivation channels solve a problem, and more just an example of an environment that has more complex rules for generating rewards being more challenging to learn about, especially if latent variables are not available to the value function.  Critically, it has not been shown that motivational systems are useful for artificial agents, rather the tasks themselves have been designed to attempt to be models of biological motivation.  

Personally, I am interested in motivated behaviors and think that future AI developments should take note of this field, but again, the present work does not provide actionable insights into implementing an artificial motivation system.  At the same time, this work does not provide interesting enough neurobiological results for those to stand on their own either.

Minor clarification:

"trained to perform in realistic tasks" -- the task is very simple.  I would consider this a fairly abstract model of the task.  

review_title: Official Blind Review #1

Meta-review: This paper proposes a deep RL framework that incorporates motivation as input features, and is tested on 3 simplified domains, including one which is presented to rodents. 

While R2 found the paper well-written and interesting to read, a common theme among reviewer comments is that it’s not clear what the main contribution is, as it seems to simultaneously be claiming a ML contribution (motivation as a feature input helps with certain tasks) as well as a neuroscientific contribution (their agent exhibited representations that clustered similarly to those in animals). In trying to do both, it’s perhaps doing both a disservice. 

I think it’s commendable to try to bridge the fields of deep RL and neuroscience, and this is indeed an intriguing paper. However any such paper still needs to have a clear contribution. It seems that the ML contributions are too slight to be of general practical use, while the neuroscientific contributions are muddled somewhat. The authors several times mentioned the space constraints limiting their explanations. Perhaps this is an indication that they are trying to cover too much within one paper. I urge the authors to consider splitting it up into two separate works in order to give both the needed focus. 

I also have some concerns about the results themselves. R1 and R3 both mentioned that the comparison between the non-motivated agent and the motivated agent wasn’t quite fair, since one is essentially only given partial information. It’s therefore not clear how we should be interpreting the performance difference. Second, why was the non-motivated agent not analyzed in the same way as the motivated agent for the Pavlovian task? Isn’t this a crucial comparison to make, if one wanted to argue that the motivational salience is key to reproducing the representational similarities of the animals?  (The new experiment with the random fixed weights is interesting, I would have liked to see those results.) For these reasons and the ones laid out in the extensive comments of the reviewers, I’m afraid I have to recommend reject.

Decision: Reject