An Architecture for Distinguishing between Predictors and Inhibitors in Reinforcement Learning
Authors: Patrick C. Connor, Thomas P. Trappenberg
ICLR 2014

Abstract: Reinforcement learning treats each input, feature, or stimulus as having a positive or negative reward value. Some stimuli, however, negate or inhibit the values of certain other predictors (excitors) when presented with them, but are otherwise neutral. We show that both linear and non-linear value-function approximators assign inhibitory features a strong value with the opposite valence of the predictor it inhibits (i.e., inhibitor= -excitor). In one circumstance, this gives a correct prediction (i.e., excitor + inhibitor = neutral outcome). Importantly, however, value-function approximators incorrectly predict that when the inhibitor is presented alone, a negative or oppositely valenced outcome will follow whereas the inhibitor alone is actually followed by a neutral outcome. Essentially, we show that having reward value as a direct predictive target can make inhibitors indistinguishable from excitors that predict the oppositely valenced outcome. We show that this problem can be easily avoided if the reinforcement learning problem is broken into 1) a supervised learning module that predicts the positive appearance of primary reinforcements and 2) a reinforcement learning module which sums their agent-defined values.


Peer review:
[Review]:
I believe this paper mis-characterizes the Reinforcement Learning (RL) problem and the problem it purports to solve is, in fact, not a problem at all in a general reinforcement learning context.  

In particular, the first sentence of the abstract states: 
'Reinforcement learning treats each input, feature, or stimulus as having a positive or negative reward value', but this is simply not true in general for reinforcement learning.

In the most general setting, the reward at each step t+1 (r_{t+1}) is calculated as a function of the state at time t, the state at time t+1, and the action at time t.  So r_{t+1} = f(s_t,a_t,a_{t+1}).  Even if we specify the problem somewhat so that reward is only a function of the state (r_{t+1}  = f(s_{t+1}), this is still a general enough framework to incorporate arbitrary interactions between different variables that compose the state.  

In contrast, this paper assumes linear interactions between at least some subset of variables that specify the state.

Succinctly, it is not each input, feature, or stimulus that has a positive or negative reward, rather, it is the totality of the environment's state that has an associated reward.  

I am not aware of common practices in RL, so it may be that a linearity assumption such as the authors have made is commonly used in practice.  And it may be that their approach to weakening this assumption could be useful in practice, or theoretically.  But I find the evidence presented extremely unconvincing on this point.  Primarily this is because there is no presentation of other approaches to weakening the imposed linearity constraint.