An Architecture for Distinguishing between Predictors and Inhibitors in Reinforcement Learning
Authors: Patrick C. Connor, Thomas P. Trappenberg
ICLR 2014

Abstract: Reinforcement learning treats each input, feature, or stimulus as having a positive or negative reward value. Some stimuli, however, negate or inhibit the values of certain other predictors (excitors) when presented with them, but are otherwise neutral. We show that both linear and non-linear value-function approximators assign inhibitory features a strong value with the opposite valence of the predictor it inhibits (i.e., inhibitor= -excitor). In one circumstance, this gives a correct prediction (i.e., excitor + inhibitor = neutral outcome). Importantly, however, value-function approximators incorrectly predict that when the inhibitor is presented alone, a negative or oppositely valenced outcome will follow whereas the inhibitor alone is actually followed by a neutral outcome. Essentially, we show that having reward value as a direct predictive target can make inhibitors indistinguishable from excitors that predict the oppositely valenced outcome. We show that this problem can be easily avoided if the reinforcement learning problem is broken into 1) a supervised learning module that predicts the positive appearance of primary reinforcements and 2) a reinforcement learning module which sums their agent-defined values.


Peer review:
[Review]:
Hi, David.  It's wonderful to have this venue for receiving feedback and being able to discuss our work.  I think I understand your feedback and that some of the early statements of our paper have unfortunately distracted from its main theme.  We agree with you that Reinforcement Learning in general is not confined to attributing a positive or negative value to each contributing feature of a state, such that the state's apparent value is always the linear sum of their effects.  Rather, it encompasses more than this.  We did not intend the received meaning, though we can see the value of making minor textual changes to clear this up.

The problem we find, though, is that given data which suggests linear contributions of individual features (the 'partial' dataset in our simulations), value function approximators (VFAs), whether linear (LR-P results in the paper) or non-linear (SV-P results), treat the features as linear, which comes as no surprise.  Essentially we want to highlight a real world case where making this 'linear assumption' is actually inappropriate, since it will add to prediction errors.  

Whereas features/predictors may be associated with either reward, punishment or neither (whether linearly as individuals or in non-linear combinations as in XOR), features can also be associated with cancelling these otherwise predicted outcomes.  That is, some features can predict outcomes and others can cancel those predictions.  We show that linear and non-linear VFAs will see canceling features as having the opposite effect (say, value=-1) on the state-value as do the features which predict the associated outcomes.  Then, when a predictor (value=1) and a cancelling feature (value=-1) are presented together, a neutral outcome (1-1=0) is expected.  But what happens when these canceling features are presented alone, having no prediction to cancel?  According to the VFAs, they are not neutral but still have their opposing effect if they have not been trained on this case (i.e., value = 0 - 1 = -1).  This mistaken sense of opposing value adds to the prediction error.  Ideally, one would have the 'full' dataset, which tells us the non-linear nature of such cancelling features, but we are unlikely to have it all (or most?) of the time.

We bring all of this up in the paper because we see a simple way to avoid the problem when only the 'partial' (i.e., 'linear') dataset is available.  We change the representation or the prediction target from reward-value to the outcomes of interest (e.g. food, shock, etc.).  Then, when a canceling feature of an outcome is presented alone, it's opposing (-1) contribution can be truncated (i.e., -1 -> 0) since the outcome will never be less than 0 (e.g., there will never be less than zero shock).

It is true that we do not show evidence for other ways to correct for or weaken the 'linear assumption' here, because we have not seen any previous acknowledgement of this issue, and thus no attempts at solving it as of yet.  I wonder if this might be because having enough data (i.e., the 'full' dataset) makes the problem moot, just as having enough supervised data (i.e., an exhaustive mapping) can make the use of a deep learning representation moot, for example.  Instead, I think the most important evidence we show is that even non-linear VFAs are subject to treating the partial, linear data in a linear fashion, which is inappropriate in this case.

In summary, we recognize the need to do some rephrasing in the early part of the paper regarding the linearity of feature contributions to state-value.   Yet, we focus on showing that when the data does suggest linear contributions, they sometimes represent a specific non-linear case that VFAs fail to recognize but that can be easily accommodated without error by changing the prediction target from reward-value to outcome.  We are the first to acknowledge this as far as we know, and thus offer the only existing way to avoid accruing the associated VFA prediction errors.  

Take care,
   - Patrick