An Architecture for Distinguishing between Predictors and Inhibitors in Reinforcement Learning
Authors: Patrick C. Connor, Thomas P. Trappenberg
ICLR 2014

Abstract: Reinforcement learning treats each input, feature, or stimulus as having a positive or negative reward value. Some stimuli, however, negate or inhibit the values of certain other predictors (excitors) when presented with them, but are otherwise neutral. We show that both linear and non-linear value-function approximators assign inhibitory features a strong value with the opposite valence of the predictor it inhibits (i.e., inhibitor= -excitor). In one circumstance, this gives a correct prediction (i.e., excitor + inhibitor = neutral outcome). Importantly, however, value-function approximators incorrectly predict that when the inhibitor is presented alone, a negative or oppositely valenced outcome will follow whereas the inhibitor alone is actually followed by a neutral outcome. Essentially, we show that having reward value as a direct predictive target can make inhibitors indistinguishable from excitors that predict the oppositely valenced outcome. We show that this problem can be easily avoided if the reinforcement learning problem is broken into 1) a supervised learning module that predicts the positive appearance of primary reinforcements and 2) a reinforcement learning module which sums their agent-defined values.


Peer review:
[Review]:
The reviewers brought up very good comments so that we previously considered withdrawing the paper to have more time to work on it.  We now feel we that we are able to address the reviewer's concerns, especially given your recent indication of acceptance.  Therefore, we now retract our statement of withdrawal. Below we answer each of the reviews and will soon incorporate the associated changes into our paper.  We apologize for the lateness of our responses, but have only come to this decision recently.
 
To summarize the most significant proposed changes:
 - Adjusting the abstract and opening paragraphs to avoid implying that reinforcement learning does something in general that it does not do (see David's review and my response above)
 - Replace the partial state-prediction of the first stage of our architecture with a reward-prediction and punishment prediction learners, which are rectified.
 - Enhancing the simulations to include real values instead of binary only and to vary the number of data points that expose the non-linear nature of the problem.  Also, we will add two non-linear VFAs (MLP and CART) to show that the problem with VFAs is not specific to support vector regression.
 - Strengthening the discussion by providing greater distinction between the present work and previous work (as noted in the first anonymous review) and address the classical conditioning comments of the second anonymous review.

This represents a significant change in the substance of the paper, but the ultimate conclusions do not change.  The changes will result in focusing more squarely on our main point and strengthening the support for it through more generalized simulations.

Take care,
  - Patrick Connor (and Thomas Trappenberg)