An Architecture for Distinguishing between Predictors and Inhibitors in Reinforcement Learning
Authors: Patrick C. Connor, Thomas P. Trappenberg
ICLR 2014

Abstract: Reinforcement learning treats each input, feature, or stimulus as having a positive or negative reward value. Some stimuli, however, negate or inhibit the values of certain other predictors (excitors) when presented with them, but are otherwise neutral. We show that both linear and non-linear value-function approximators assign inhibitory features a strong value with the opposite valence of the predictor it inhibits (i.e., inhibitor= -excitor). In one circumstance, this gives a correct prediction (i.e., excitor + inhibitor = neutral outcome). Importantly, however, value-function approximators incorrectly predict that when the inhibitor is presented alone, a negative or oppositely valenced outcome will follow whereas the inhibitor alone is actually followed by a neutral outcome. Essentially, we show that having reward value as a direct predictive target can make inhibitors indistinguishable from excitors that predict the oppositely valenced outcome. We show that this problem can be easily avoided if the reinforcement learning problem is broken into 1) a supervised learning module that predicts the positive appearance of primary reinforcements and 2) a reinforcement learning module which sums their agent-defined values.


Peer review:
[Review]:
This paper proposes to break value prediction into prediction of a specific outcome (e.g., food) and a value for that outcome allows reinforcement learning to make the correct predictions for inhibitors -- as opposed to standard RL which predicts negative value for a reward inhibitor. In practice, this is implemented by separately predicting positive (reward) and negative (punishment) values using rectified predictors. Tests on a toy problem with 16 training points show that this setting indeed avoids the usual prediction of opposite valence for an inhibitor.

Inhibitors are a difficulty for RL and present an interesting puzzle, however this paper fails to make a good case:

1) There is no real technical contribution here: separating positive from negative values while using standard predictors has been done often (e.g. in computer vision); the toy test is exceedingly simple.
The bulk of the technical part of the paper is a lengthy explanation of simple maximum likelihood estimation of a thresholded linear function.

2) The paper is motivated as 'tak[ing] a cue from biological systems', but there are problems with this too:
- 'we train function approximators to predict specific primary reinforcements (e.g., food, charging station, etc.).' This makes the job of an inhibitor easier indeed, but predicts that a function approximator should not be able to produce trans-reinforcer blocking, where a predictor trained on one specific primary reinforcement (e.g., shock) blocks conditioning to another reinforcement (e.g., loud noise). See:
Bakal, C. W., Johnson, R. D. & Rescorla, R. A. The effect of change in US quality on the blocking effect. Pavlov. J. Biol. Sci. 9, 97–103 (1974).
- 'we might take a cue from biological systems, where it seems that the dynamic revaluation of a primary reinforcer due to satiety is second nature': while this is true of primary reinforcers, this is not always true of secondary reinforcers, that RL needs to train as well. There is a lot of literarture on sensitivity to outcome devaluation, e.g.: 
Motivational control of goal-directed action
Anthony Dickinson, Bernard Balleine
Animal Learning & Behavior
March 1994, Volume 22, Issue 1, pp 1-18

In its current form, the paper does not seem to offer enough for acceptance, in terms of either technical novelty or insight into natural behavior.