An Architecture for Distinguishing between Predictors and Inhibitors in Reinforcement Learning
Authors: Patrick C. Connor, Thomas P. Trappenberg
ICLR 2014

Abstract: Reinforcement learning treats each input, feature, or stimulus as having a positive or negative reward value. Some stimuli, however, negate or inhibit the values of certain other predictors (excitors) when presented with them, but are otherwise neutral. We show that both linear and non-linear value-function approximators assign inhibitory features a strong value with the opposite valence of the predictor it inhibits (i.e., inhibitor= -excitor). In one circumstance, this gives a correct prediction (i.e., excitor + inhibitor = neutral outcome). Importantly, however, value-function approximators incorrectly predict that when the inhibitor is presented alone, a negative or oppositely valenced outcome will follow whereas the inhibitor alone is actually followed by a neutral outcome. Essentially, we show that having reward value as a direct predictive target can make inhibitors indistinguishable from excitors that predict the oppositely valenced outcome. We show that this problem can be easily avoided if the reinforcement learning problem is broken into 1) a supervised learning module that predicts the positive appearance of primary reinforcements and 2) a reinforcement learning module which sums their agent-defined values.


Peer review:
[Review]:
An Architecture for Distinguishing between Predictors and Inhibitors in Reinforcement Learning
Patrick C. Connor, Thomas P. Trappenberg

Summary: The authors discuss benefits of reinforcement learners with a supervised module that predicts primary reinforcements and a reinforcement learner which sums their values.

Quite a few references to relevant previous work seem to be missing, e.g., see below.

p 2: 'The notion of predicting future observations or states is not new to RL [9, 10].'

These references are of 2004 and 2005. Two much older references from 1990 on this:

R. S. Sutton. First Results with DYNA, an Integrated Architecture for Learning, Planning and Reacting. Proceedings of the AAAI Spring Symposium on Planning in Uncertain, Unpredictable, or Changing Environments, 1990.

J.  Schmidhuber. An on-line algorithm for dynamic reinforcement learning and planning in reactive environments. In Proc. IEEE/INNS International Joint Conference on Neural Networks, San Diego, volume 2, pages 253-258, 1990.

p 6: 'the architecture we propose is not trained using a scalar reward prediction error, but rather a vector of state feature-speciﬁc prediction errors'

So it is like the vector-valued adaptive critic for multi-dimensional reinforcement signals of the old system from 1990 below - please discuss the differences: 

J. Schmidhuber. Recurrent networks adjusted by adaptive critics. In Proc. IEEE/INNS International Joint Conference on Neural Networks, Washington, D. C., volume 1, pages 719–722, 1990

J. Schmidhuber. Additional remarks on G. Lukes’ review of Schmidhuber’s paper ‘Recurrent networks adjusted by adaptive critics’. Neural Network Reviews, 4(1), 1990.

Below other old predictors of state feature-speciﬁc prediction errors. RL is based on training a recurrent system that combines the world model and the actor. Which are the relative advantages or drawbacks of the approach of the authors?:

N. Nguyen and B. Widrow. The truck backer-upper: An example of self learning in neural networks. Proceedings of the International Joint Conference on Neural Networks, 357-363, IEEE Press, Piscataway, NU, 1989

J. Schmidhuber and R. Huber. Learning to generate artificial fovea trajectories for target detection. International Journal of Neural Systems, 2(1 & 2):135-141, 1991

Experiments: this is a very simple RL task. 

General recommendation: It is not quite clear to this reviewer how this work goes beyond the previous work from 20 years ago mentioned above. At the very least, the authors should make the differences very clear.