An Architecture for Distinguishing between Predictors and Inhibitors in Reinforcement Learning
Authors: Patrick C. Connor, Thomas P. Trappenberg
ICLR 2014

Abstract: Reinforcement learning treats each input, feature, or stimulus as having a positive or negative reward value. Some stimuli, however, negate or inhibit the values of certain other predictors (excitors) when presented with them, but are otherwise neutral. We show that both linear and non-linear value-function approximators assign inhibitory features a strong value with the opposite valence of the predictor it inhibits (i.e., inhibitor= -excitor). In one circumstance, this gives a correct prediction (i.e., excitor + inhibitor = neutral outcome). Importantly, however, value-function approximators incorrectly predict that when the inhibitor is presented alone, a negative or oppositely valenced outcome will follow whereas the inhibitor alone is actually followed by a neutral outcome. Essentially, we show that having reward value as a direct predictive target can make inhibitors indistinguishable from excitors that predict the oppositely valenced outcome. We show that this problem can be easily avoided if the reinforcement learning problem is broken into 1) a supervised learning module that predicts the positive appearance of primary reinforcements and 2) a reinforcement learning module which sums their agent-defined values.


Peer review:
[Review]:
Summary
This paper proposes a value function approximation architecture which explicitly models inhibition effects for reinforcement learning. In this context, inhibition refers to a stimulus that eliminates a reward when presented along with a stimulus which usually produces a reward. For example, the stimulus pair ‘NP’ does not produce a reward whereas ‘P’ alone does, and ‘N’ alone produces a reward of 0. The authors address the issue of value function approximators not correctly modeling ‘N’ in isolation having reward 0, as opposed to a negative reward. The authors propose decomposing the value approximation problem into two halves, one to estimate negative rewards and another to estimate positive rewards. The outputs of these functions are then combined to form a final value estimate. The authors derive a linear regression function approximator which rectifies its output to properly fit into the proposed two-stage scheme. The method is evaluated on a toy dataset of 16 examples which illustrates the problem.

Review
The abstract is very hard to follow. Please rewrite it to more clearly describe the problem and your contribution.

The problem of confusing negative reward stimuli with inhibitors does not seem inherent to the reinforcement learning problem. Instead it seems to be a property of the chosen value function approximation algorithm. The authors should discuss, and perhaps test, whether a more expressive class of value function approximators exhibit this property. Given the conference, a neural network function approximator could be appropriate. Additionally, why not decompose the estimator into positive / negative reward estimates but use neural nets to approximate each half? They can naturally output rectified rewards without additional derivation. 

The derivation equations are easy to follow, but their motivation is unclear. Please add more text discussing why it is necessary to make such derivations instead of choosing a different estimator

The experimental evaluation leaves much to be desired. The toy example is nice as a control experiment, but far from comprehensive. I would like to see a larger problem in which your approach performs better or discovers inhibition structure other value function approximators can not discover.

Key points
+ Proposed approach of decomposing value function approximation seems interesting and could lead to approximating functions which are easier to understand or debug
- Unclear whether a more expressive class of value function approximators would fail to learn about inhibition stimuli automatically
- Toy dataset evaluation
- Writing is okay but does not clearly introduce the problem and motivate the approach