Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 1: Reject
review_content: The paper suggests that one common problem encountered by reinforcement learning algorithms in open environments is "data confusion", which essentially means showing the same input data with different --possibly contradictory-- labels/targets.

The proposed solution to this conceptual problem is to split the original MDP "M" up into multiple simpler MDPs "Mk", where M does contain possibly contradictory ("confusing") data, while each individual Mk does not contain any such problem and, even better, is stationary.

The "subjectivity" function "h" then has as role to split any data tuple across Mk, possibly using extra information kappa.

Furthermore, several theorems show that under several conditions, the return of the subjective policy (learned via Mk) is not worse than that of the "traditional" policy.


I lean towards rejecting this paper. The whole gist of the framework can be crudely summarized as "if data contradicts, split up into non-contradictory sets using extra info." The motivation keeps repeating that no task-specific prior knowledge being necessary, but I believe this hinges on "h" being sensible, which might not be feasible without task-specific prior knowledge.

Furthermore, and this is my main concern, there is not a single experiment demonstrating how any of this would behave in practice. It would be good to have one (possibly constructed) experiment showing that data confusion indeed is a problem in practice (intuitively, it is), and then a specific instantiation of the framework that solves this example. Furthermore, I am not convinced that the proposed bounds can easily be concretized for an instantiation of the proposed framework, especially when considering deep networks; again, this concern could be alleviated by an example instantiation. Proposing something that is in principle more general and "in principle cannot be worse" but then not demonstrating that it actually is the case is, in my opinion, not enough.



Finally, and this is not a deciding factor in my rating, the paper has quite some writing problems. On the first page alone, I found a lot of spelling and grammatical mistakes (see list at end) and the notation is sometimes confusing to me. For example, "R" is defined as a mapping of S x /R -> [0,1], but what is "/R" (curly R)? And then in (1) R is used with a single argument while in (2) not anymore. I can guess what is meant, but it feels inconsistent. In Theorem 1, I believe it should be "the gap \delta >= 0" and not "the gap g >= 0", no?

Abstract and 1st paragraph mistakes (unfortunately, no line numbers in this template!): "researches" -> "research", "algorithm designing" -> "algorithm design", "task-specific prior knowledge about tasks." -> "task-specific prior knowledge.", "not known in prior" -> "not known a priori", "Classical RL model environment..." -> "Classical RL models environment...".
Also, quite some citations are missing the year, e.g. Schaul et al., Papavassiliou&Russell, ...
review_title: Official Blind Review #3

final_score: rating: 3: Weak Reject
review_content: The paper introduces the Subjective Reinforcement Learning framework to formalize the problem of using extra information to split large, nonstationary environments into separate, simple, stationary MDPs. First, the paper introduces and motivates the problem on a high level: the same state-action pairs may transit to different successive states and rewards because of nonstationary dynamics/reward function, or variance in the environment or tasks. This phenomenon is termed "data confusion". The paper then summarizes some related approaches to dealing with this phenomenon. Next, the paper introduces the subjective RL framework in detail in section 3:
- Extra information (kappa) is needed to resolve the data confusion.
- The "subjectivity" (h) is a function that maps the extra information to a vector of weights over "subjective" MDPs.
- A policy is maintained for each subjective MDP, and the overall policy is the vector product of h and the vector of subjective policies.
In section 4, the paper presents 3 theorems arguing that using the subjective RL framework doesn't harm performance. The paper then gives brief guidelines for designing algorithms using the subjective RL framework before concluding.

At the present time I recommend rejecting the paper. It does not actually present a concrete solution method, instead simply giving brief guidelines for the reader to design algorithms by. The subjective RL framework unifies and subsumes several existing approaches, but I don't feel that in itself is a significant enough contribution to warrant publication. The theorems presented essentially argue that using the subjective RL framework does not harm performance, but there are no mentions of the computational costs involved with maintaining policies for each subjective MDP. In addition, it's not clear where the subjective MDPs come from.
The paper also had issues with clarity, including many grammatical errors.

I think this paper tackles an important problem from an interesting point of view, but stops short of giving a concrete algorithm that can be implemented and tested. It seems like a good candidate for a workshop, which could be a good opportunity for discussion and feedback.

General comments:
- The definition of rewards is confusing; script R was never defined.
- "minimize objective 1" should be actually be "maximize objective 1"?
- Rewards are usually defined over state-action pairs, not just states. Why choose this unconventional formulation for rewards?
review_title: Official Blind Review #2

final_score: rating: 3: Weak Reject
review_content: This paper introduces a new framework for reinforcement learning (named subjective reinforcement learning) which aims to resolve some of the inherent problems with RL in open environments.  The authors posit that one problem with RL in open settings is “data confusion”, which they describe as being situations where there are external factors (e.g. timeframe) that affect the action space differently.  They propose a “subjective reinforcement learning framework” which, as I understand it, can be described as an ensemble of traditional MDP’s subject to external factors k.  The paper evaluates how this subjective policy compares to traditional MDP’s in terms of theoretical bounds on performance. 

Although this learning framework seems like a potentially interesting future research direction, I tend to lean towards rejection for the reasons that: (1) the theoretical analysis is difficult to follow and there is sometimes a lack of clarity throughout the paper, (2) it isn’t very clear how easy this framework would be to implement aside from the theoretical guarantees and there aren’t any experiments or proofs of concept that would demonstrate the feasibility or practicality of the proposed framework in a real scenario, (3) the paper would benefit from more discussion of how their work differs from related techniques (like hierarchical RL, various forms of meta-learning, etc.).

Here are some more concrete points:
(1) The feasibility of this framework in a real-world scenario seems a bit hard to imagine and a strong use-case or proof-of-concept would be very helpful.  I liked that this paper provided some theoretical analysis for guiding the design of these systems.  However, it seems like these claims would also benefit from detailed empirical analysis and experiments.  Without empirical results, I feel a bit skeptical about how straightforward it would be to implement such a system or whether it would really be significantly useful in practice.  Similarly, though there are theory-based suggestions for how to optimally design such a system, it might be difficult to implement this system with optimal hyperparameters in a real-world use-case and the challenges in doing so are not really addressed.
(2) In spite of claims that this method is able to be trained without domain knowledge, it seems like domain knowledge would still be necessary for things like determining what external information (K) is available and necessary, determining the appropriate N_S, etc.  It may be helpful for the authors to explain a bit more about how these things can be determined in a truly agnostic way.
(3) It seems like there should be more discussion of the difference from hierarchical reinforcement learning.  In practice, hierarchical RL also can be used in similar ways to what’s described here.  As the authors point out, hierarchical RL is not necessarily splitting into submodels that handle data confusion problems, it seems like that is a constraint that could be added into a hierarchical framework’s design.
(4)  I appreciate that the authors provide detailed theoretical analysis, but it can sometimes be confusing and difficult to follow.  I had some trouble evaluating the correctness of several of the proofs.  It may benefit from re-writing with more concise definitions of all of the variables and more clearly stated assumptions about observable information.  Here are some points of confusion for me:
    - It seems like certain letters (eg. A or K or N_s vs N_d) are being overloaded as variable names with different fonts.  I realize that this is somewhat unavoidable, but I would recommend that the authors try to disentangle the naming a bit for improved clarity.
    - In theorem 1, you stated "the number of all possible data samples tends to infinity when the total number of samples N_d approaches infinity".  This proposition seemed confusingly worded to me.  Maybe I am misunderstanding the wording, but it seems like possibly a tautology?
    - I’m not sure I understand what is meant by “fake subjective policies” in Theorem 1.  Could you explain what is meant by that and the intuition here a bit more?
    - It’s still unclear to me how K (that is, the external information) is being collected at any given timeframe.  Are you assuming that the necessary types of external information corresponding to K (your examples are ‘state history, out-of-MDP task encodings, samples from related tasks, etc.’) have been pre-determined?  If so, how could the most-appropriate type of external information be chosen in a practical way?

I also noticed a few (very minor) grammar errors that authors may want to fix, though they did not affect my review:
- page 1: "tasks in open environments poses difficulties" --> "tasks in open environments pose difficulties"
- page 1: "Problem is that both..." --> "The problem is that both..."
- page 2: "we propose a novel framework named as Subjective reinforcement" --> "we propose a novel framework named Subjective reinforcement"
- page 4: "should contain no data confusion" --> "should not contain data confusions"
review_title: Official Blind Review #1

Meta-review: The authors propose a learning framework to reframe non-stationary MDPs as smaller stationary MDPs, thus hopefully addressing problems with contradictory or continually changing environments. A policy is learned for each sub-MDP, and the authors present theoretical guarantees that the reframing does not inhibit agent performance.

The reviewers discussed the paper and the authors' rebuttal. They were mainly concerned that the submission offered no practical implementation or demonstration of feasibility, and secondarily concerned that the paper was unclearly written and motivated. The authors' rebuttal did not resolve these issues.

My recommendation is to reject the submission and encourage the authors to develop an empirical validation of their method before resubmitting.
Decision: Reject