Subjective Reinforcement Learning for Open Complex Environments
Authors: Zhile Yang*
Haichuan Gao*
Xin Su
Shangqi Guo
Feng Chen
Venue:  (2020)
Keywords: reinforcement learning theory
subjective learning

Abstract: Solving tasks in open environments has been one of the long-time pursuits of reinforcement learning researches. We propose that data confusion is the core underlying problem. Although there exist methods that implicitly alleviate it from different perspectives, we argue that their solutions are based on task-specific prior knowledge that is constrained to certain kinds of tasks and lacks theoretical guarantees. In this paper, Subjective Reinforcement Learning Framework is proposed to state the problem from a broader and systematic view, and subjective policy is proposed to represent existing related algorithms in general. Theoretical analysis is given about the conditions for the superiority of a subjective policy, and the relationship between model complexity and the overall performance. Results are further applied as guidance for algorithm designing without task-specific prior knowledge about tasks.


Under review as a conference paper at ICLR 2020
SUBJECTIVE REINFORCEMENT LEARNING FOR OPEN COMPLEX ENVIRONMENTS
Anonymous authors Paper under double-blind review
ABSTRACT
Solving tasks in open environments has been one of the long-time pursuits of reinforcement learning research. We propose that data confusion is the core un- derlying problem. Although there exist methods that implicitly alleviate it from different perspectives, we argue that their solutions are based on task-specific prior knowledge that is constrained to certain kinds of tasks and lacks theoretical guar- antees. this paper, Subjective Reinforcement Learning Framework is proposed state the problem from broader and systematic view, and subjective policy is proposed to represent existing related algorithms in general. Theoretical anal- ysis is given about the conditions for the superiority of subjective policy, and the relationship between model complexity and the overall performance. Results are further applied as guidance for algorithm design without task-specific prior knowledge.
INTRODUCTION
One important target of reinforcement learning (RL) is to realize an autonomous agent that can help people solve various kinds of tasks in open environments, where the agent's 's observations and the environment's 's dynamics is both diverse and unpredictable (Wu et al., 2019). For example, a vision- based robot doing housework in all kinds of room-settings may observe various combinations of indoor scenes; different tasks may require it to take different actions under the same observation; the successor observation after performing an action not predictable, because the shape of rooms, fur- niture settings and pets" movements, etc. are not known a priori. Classical RI models environment and tasks as Markov Decision Process (MDP). From this perspective, tasks in open environments pose difficulties that the state space S is large, the state transition dynamics P is nonstationary, and the reward function R may be contradictory( (Brodeur et al. 2017 Xu et al. 2018).
We argue that the key problem underlying these difficulties is data confusioo,, that is every sample the agent acquires from an open environment may reflect the properties of only a small part of the entire environment for specific time period (Kempka et al., 2016), thus data samples with similar current observations may have contradictory meanings under different circumstances. ie. different transition dynamics, state (or state-action) values or action preferences. For this problem, current RL. methods introduce out-of-MDP information such as task-encoding (Zhu et al. (2066) and doal-description (Wang et al. 2018)) or expand the state space into history space, in which every element is the concatenation of all information gained so far Tr S1a (Farias et al.) 2007]. As for the former category, the quality and quantity of extra information are decided by the designer (Schaul et al. 2015). As for the latter category, although the whole history is what we can use best to alleviate data confusion in theory, it increases the complexity to a maximum point and is hard to achieve in practice; algorithms in this category thus either introduce regularization terms or transform the optimization target easier ones. The problem that both these categories depend on humans' understandings to certain tasks and lack theoretical guarantees; consider that tasks in open environments are diverse and unpredictable, it is necessary to build a system that can dynamically fit tasks without prior knowledge.
Intuitively, we humans also face data confusion problems in the real world, e.g. when going to office from home, different modes of transportation may include conflict behaviors e.g. walking in opposite directions. However, as long as we utilize arbitrary information and subjectively choose one of the modes, e.g. taking the bus, this task becomes concrete and without confusion; also may

---

Under review as a conference paper at ICLR 2020
subjectively focus more concise aspects of the problem e.g. selecting bus numbers, which make the tasks simpler. Inspired by this, we propose a novel framework named Subjective Reinforcement Learning Framework that models the environment and tasks as a set of simple MDPs with data confusion problem, and a subjectivity model over them; the corresponding subjective policy then includes two parts: acquiring subjectivity for each state with subjective samples, and choosing an action under the selected subjectivity. From this perspective, all aforementioned existing methods can be summarized as introducing subjective samples, i.c. state history or extra task information, beyond current observation to form a subjectivity so that the resulting iibbecttiity-cooniiiooeedd policy does not suffer from data confusion problem and thus have better performance. Therefore they can be recognized as instantiations of our subjective policy, and analysis based on our framework is general and meaningful for all these methods.
In this paper, we discuss the conditions under which the subjective policy can achieve better results, and analyze theory the relationship between overall performance and some key elements that characterize subjective policy. The key elements are concrete and modifiable, and the analytical results guide us to design policy models that fit the need of problems in open complex environments without task- prior knowledge and with theoretically guarratted performance.
To sum up. we think our contributions are as follows:
1. To our best knowledge, we are the first point out data confusion as key problem to solving tasks in open complex environments;
2. We propose novel framework named Subjective RL Framework and give theoretical anal- ysis from broader and systematic view;
3. We provide guidances for algorithm design based on our theoretical results.
RELATED WORK
Our subjective RL framework deals with data confusion problem facing tasks in open environments, which has also been implicitly considered in several RL fields: multi-task RL (Zhu et al. 2016). meta-RL (Rakelly et al. 2019), continuous adaptation (Yu et al. 2018) and hierarchical RL (Al- | 2015). We analyze their inherent defects dependence on designers' knowledge for tasks in the following three paragraphs.
The field of multi-task RL considers an agent performing different tasks in possibly different envi- ronments, where data confusion expected to happen across tasks. One direct category methods equips the agent with extra task-specific information (Riedmiller et al.) 2018), e.g. vector 9, describing the goal of task T;; these methods requires the extra information to be available and suf- ficient enough to distinguish tasks from each other, which is guaranteed by designers' knowledge. Considering cases where suitable extra information is not available, various task-encoding methods have been proposed (Sung et al.) a (017),, which generally extract a task feature vector from the whole trajectory r and build the policy under condition of such feature vectors. These methods can be concluded as adding regularization terms into the complex end-to-end framework according perts' prior knowledge, which includes the length (Sorokin & Burtsev 2019) and prior distribution (Rakelly al. 2019) of the feature vector.
Meta-RL methods, or learning-to-learn, turns to model the learning process instead of the original task (Gupta et al. 2018a|b) b), which implicitly bypass the problem of data confusion. However, these methods still treat the learning process as a single-value function, e.g. the learning process of task embedding (Lan et al. 2019) and an exploration strategy (Garcia & Thomas 2019); thus their successes are based on the assumption that there are no data confusion in the meta learning phase. Similarly, continuous adaptation problems focuses on the transfer performance from source to target tasks and neglect the data confusion problem by assuming that the relationship between successive tasks can be correctly modeled by single-value mappings (Al-Shedivat et al. 2018).
Hierarchical RL is broad field where methods aim to divide complex tasks into many different sub-tasks so that the overall data-efficiency can be improved (Barto & Mahadevan 2003 AI- Emnan 2015). Although this idea of -nd----n-- may result in the confusing data samples being divided into different sub-models and thus each sub-model handles no confusion, which is one typical target of subjectivity in our framework, the actual performance is determined by the design
2

---

Under review as a conference paper at ICLR 2020
of the hierarchical structure, e.g. number of options (Jain et al., 2018) and capacity of both the top-level and low-level models, which are still designed according to prior knowledge and adjusted through empirical results. Instead, our framework takes all these parameers in consideration and can adjust to tasks without prior knowledge.
The data confusion problem in open environments also be recognized from the perspective of emrors in function approximation. Existing research on function approximators in RL consider some specific types of model like linear approximations neural networks, and analysis the overall convergence or performance of certain RL algorithms (Schoknecht 2003 Achiam et al., 2019 Papavassiliou Russell 1999). However, approximators considered are end-to-ent and do not consider the utilization of extra information. Different from these, our framework formalizes both original data samples and extra information used to construct subjectives, and so has the capac- ity to analyze the effect of extra information.
given about
3 SUBJECTIVE REINFORCEMENT LEARNING FRAMEWORK
In this section we describe the data confusion problem of tasks in open environments. Then give mathematical formulation of our subjectivity reinforcement learning framework that states the data confusion problem from broader and systematic view.
3.1 PROBLEM STATEMENT
In traditional RL, the environment and tasks are modeled as (T,S,A, 7, P), where T = {0, 1, is the set of considered time steps, S {s;] the state space,
{a,} the action space, P (8 [8,, az.) ) the transition probabilities from each state-action pair to successive states, R:S x R + [0. 1] the reward distribution over states that expresses the task, y the discount factor that describes the relative importance between short-sight and far-sight rewards, and p the proba- bilistic distribution of initial observations. The goal traditional RL is to find a policy TT : A that maximizes the following return:
max G (TT)
Es(s)) Σ S0ES
drl
Σ pso) Van (80)
(1)
FrEllT
1=1
E
where Vxp (s) is the state-value function that satisfies the following Bellman equation:
Σ (() f. R (s'ss') VVan (s') dr s'ES
Vay (s)
Σ Tr (als) aEA
(2)
RL agents interact with the environment to collect transitions (st, ae, $141, T++1)- Different RL algorithms utilize the collected transitions and equation (2) in different ways to construct functions that represent polices so as to maximize objective T Generally they can be summarized into three categories: -onee-construuctng, value-based, and direct action mapping. which respectively process the transitions into forms of a,) + $t+1. 8, + V(or (s,, a,) + Q for state-action value-based methods) and 8; 0g. In this paper call such correspondences "data samples" emphasize that they are the actual data used to construct function that represent the policy, and use = (s, y) to generally represent the content and label of data sample. Further, we denote the function gr()), and the optimization problem becomes:
min
Cly, gr (z:))
where D is the set of data samples and No is the number of them; L is predefined loss function, e.g. 12-norm.
Note that data samples reflect properties of environment, policy and tasks, and they depend on the sampling process RL. Because (r(-) a single-value mapping, normally the optimization problem 3 results in the expectation of data samples and alleviate the variances induced by the sampling processes. However, in open environments the observation space large, the transition dynamics P is nonstationary, and reward function R may be contradictory; thus same state-action
3

---

Under review as a conference paper at ICLR 2020
pairs may transit to contradictory successive states and rewards; then the Bellman equation 2 may return different label y for the same data content æ. Such differences come from the variances in environment or tasks, and should not be entangled through expectation. In this paper name this problem data confusion. To eliminate this problem, extra information, which we denote as K, need to be introduced, and the key problem is how to select K and efficiently use it eliminate data confusion at the minimum cost of simplicity. In the next subsection we introduce Subjective RL Framework formalize this problem.
As aforementioned, methods that expand the state space directly into the history space and try to construct the mapping from history to labels does not necessarily eliminates this problem practice, so it remains meaningful to analyze the data confusion problem under the assumption that function approximators can only handle finite length of states. To make notations simple, here we focus on confusing mapping from single states and leave the history to the subjectivity part, without loss of generality.
3.2 FRAMEWORK
We propose to model tasks in open environments as M*), where h is the subjectivity that "sub- jectively" treats the entire large. nonstationary and possibly contradictory environment as some sep- arate, simple and stationary MDPs M = (Th S AA, pk R * * p²) which we call subjective MDPs. One key property of subjective MDP is that there are not data confusion problems for the functions used to construct corresponding policies, i.e n (als), V (s). Q (s, a) and PSS (s, a) should not contain data confusions. The expected role of h is to utilize extra information to divide contradictory data samples into different sets. A formal definition of it is given below.
Definition 1. The subjectivity in subjective RI. framework is a function that maps piece of extra information i to the sum-to-one vector that corresponds to the weights of all possible subjective MDPs M
Correspondingly, we maintain subjective policy T (als, x) for each subjective MDP and the over- all policy can be formulated as:
No. he (s, 15) T (als)
Te (a)s,
where he is the k-th element of h and Ns is a variable characterizing the number of maintained subjective MDPs; ~ FK) where F(x) is distribution that is assumed to be available. Here may correspond to multiple types of information. including state history, out-of-MDP task encoding, samples from related tasks, etc. Similarly the overall value functions model transitions can be represented in similar forms, e.g. V*, (s, x) DN', ha (s, Vg (s).
Then the overall global optimization problem of the task becomes:
f (1
max G (T²2 = Ee(s) Ell,
72
Σ (50) Vs, (5o)
(5)
MOES
MOES
where
Na he (s) ph (s'1s,0)
Va, (s,k)= = Σ
[TT (als,
aEAA
s'ES
A=1
(6)
Σ hu (s, J. (R* (rls') VVv, (s (s.
Ns
)
Note that the subjectivities only disentangle the original problem into a variable number of subjec- tive MDPs and does not change the original properties, thus the global return G reflects the same optimization goal.
From the perspective of data samples, the policy function (=(-) becomes:
No.
gz (x, k)) = Σ he (z, k) g (x) = h (z. gs (x)
(7)
4

---

Under review as a conference paper at ICLR 2020
Stream of extra information
range of effect
( ki)
Stream of data sumples (x,, yi)
X
X y2
episode
Attributed Subjective MDP
M;
M
Mj
Mi
M,
Mi
Mi
Figure 1: Samples in the Subjective RL Framework
where Ø the k-th mapping for data samples from the corresponding subjective MDP, and gs is the concatenation of all the mappings.
One important property of extra information к is that may become available only after certain time step and its effect subjectivity about data samples may only last for finite time-period, e.g. the history of states Tr appear at time-step and expires after the possibly existing episode ends. Therefore use Kę to denote the extra information that becomes available time-step t, and introduce a data-related mask to characterize its range of effect: & (Ij, k4) if K; can affect the subjectivity about data content zj. and O otherwise. Note that only depends on the arriving time of extra information and the data samples, so it should be available under most conditions. Then the constraint on h can formalized as
lok (2r, Ki))
if + (2, for k 1,2,..., Ns
Ns
Figure I provides an illustration of the relationship between data samples, extra information and the mask.
We parametrize and gs with aj and as respectively, and require the range 4д to satisfy the constraint expressed in eq. (8). The optimization problem about data samples then becomes:
ND.
1
min ag
Σ Clu, (2i, re)) gs (2:))
ND
where N₂ is the number of available extra information.
THEORETICAL ANALYSIS
In the above section we formalize our Subjective Reinforcement Learning Framework which for- mally considers the data confusion problem and the extra information used to eliminate it. Below we analyze in theory the performance a subjective policy. In the first part we focus on the return G which is the optimization target of RL; in the second part turn analyze the risk bound of the policy function which relates closely to G, is general, and does not depend on certain algorithms.
4.1 RETURN GAP
In reinforcement learning, the return G is the overall criterion to evaluate the performance of policy. Here we firstly define the return gap as:
max G (TT) max G (TT) I.
(10)
Frelle
where n. refers the subjective policy and Fт the policies in traditional forms.
For tasks in open environments, there may be more than one optimal trajectories 7" that maximize the global return. For simplicity in notation here we consider cases where there is only one initial state So and only one optimal trajectory, without loss of generality. Intuitively the loss function (3)
and (9) need not be zero for optimal policy, but the policy function must represent data samples from optimal trajectories well so as to be optimal; thus when data confusion problem happens within such data samples, the subjective policy may gain better performance.
5

---

Under review as a conference paper at ICLR 2020
In the following analysis denote the optimal subjective policy m (which includes hg and m and let the optimal trajectory achieved by it be - = {(Sy, au), (1..a))..... ..., (#I. ar)). ). In order to express the changing of extra information K, information at time t.
subscript i.c. Ke to denote the available extra
Theorem 1. Assume that all possible data samples appear infinite times when the total number of samples ND - oc, then the return gap 2 O holds true;
Proof. In our framework we have subjectivity h that aims divide confused data samples under the constraint of (8. Consider a given piece of extra information k and any subjectivity ho, our subjective policy is defined as eq. (4). As for any traditional policy nr (als), can define fake subjective policies as:
eq.44..
Ns
Ty fake (a/ss k:) =
Σ NOR (8, K) faake (als)
(11)
where T². fake (als) = TT (als) for any s, a, k. Then we have fa,a (als) = TT (a|s) hold true for any 8, a, Hi, which means the policy space IT is equally transformed to IT f,ae Because Ts Sake (als) are defined to be the same with reference to we have fake C II.. Therefore, under the condition that samples required to estimate the optimal policy are plenty, which ensures that an optimal policy can be learned out of the data, the introduced subjectivity extends the policy space and theoretically provides no worse result than aigle-value-basee methods, that 8 2
D
Corollary 1. Further, if there exists steps i, E {l,..., T such that transitions from T satisfies 8, =Sy 83 but nz (als;, K4) + πı (als;, then 8 >
Proof: We have TT (als;) = (als}) because s; = sj. Because of π₃ (a|s;, K.) * πΙ (als), Ky), Tr (als;) = ag and कт (a]s;) a, cannot hold at the same time. Thus T cannot be fully presented by TT. As analyzed in part (1), the policies that can be represented by rr is a subset of and according to the assumption that T is unique, we reach the conclusion that 0.
The following theorem transforms the return gap into the analysis of risk bound of the policy func- tions:
Theorem 2. Under the assumption of sufficient sampling and exploration, the gap between approx- imate return and the optimal return satisfies the following inequality:
δeom + 27E
lim |G (TT.) - G
(12)
(1 7)2
where Tw is the policy at the with iteration, ?i the discount rate, 8eom the bound error incurred in computation of policy update and e the worst-case bound of error on function approximation Bertsekas & Tsitsiklis [997):
max SES
V. (8)
VAW (5) Se. = 1122....
(13)
Proof: Bertsekas & Tsitsiklis (197) proves that:
lim sup max VVa (s) - V (s) 8com + 2yc SES
(14)
t-+00
(1 (1
where 4 the optimal state value function. Recall that the overall optimization problem is defined as:
max (T) Σ p (so) Vx (so)
(15)
Fell
SoES
Because p (so) 2 0 and (so) = for any su E S. the following equation gives the proof:
lim G (mT) - G
lim
Σ (3o) VaL (so) - Σ (50) V (so)
S0ES
lim
(so) VTw (so) V (so)
(16)
8 8rom 2
Σ (50)
lim VaL (so) 4 (so) 5
w 200
- 7)
E

---

Under review as a conference paper at ICLR 2020
In practice, Y is a part of the environment, and doom can be made small (Bertsekas & Tsitsiklis 1997). Thus the above theorem shows that decreasing is necessary for improving the overall performance. In the following subsection we analyze the relationship between and the general design of policy functions.
4.2 RISK BOUND
In order to analyze the errors in approximation of policy functions, we transform the loss L in the optimization target of the policy to compact and probabilistic form. First we concatenate the denotation of extra information number of available di to be and thus use d, represent the imput of subjectivity. We then use b to denote the weight vector output of the subjectivity h (d:); then we combine the Ns subjective MDPs base on as y (d, b) = b gs (d:) where g(-) and gs does not depend on the k part dæ finally equally express the original subjectivity in the probabilistic form h (d,, b) P (bld)) and reach the transformation of L:
and data sample (Ij. Uj) as dip = (K₃, 2j Uj); we consider the
L [v,, h (z) Is (a:)) Bo 2 (vs. 9 (di, b)] h (d,, b)
(17)
We denote the distribution of b and d respectively as F(b) and F(d); the parameters of h and g still an and as. Then we define the expected risk and empirical risk as follows:
Definition 2. The expected risk functional in subjective reinforcement learning is:
Risk (oen as) = /] yy y (di, b)] h (di, (b)
(18)
For simplicity, C g (d;, b)] h (di, b). Then the above definition results in:
to denote the combined set of an and as, and s(d, a)
Risk (ax) = /ʃc ((d.
=
(19)
We denote the considered number of possible to m. under specific design of the output of h (zr, s)., m can be represented using Ns. e.g. in one-hot cases we have m Ns. Then we can define the empirical risk functional as follows.
Definition 3. The empirical risk functional in subjective reinforcement learning is:
m
Riskemp (a, m. n) m
(d,, b)
(20)
f=1
Note that the Risk (a: here expresses the in eq. 12 In order to minimize it, we consider the maximum possible risk gap between Riskemmp and Risk (a):
((...) = sup RRisk (a) Riskemp (a, m, m,))]
(21)
The above form of expression enable to adopt results from Su et al. 2019), which is summarized in the following theorem:
Theorem 3. Consider a" that minimizes the empirical risk 20 the following inequality takes place with probability n
Risk (a") < Räskemp (a", n,,) + Cn.m
(22)
where
4 4epp
{[ 11000000 (1+In
2m,
(5m.m - 1)² m [mm++4d +4
{[ Inm ud (1+In 2n)
2n,
(Sm.m - 1
1r
]m1]
ub (Ba Ab)
Ud
(Ba - Aa)
(23)
Ab. Bb, As and Ba are bounds on functions:
Ab Riskiocal (a, 5 Bb,
Aa 5 sdd Ba
(24)
7

---

Under review as a conference paper at ICLR 2020
where Risklocal (a, b) f (d, (d, dF (d); us and ua are the VC dimension of function h and respectively.
Note that Ab. Bb. Aa and Ba are fixed once the actual form of functions determined. We also want to emphasize that añ and ag, which compose a", should respectively satisfy cq.(8 and the independence of g()) on
5 GUIDANCE FOR ALGORITHM DESIGN
In previous section analyze in theory how the performance of subjective policy relates dif- ferent factors. Now we discuss how these results can be utilized to guide the designing of algorithms for tasks open environments.
As analyzed in section 3, there is data confusion problem in tasks in open environments. Generally extra information should be introduced, but how to select the capacity of functions so that the overall performance is optimal becomes an important problem, because there are not tasks-specific prior knowledges in open environments. In theorem we derive that, under the assumption of sufficient exploration and sampling that ensures the unbiasedness of data samples, the gap between achieved overall return and the optimal return tends to a value that is bounded according to E; then in theorem 3, this is further bounded by Sn,m plus the empirical risk with probability 77, considering their maximum values over the whole learning process.
Note that the elements in eq. 23 i.e. Wb. Ud, m, Ag. Bg, Ad and Ba, are all accessible and modifi- able as long as the data and the designed functions are determined, and can have concrete meanings if actual algorithms and the option-based algorithms are adopted; Ab. Bb, Aa and B& may be directly accessible if sig- moid functions appended function h and g; uss Ud be determined once the functionals and the mask & (z, k) are given.
given. Concretely, may be the number of options if b is set to be one-hot
Therefore, for tasks in open complex environments, we may first select 7) which reflects the needed confidence level of the analytical results. Then, after data samples from the tasks being collected, eq. (2) guides to reduce Sn.m through the design of elements ut. ud, m. Ab. B,, Ad and Ba, which can then be used for the design of actual forms or hyper-parameters Note that the expected risk Risk also includes Riskemp which may also be affected by such adjustments, but as both Riskemp and Sn.m have been expressed explicitly, we argue that a satisfying adjustment is still achievable despite the possible difficulty computation. may also be affected by such adjustments. Besides, the data samples change the policy evolves during training, which may lead to the change of ideal adjustments; for this case, changes to m, up and uy may require the re-tunning of the parameters in e.g. neural networks, one solution is to calculate adjustments with strict 17 but change them when loose 7) is not satisfied, another solution may require designing form of parameter- flexible function model.
6 CONCLUSION
In this paper we consider tasks in open complex environment and propose data confusion one core problem. Although current methods relieve this problem to some extent, their reliance on task-specific prior knowledge limits their capacity for tasks open environments. We propose sub- jective reinforcement learning framework to represent data confusion problem from broader view, and subjective policy that generally represents many existing methods. Our theoretical analysis shows the conditions under which subjective policy outperforms traditional ones, and the relation- ship between overall performance and the key elements that characterize subjective policy. As the elements are concrete and modifiable, analysis further provides guidance for designing algo- rithms with theoretically guaranteed performance. We also point out several difficulties in practical implementation, which include the complexity in calculating ideal adjustments the elements and the probable necessity for re-tuning when considering specific kind models; these problems are planned as future works.
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
Joshua Achiam, Ethan Knight, and Pieter Abbeel. Towards Characterizing Divergence in Deep Q-Learning, arXiv:1905. 088940484444 [es], 2019.
Mostafa Al-Emran. Hierarchical Reinforcement Learning: Survey. International Journal of Computing and Digital Systems, 4:2210-142, 2015.
Maruan Al-Shedivat, Trapit Bansal, Yura Burda, Ily Sutskever, Igor Mordatch, and Pieter Abbeel. Continuous Adaptation via Meta-Leaming in Nonstationary and Competitive Environments. pp. 21, 2018.
Andrew G. Barto and Sridhar Mahadevan. Recent Advances in Hierarchical Reinforcement Learn- ing. Discrete Event Dynamic Systems, 13, 2003.
Dimitri P Bertsekas and John N Tsitsiklis. Neuro-Dynamic Programming. 1997.
Simon Brodeur, Ethan Perez, Ankesh Anand, Florian Golemo, Luca Celotti, Florian Strub, Jean Rouat, Hugo Larochelle, and Aaron Courville. HoME: a Houschold Multimodal Environment. arXiv:1711.11011 [cs, eess], 2017. 2017.
Vivek F. Farias, Ciamac C. Moallemi, Tsachy Weissman, and Benjamin Van Roy. Universal Rein- forcement Learning. arXiv:0707.3000 [cs, math), 2007.
Francisco M. Garcia and Philip S. Thomas. A Meta-MDP Approach to Exploration for Lifelong Reinforcement Learning. Proceedings of the 18th International Conference on Autonomous Agents and MultiAgent Systems, AAMAS '19, pp. 1976-1978. International Foundation for Au- tonomous Agents and Multiagent Systems, 2019.
Abhishek Gupta, Benjamin Eysenbach, Chelsea Finn, and Sergey Levine Unsupervised Meta-
Learning for Reinforcement Learning. arXiv:1806.04600 [es, stat], 2018a.
Abhishek Gupta, Russell Mendonca, YuXuan Liu, Pieter Abbeel, and Sergey Levine. Meta- Reinforcement Learning of Structured Exploration Strategies. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Br Dasa-Bianchi, and R. Garnett (eds.), Advances in Neural Infor- mation Processing Systems 31, pp. 5302-55005 1002-5311. Curran Associates, Inc., 2018b.
Arushi Jain, Khimya Khetarpal, and Doina Precup. Safe Option-Critie: Learning Safety in the Option-Critic Architecture. 1r.Xiv:8077.0ee [cs], 2018.
Michal Kempka, Marek Wydmuch, Grzegorz Runc, Jakub Toczek, and Wojciech Jaskowski. Viz. Doom: A basem Al research platform for visual reinforcement learning. pp. 1-8, Santorini, Greece, 2016. IEEE.
Lin Lan, Zhenguo Li, Xiaohong Guan, and Pinghui Wang. Meta Reinforcement Learning with Task Embedding and Shared Policy. Proceedings of the Twenty-Eights International Joint Conference on Artificial Intelligence, pp. 2792-2222222 2800. International Joint Conferences on Artificial Intelligence Organization, 2019.
Vassilis A. Papavassiliou and Stuart Russell. Convergence of reinforcement learning with gen- eral function approximators. In Proceedings of the Sixteenth International Joint Conference Artificial Intelligence, pp. 748-757, 1999.
Kate Rakelly, Aurick Zhou, Deirdre Quillen, Chelsea Finn, and Sergey Levine. Efficient Off-Policy Meta- Rett-RRiiforremmet Learning via Probabilistic Context Variables. arXiv:1903.08254 [cs, stat], 2019.
Martin Riedmiller, Roland Hafner, Thomas Lampe, Michael Neunert, Jonas Degrave, Tom Van de Wiele, Volodymyr Mnih, Nicolas Heess, and Jost Tobias Springenberg. Learning by Playing Solving Sparse Reward Tasks from Scratch. arXiv: 10567 [cs, stat), 2018.
Tom Schaul, Daniel Horgan, Karol Gregor, and David Silver. Universal value function approxima- tors. In Proceedings of the 32nd International Conference on Machine Learning, pp. 1320,1320, 2015.

---

Under review as a conference paper at ICLR 2020
Ralf Schoknecht. Optimality of Reinforcement Learning Algorithms with Linear Function Ap- proximation. In Becker, S. Thrun, and K. Obermayer (eds.), Advances in Neural Information Processing Systems 15, pp. 1587-1594, MIT Press, 2003.
Artyom Sorokin and Mikhail S. Burtsev. Continual and Multi-task Reinforcement Learning With Shared Episodic Memory. 1r0xiv:905.002 [cs]. 2019.
Xin Su, Shangqi Guo, and Feng Chen. Subjectivity Learning Theory towards Artificial General Intelligence. (rXXv::909.03799 [cs], 2019.
Flood Sung, Li Zhang, Tao Xiang. Timothy Hospedales, and Yongxin Yang. Learning to Learn: Meta-Critic Networks for Sample Efficient Learning. /xxx::/706.09529 [es], 2017.
Xin Wang, Qiuyuan Huang, Asli Celikyilmaz, Jianfeng Gao, Dinghan Shen, Yuan-Fang Wang, William Yang Wang. and Lei Zhang. Reinforced Cross-Modal Matching and Self-Supervised Imitation Learning for Vision-Language Navigation.
[cs], 2018.
Yi Wu, YuXin Wu, Georgia Gkioxari, and Yuandong Tian. Building Generalizable Agents with Realistic and Rich 3d Environment, arXiv: /801 .22009 [cs], 2019.
Nayun Xu, Bowen Tan, and Bingyu Kong. Autonomous Driving in Reality with Reinforcement Learning and Image Translation. arXiv:1801.05220 [cs], 2018.
Tianhe Yu, Chelsea Finn, Annie Xie, Sudeep Dasari, Tianhao Zhang, Pieter Abbeel, and Sergey Levine. One-Shot Imitation from Observing Humans via Domain-Adaptive Meta-Learning. 180.:0580 [cs], 2018.
Yuke Zhu, Roozbeh Mottaghi, Eric Kolve, Joseph J. Lim, Abhinav Gupta, Li Fei-Fei, and Ali Farhadi. Target-driven Visual Navigation in Indoor Scenes using Deep Reinforcement Learning. 1609.05400 [cs], 2016.
10


Reviews:
[{"review_id": "rJg8QSNJ9S", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper suggests that one common problem encountered by reinforcement learning algorithms in open environments is \"data confusion\", which essentially means showing the same input data with different --possibly contradictory-- labels/targets.\n\nThe proposed solution to this conceptual problem is to split the original MDP \"M\" up into multiple simpler MDPs \"Mk\", where M does contain possibly contradictory (\"confusing\") data, while each individual Mk does not contain any such problem and, even better, is stationary.\n\nThe \"subjectivity\" function \"h\" then has as role to split any data tuple across Mk, possibly using extra information kappa.\n\nFurthermore, several theorems show that under several conditions, the return of the subjective policy (learned via Mk) is not worse than that of the \"traditional\" policy.\n\n\nI lean towards rejecting this paper. The whole gist of the framework can be crudely summarized as \"if data contradicts, split up into non-contradictory sets using extra info.\" The motivation keeps repeating that no task-specific prior knowledge being necessary, but I believe this hinges on \"h\" being sensible, which might not be feasible without task-specific prior knowledge.\n\nFurthermore, and this is my main concern, there is not a single experiment demonstrating how any of this would behave in practice. It would be good to have one (possibly constructed) experiment showing that data confusion indeed is a problem in practice (intuitively, it is), and then a specific instantiation of the framework that solves this example. Furthermore, I am not convinced that the proposed bounds can easily be concretized for an instantiation of the proposed framework, especially when considering deep networks; again, this concern could be alleviated by an example instantiation. Proposing something that is in principle more general and \"in principle cannot be worse\" but then not demonstrating that it actually is the case is, in my opinion, not enough.\n\n\n\nFinally, and this is not a deciding factor in my rating, the paper has quite some writing problems. On the first page alone, I found a lot of spelling and grammatical mistakes (see list at end) and the notation is sometimes confusing to me. For example, \"R\" is defined as a mapping of S x /R -> [0,1], but what is \"/R\" (curly R)? And then in (1) R is used with a single argument while in (2) not anymore. I can guess what is meant, but it feels inconsistent. In Theorem 1, I believe it should be \"the gap \\delta >= 0\" and not \"the gap g >= 0\", no?\n\nAbstract and 1st paragraph mistakes (unfortunately, no line numbers in this template!): \"researches\" -> \"research\", \"algorithm designing\" -> \"algorithm design\", \"task-specific prior knowledge about tasks.\" -> \"task-specific prior knowledge.\", \"not known in prior\" -> \"not known a priori\", \"Classical RL model environment...\" -> \"Classical RL models environment...\".\nAlso, quite some citations are missing the year, e.g. Schaul et al., Papavassiliou&Russell, ...", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "N/A", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "SklceeAvKS", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper introduces a new framework for reinforcement learning (named subjective reinforcement learning) which aims to resolve some of the inherent problems with RL in open environments.  The authors posit that one problem with RL in open settings is “data confusion”, which they describe as being situations where there are external factors (e.g. timeframe) that affect the action space differently.  They propose a “subjective reinforcement learning framework” which, as I understand it, can be described as an ensemble of traditional MDP’s subject to external factors k.  The paper evaluates how this subjective policy compares to traditional MDP’s in terms of theoretical bounds on performance. \n\nAlthough this learning framework seems like a potentially interesting future research direction, I tend to lean towards rejection for the reasons that: (1) the theoretical analysis is difficult to follow and there is sometimes a lack of clarity throughout the paper, (2) it isn’t very clear how easy this framework would be to implement aside from the theoretical guarantees and there aren’t any experiments or proofs of concept that would demonstrate the feasibility or practicality of the proposed framework in a real scenario, (3) the paper would benefit from more discussion of how their work differs from related techniques (like hierarchical RL, various forms of meta-learning, etc.).\n\nHere are some more concrete points:\n(1) The feasibility of this framework in a real-world scenario seems a bit hard to imagine and a strong use-case or proof-of-concept would be very helpful.  I liked that this paper provided some theoretical analysis for guiding the design of these systems.  However, it seems like these claims would also benefit from detailed empirical analysis and experiments.  Without empirical results, I feel a bit skeptical about how straightforward it would be to implement such a system or whether it would really be significantly useful in practice.  Similarly, though there are theory-based suggestions for how to optimally design such a system, it might be difficult to implement this system with optimal hyperparameters in a real-world use-case and the challenges in doing so are not really addressed.\n(2) In spite of claims that this method is able to be trained without domain knowledge, it seems like domain knowledge would still be necessary for things like determining what external information (K) is available and necessary, determining the appropriate N_S, etc.  It may be helpful for the authors to explain a bit more about how these things can be determined in a truly agnostic way.\n(3) It seems like there should be more discussion of the difference from hierarchical reinforcement learning.  In practice, hierarchical RL also can be used in similar ways to what’s described here.  As the authors point out, hierarchical RL is not necessarily splitting into submodels that handle data confusion problems, it seems like that is a constraint that could be added into a hierarchical framework’s design.\n(4)  I appreciate that the authors provide detailed theoretical analysis, but it can sometimes be confusing and difficult to follow.  I had some trouble evaluating the correctness of several of the proofs.  It may benefit from re-writing with more concise definitions of all of the variables and more clearly stated assumptions about observable information.  Here are some points of confusion for me:\n    - It seems like certain letters (eg. A or K or N_s vs N_d) are being overloaded as variable names with different fonts.  I realize that this is somewhat unavoidable, but I would recommend that the authors try to disentangle the naming a bit for improved clarity.\n    - In theorem 1, you stated \"the number of all possible data samples tends to infinity when the total number of samples N_d approaches infinity\".  This proposition seemed confusingly worded to me.  Maybe I am misunderstanding the wording, but it seems like possibly a tautology?\n    - I’m not sure I understand what is meant by “fake subjective policies” in Theorem 1.  Could you explain what is meant by that and the intuition here a bit more?\n    - It’s still unclear to me how K (that is, the external information) is being collected at any given timeframe.  Are you assuming that the necessary types of external information corresponding to K (your examples are ‘state history, out-of-MDP task encodings, samples from related tasks, etc.’) have been pre-determined?  If so, how could the most-appropriate type of external information be chosen in a practical way?\n\nI also noticed a few (very minor) grammar errors that authors may want to fix, though they did not affect my review:\n- page 1: \"tasks in open environments poses difficulties\" --> \"tasks in open environments pose difficulties\"\n- page 1: \"Problem is that both...\" --> \"The problem is that both...\"\n- page 2: \"we propose a novel framework named as Subjective reinforcement\" --> \"we propose a novel framework named Subjective reinforcement\"\n- page 4: \"should contain no data confusion\" --> \"should not contain data confusions\"", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "N/A", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}, {"review_id": "SJlHqPKaKS", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper introduces the Subjective Reinforcement Learning framework to formalize the problem of using extra information to split large, nonstationary environments into separate, simple, stationary MDPs. First, the paper introduces and motivates the problem on a high level: the same state-action pairs may transit to different successive states and rewards because of nonstationary dynamics/reward function, or variance in the environment or tasks. This phenomenon is termed \"data confusion\". The paper then summarizes some related approaches to dealing with this phenomenon. Next, the paper introduces the subjective RL framework in detail in section 3:\n- Extra information (kappa) is needed to resolve the data confusion.\n- The \"subjectivity\" (h) is a function that maps the extra information to a vector of weights over \"subjective\" MDPs.\n- A policy is maintained for each subjective MDP, and the overall policy is the vector product of h and the vector of subjective policies.\nIn section 4, the paper presents 3 theorems arguing that using the subjective RL framework doesn't harm performance. The paper then gives brief guidelines for designing algorithms using the subjective RL framework before concluding.\n\nAt the present time I recommend rejecting the paper. It does not actually present a concrete solution method, instead simply giving brief guidelines for the reader to design algorithms by. The subjective RL framework unifies and subsumes several existing approaches, but I don't feel that in itself is a significant enough contribution to warrant publication. The theorems presented essentially argue that using the subjective RL framework does not harm performance, but there are no mentions of the computational costs involved with maintaining policies for each subjective MDP. In addition, it's not clear where the subjective MDPs come from.\nThe paper also had issues with clarity, including many grammatical errors.\n\nI think this paper tackles an important problem from an interesting point of view, but stops short of giving a concrete algorithm that can be implemented and tested. It seems like a good candidate for a workshop, which could be a good opportunity for discussion and feedback.\n\nGeneral comments:\n- The definition of rewards is confusing; script R was never defined.\n- \"minimize objective 1\" should be actually be \"maximize objective 1\"?\n- Rewards are usually defined over state-action pairs, not just states. Why choose this unconventional formulation for rewards?", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "N/A", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors propose a learning framework to reframe non-stationary MDPs as smaller stationary MDPs, thus hopefully addressing problems with contradictory or continually changing environments. A policy is learned for each sub-MDP, and the authors present theoretical guarantees that the reframing does not inhibit agent performance.

The reviewers discussed the paper and the authors' rebuttal. They were mainly concerned that the submission offered no practical implementation or demonstration of feasibility, and secondarily concerned that the paper was unclearly written and motivated. The authors' rebuttal did not resolve these issues.

My recommendation is to reject the submission and encourage the authors to develop an empirical validation of their method before resubmitting.
Decision: Reject