Subjective Reinforcement Learning for Open Complex Environments

1 INTRODUCTION . One important target of reinforcement learning ( RL ) is to realize an autonomous agent that can help people solve various kinds of tasks in open environments , where the agent ’ s observations and the environment ’ s dynamics is both diverse and unpredictable ( Wu et al. , 2019 ) . For example , a visionbased robot doing housework in all kinds of room-settings may observe various combinations of indoor scenes ; different tasks may require it to take different actions under the same observation ; the successor observation after performing an action is not predictable , because the shape of rooms , furniture settings and pets ’ movements , etc . are not known a priori . Classical RL models environment and tasks as a Markov Decision Process ( MDP ) . From this perspective , tasks in open environments pose difficulties that the state space S is large , the state transition dynamics P is nonstationary , and the reward function R may be contradictory ( Brodeur et al. , 2017 ; Xu et al. , 2018 ) . We argue that the key problem underlying these difficulties is data confusion , that is every sample the agent acquires from an open environment may reflect the properties of only a small part of the entire environment for a specific time period ( Kempka et al. , 2016 ) , thus data samples with similar current observations may have contradictory meanings under different circumstances , i.e . different transition dynamics , state ( or state-action ) values or action preferences . For this problem , current RL methods introduce out-of-MDP information such as task-encoding ( Zhu et al. , 2016 ) and goal-description ( Wang et al. , 2018 ) , or expand the state space into history space , in which every element is the concatenation of all information gained so far τt = s1 : t ( Farias et al. , 2007 ) . As for the former category , the quality and quantity of extra information are decided by the designer ( Schaul et al. , 2015 ) . As for the latter category , although the whole history is what we can use best to alleviate data confusion in theory , it increases the complexity to a maximum point and is hard to achieve in practice ; algorithms in this category thus either introduce regularization terms or transform the optimization target to easier ones . The problem is that both these categories depend on humans ’ understandings to certain tasks and lack theoretical guarantees ; consider that tasks in open environments are diverse and unpredictable , it is necessary to build a system that can dynamically fit tasks without prior knowledge . Intuitively , we humans also face data confusion problems in the real world , e.g . when going to office from home , different modes of transportation may include conflict behaviors e.g . walking in opposite directions . However , as long as we utilize arbitrary information and subjectively choose one of the modes , e.g . taking the bus , this task becomes concrete and without confusion ; also we may subjectively focus on more concise aspects of the problem e.g . selecting bus numbers , which make the tasks simpler . Inspired by this , we propose a novel framework named Subjective Reinforcement Learning Framework that models the environment and tasks as a set of simple MDPs with no data confusion problem , and a subjectivity model over them ; the corresponding subjective policy then includes two parts : acquiring subjectivity for each state with subjective samples , and choosing an action under the selected subjectivity . From this perspective , all aforementioned existing methods can be summarized as introducing subjective samples , i.e . state history or extra task information , beyond current observation st to form a subjectivity so that the resulting subjectivity-conditioned policy does not suffer from data confusion problem and thus have better performance . Therefore they can be recognized as instantiations of our subjective policy , and analysis based on our framework is general and meaningful for all these methods . In this paper , we discuss the conditions under which the subjective policy can achieve better results , and analyze in theory the relationship between overall performance and some key elements that characterize a subjective policy . The key elements are concrete and modifiable , and the analytical results guide us to design policy models that fit the need of problems in open complex environments without task-specific prior knowledge and with theoretically-guaranteed performance . To sum up , we think our contributions are as follows : 1 . To our best knowledge , we are the first to point out data confusion as a key problem to solving tasks in open complex environments ; 2 . We propose a novel framework named Subjective RL Framework and give theoretical analysis from a broader and systematic view ; 3 . We provide guidances for algorithm design based on our theoretical results . 2 RELATED WORK . Our subjective RL framework deals with data confusion problem facing tasks in open environments , which has also been implicitly considered in several RL fields : multi-task RL ( Zhu et al. , 2016 ) , meta-RL ( Rakelly et al. , 2019 ) , continuous adaptation ( Yu et al. , 2018 ) and hierarchical RL ( AlEmran , 2015 ) . We analyze their inherent defects—dependence on designers ’ knowledge for tasks— in the following three paragraphs . The field of multi-task RL considers an agent performing different tasks in possibly different environments , where data confusion is expected to happen across tasks . One direct category of methods equips the agent with extra task-specific information ( Riedmiller et al. , 2018 ) , e.g . a vector gi describing the goal of task Ti ; these methods requires the extra information to be available and sufficient enough to distinguish tasks from each other , which is guaranteed by designers ’ knowledge . Considering cases where suitable extra information is not available , various task-encoding methods have been proposed ( Sung et al. , 2017 ) , which generally extract a task feature vector from the whole trajectory τ and build the policy under condition of such feature vectors . These methods can be concluded as adding regularization terms into the complex end-to-end framework according to experts ’ prior knowledge , which includes the length ( Sorokin & Burtsev , 2019 ) and prior distribution ( Rakelly et al. , 2019 ) of the feature vector . Meta-RL methods , or learning-to-learn , turns to model the learning process instead of the original task ( Gupta et al. , 2018a ; b ) , which implicitly bypass the problem of data confusion . However , these methods still treat the learning process as a single-value function , e.g . the learning process of task embedding ( Lan et al. , 2019 ) and an exploration strategy ( Garcia & Thomas , 2019 ) ; thus their successes are based on the assumption that there are no data confusion in the meta learning phase . Similarly , continuous adaptation problems focuses on the transfer performance from source to target tasks and neglect the data confusion problem by assuming that the relationship between successive tasks can be correctly modeled by single-value mappings ( Al-Shedivat et al. , 2018 ) . Hierarchical RL is a broad field where methods aim to divide complex tasks into many different sub-tasks so that the overall data-efficiency can be improved ( Barto & Mahadevan , 2003 ; Al-Emran , 2015 ) . Although this idea of divide-and-conquer may result in the confusing data samples being divided into different sub-models and thus each sub-model handles no confusion , which is one typical target of subjectivity in our framework , the actual performance is determined by the design of the hierarchical structure , e.g . number of options ( Jain et al. , 2018 ) and capacity of both the top-level and low-level models , which are still designed according to prior knowledge and adjusted through empirical results . Instead , our framework takes all these hyper-parameters in consideration and can adjust to tasks without prior knowledge . The data confusion problem in open environments can also be recognized from the perspective of errors in function approximation . Existing research on function approximators in RL consider some specific types of model like linear approximations or neural networks , and analysis are given about the overall convergence or performance of certain RL algorithms ( Schoknecht , 2003 ; Achiam et al. , 2019 ; Papavassiliou & Russell , 1999 ) . However , approximators considered are end-to-end and do not consider the utilization of extra information . Different from these , our framework formalizes both original data samples and extra information used to construct subjectives , and so has the capacity to analyze the effect of extra information . 3 SUBJECTIVE REINFORCEMENT LEARNING FRAMEWORK . In this section we describe the data confusion problem of tasks in open environments . Then we give mathematical formulation of our subjectivity reinforcement learning framework that states the data confusion problem from a broader and systematic view . 3.1 PROBLEM STATEMENT . In traditional RL , the environment and tasks are modeled as 〈T , S , A , P , R , γ , ρ〉 , where T = { 0 , 1 , 2 , . . . } is the set of considered time steps , S = { st } the state space , A = { at } the action space , P ( st+1 |st , at ) the transition probabilities from each state-action pair to successive states , R : S × R → [ 0 , 1 ] the reward distribution over states that expresses the task , γ the discount factor that describes the relative importance between short-sight and far-sight rewards , and ρ the probabilistic distribution of initial observations . The goal of traditional RL is to find a policy πT : S → A that maximizes the following return : max πT∈ΠT G ( πT ) = ∑ s0∈S ρ ( s0 ) { ∞∑ t=1 γt ∫ r [ rR ( r |st ) dr ] } = ∑ s0∈S ρ ( s0 ) VπT ( s0 ) ( 1 ) where VπT ( s ) is the state-value function that satisfies the following Bellman equation : VπT ( s ) = ∑ a∈A πT ( a|s ) ∑ s′∈S P ( s′| s , a ) ∫ r R ( r |s′ ) [ r + γVπT ( s′ ) ] dr ( 2 ) RL agents interact with the environment to collect transitions 〈st , at , st+1 , rt+1〉 . Different RL algorithms utilize the collected transitions and equation ( 2 ) in different ways to construct functions that represent polices so as to maximize objective ( 1 ) . Generally they can be summarized into three categories : model-constructing , value-based , and direct action-mapping , which respectively process the transitions into forms of ( st , at ) → st+1 , st → V ( or ( st , at ) → Q for state-action value-based methods ) and st → at . In this paper we call such correspondences “ data samples ” to emphasize that they are the actual data used to construct function that represent the policy , and use z = ( x , y ) to generally represent the content and label of a data sample . Further , we denote the function as gT ( · ) , and the optimization problem becomes : min gT ND∑ i=1 L [ yi , gT ( xi ) ] ( 3 ) where D is the set of data samples and ND is the number of them ; L is a predefined loss function , e.g . l2-norm . Note that data samples reflect properties of environment , policy and tasks , and they depend on the sampling process in RL . Because gT ( · ) is a single-value mapping , normally the optimization problem ( 3 ) results in the expectation of data samples and alleviate the variances induced by the sampling processes . However , in open environments the observation space S is large , the transition dynamics P is nonstationary , and reward function R may be contradictory ; thus same state-action pairs may transit to contradictory successive states and rewards ; then the Bellman equation ( 2 ) may return different label y for the same data content x . Such differences come from the variances in environment or tasks , and should not be entangled through expectation . In this paper we name this problem data confusion . To eliminate this problem , extra information , which we denote as κ , need to be introduced , and the key problem is how to select κ and efficiently use it to eliminate data confusion at the minimum cost of simplicity . In the next subsection we introduce Subjective RL Framework to formalize this problem . As aforementioned , methods that expand the state space directly into the history space and try to construct the mapping from history to labels does not necessarily eliminates this problem in practice , so it remains meaningful to analyze the data confusion problem under the assumption that function approximators can only handle finite length of states . To make notations simple , here we focus on confusing mapping from single states and leave the history to the subjectivity part , without loss of generality .

Summary: The paper introduces the Subjective Reinforcement Learning framework to formalize the problem of using extra information to split large, nonstationary environments into separate, simple, stationary MDPs. First, the paper introduces and motivates the problem on a high level: the same state-action pairs may transit to different successive states and rewards because of nonstationary dynamics/reward function, or variance in the environment or tasks. This phenomenon is termed "data confusion". The paper then summarizes some related approaches to dealing with this phenomenon. Next, the paper introduces the subjective RL framework in detail in section 3: