Learning in Non-Cooperative Configurable Markov Decision Processes

The Configurable Markov Decision Process framework includes two entities : a Reinforcement Learning agent and a configurator that can modify some environmental parameters to improve the agent ’ s performance . This presupposes that the two actors have identical reward functions . What if the configurator does not have the same intentions as the agent ? This paper introduces the Non-Cooperative Configurable Markov Decision Process , a framework that allows modeling two ( possibly different ) reward functions for the configurator and the agent . Then , we consider an online learning problem , where the configurator has to find the best among a finite set of possible configurations . We propose two learning algorithms to minimize the configurator ’ s expected regret , which exploit the problem ’ s structure , depending on the agent ’ s feedback . While a naïve application of the UCB algorithm yields a regret that grows indefinitely over time , we show that our approach suffers only bounded regret . Furthermore , we empirically validate the performance of our algorithm in simulated domains . 1 Introduction . The standard Reinforcement Learning [ RL , 40 ] framework involves an agent whose objective is to maximize the reward collected during its interaction with the environment . However , there exist real-world scenarios in which the agent itself or an external supervisor ( configurator ) can partially modify the environment . In a car racing problem , for example , it is possible to modify the car setup to better suit the driver ’ s needs . Recently , the Configurable Markov Decision Processes [ Conf-MDPs , 29 ] were introduced to model these scenarios and exploit the configuration opportunities . Solving a Conf-MDP consists of simultaneously optimizing a set of environmental parameters and the agent ’ s policy to reach the maximum expected return . In many scenarios , however , the configurator does not know the agent ’ s reward , and their intentions are different , leading to new forms of interaction between the two actors . For instance , imagine we are the owner of a supermarket , and we have to arrange the products on the shelves . Our objective is to increase the company ’ s final profit ; on the other hand , a customer aims to spend the shortest time possible inside the supermarket and buy the indispensable products only . Since we do not know the customer reward function , the only possibility is to try different dispositions and observe the customers ’ reactions . What if we knew what buyers ⇤ Work done when Giorgia Ramponi was at Politecnico di Milano . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . are most interested in ? In this case , we can strategically decide how to position other products close to the popular ones to induce the customer in a more profitable behavior for the supermarket owner . In this paper , we model these scenarios introducing the Non-Cooperative Markov Decision Processes ( NConf-MDP ) . This novel framework handles the possibility of having different reward functions for the agent and the configurator . While Conf-MDP assumes that the configurator acts to help the agent to optimize its expected reward , an NConf-MDP , instead , allows modeling a wider set of situations , including the cases in which agent and configurator display a non-cooperative behavior . Obviously , this setting can not be addressed with straightforward application of the algorithms designed for cooperative Conf-MDP . In fact , if the configurator and the agent optimize separately different objectives , they might not converge to an equilibrium strategy [ 52 , 12 , 51 , 13 ] . In this novel setting , we consider an online learning problem , where the configurator has to select a configuration , within a finite set of possible configurations , in order to maximize its own return . This framework can be seen as a leader-follower game , in which the follower ( the agent ) is selfish and optimizes its own reward function , and the leader ( the configurator ) has to decide the best configuration , based on its reward . Clearly , to adapt its decisions , the configurator has to receive some form of feedback related to the agent ’ s behavior . We analyze two settings based on whether the configurator observes just the agent ’ s actions or , in addition , a noisy version of the agent ’ s reward . Contributions In this paper , we extend the Configurable Markov Decision Process setting to deal with situations where the configurator and the agent have different reward functions . We call this new framework the Non-Cooperative Markov Decision Process ( NConf-MDP , Section 3 ) . Then , we formalize the problem of finding the best environment configuration , according to the configurator ’ s reward , as a leader-follower game , in which the agent ( follower ) reacts to each presented configuration with its best response policy ( Section 4 ) . We provide a first algorithm , Action-feedback Optimistic Configuration Learning ( AfOCL ) , to tackle this problem under the assumption that the configurator observes the agent ’ s actions only ( Section 5.1 ) . We show AfOCL achieves finite expected regret , scaling linearly with the number of admissible configurations . As far as we know , this represents the first problem-dependent regret analysis in a Multi-Agent RL setting . Then , we introduce a second algorithm , Reward-feedback Optimistic Configuration Learning ( RfOCL ) , that assumes the availability of a noisy version of the agent ’ s reward , in addition to the agent ’ s actions ( Section 5.2 ) . We prove that , under suitable conditions , RfOCL further exploits the structure underlying the decision process , removing the dependence on the number of configurations . The analysis use novel ideas , combining the suboptimality gaps of the configurator with those of the agent . Finally , we provide an experimental evaluation on benchmark domains , inspired by scenarios that motivate the NConf-MDPs framework ( Section 7 ) . The proofs of the results presented in the paper are reported in Appendix B . A preliminary version of this work was presented at “ AAAI-21 Workshop on Reinforcement Learning in Games ” [ 36 ] . 2 Preliminaries . A finite-horizon Markov Decision Process [ MDP , 35 ] is a tuple M = ( S , A , p , µ , r , H ) where S is a finite state space ( S = |S| ) , A is a finite action space ( A = |A| ) , p : S ⇥A⇥ S ! [ 0 , 1 ] is the transition model , which defines the density p ( s0|s , a ) of state s0 2 S when taking action a 2 A in state s 2 S , µ : S ! [ 0 , 1 ] is the initial state distribution , r : S ! [ 0 , 1 ] is the reward function , and H 2 N 1 is the horizon . A stochastic decision rule ⇡h : S ⇥A ! [ 0 , 1 ] with h 2 [ H ] prescribes the probability ⇡h ( a|s ) of playing action a 2 A in state s 2 S. A stochastic policy ⇡ = ( ⇡ 1 , · · · , ⇡H ) 2 ⇧H is a sequence of decision rules , where ⇧H is the set of stochastic policies over the horizon H . A finite-horizon Configurable Markov Decision Process [ Conf-MDP , 29 ] is defined as CM = ( S , A , P , µ , r , H ) and extends the MDP considering a configuration space P instead a single transition model p. The Q-value of a policy ⇡ 2 ⇧H and configuration p 2 P is the expected sum of the rewards starting from ( s , a ) 2 S ⇥A at step h 2 [ H ] : Q⇡ , ph ( s , a ) = r ( s ) + Esh0⇠p , ⇡ '' HX h0=h+1 r ( sh0 ) |sh = s , ah = a # , denoting with Esh0⇠p , ⇡ the expectation w.r.t . the state distribution induced by ⇡ and p at step h0 . The value function is given by V ⇡ , ph ( s ) = Ea⇠⇡h ( ·|s ) [ Q ⇡ , p h ( s , a ) ] and the expected return is defined as V ⇡ , p = Es⇠µ [ V ⇡ , p 1 ( s ) ] . In a Conf-MDP the goal consists in finding a policy ⇡⇤ together with an environment configuration p⇤ so as to maximize the expected return , i.e. , ( ⇡⇤ , p⇤ ) 2 argmax⇡2⇧H , p2P V ⇡ , p . 3 Non-Cooperative Conf-MDPs . The definition of Conf-MDP allows modeling scenarios in which agent and configurator share the same objective , encoded in a single reward function r. In this section , we introduce an extension of this framework to account for the presence of a configurator having interests that might differ from those of the agent . Definition 3.1 . A Non-Cooperative Configurable Markov Decision Process ( NConf-MDP ) is defined by a tuple NCM = ( S , A , P , µ , rc , ro , H ) , where ( S , A , P , µ , H ) is a Conf-MDP without reward and rc , ro : S ! [ 0 , 1 ] are the configurator and agent ( opponent ) reward functions , respectively . Given a policy ⇡ 2 ⇧H and a configuration p 2 P , for every ( s , a ) 2 S ⇥A and h 2 [ H ] we define the configurator and agent Q-values as : Q⇡ , pc , h ( s , a ) = rc ( s ) + Esh0⇠p , ⇡ '' HX h0=h+1 rc ( sh0 ) |sh = s , ah = a # , Q⇡ , po , h ( s , a ) = ro ( s ) + Esh0⇠p , ⇡ '' HX h0=h+1 ro ( sh0 ) |sh = s , ah = a # . We denote with V ⇡ , pc , h ( s ) = Ea⇠⇡h ( s ) [ Q ⇡ , p c , h ( s , a ) ] and V ⇡ , p o , h = Ea⇠⇡h ( s ) [ Q ⇡ , p o , h ( s , a ) ] the value functions and with V ⇡ , pc = Es⇠µ [ V ⇡ , p c,1 ( s ) ] and V ⇡ , p o = Es⇠µ [ V ⇡ , p o,1 ( s ) ] the expected returns for the configurator and the agent respectively . 4 Problem Formulation . While for classical Conf-MDPs [ 29 , 27 ] a notion of optimality is straightforward as agent and configurator share the same objective , in an NConf-MDP , they can display possibly conflicting interests . We assume a sequential interaction between the configurator and the agent that resembles the leaderfollower protocol [ 10 , 6 , 34 , 38 ] . First , the configurator ( leader ) selects an environment configuration p 2 P , where P is a finite set made of M stochastic transition models P = { p 1 , . . . , pM } . Then the agent ( follower ) plays a policy chosen by a best response function f : P ! ⇧H , such that : f ( p ) 2 argmax⇡2⇧H V ⇡ , po . The solution concept that we use is the well-known Stackelberg equilibrium [ 43 , 15 , 30 , 32 , 19 ] . It captures the outcome in which the configurator ’ s transition model is optimal , under the assumption that the agent will always respond optimally [ 26 ] . However , this definition includes the possibility of ties , i.e. , situations in which multiple agent optimal policies exist , with possibly different performance for the configurator . Therefore , it is necessary to employ a tie-breaking rule , i.e. , a criterion to select one agent best response . Different tie-breaking rules lead to different Stackelberg equilibria , and the two prevailing solution concepts in the literature are the Strong Stackelberg Equilibrium ( SSE ) and the Weak Stackelberg Equilibrium ( WSE ) . A policy-transition model pair ( ⇡⇤ , p⇤ ) forms an SSE if ties are broken in favor of the configurator : p⇤ 2 argmax p2P V f S ( p ) , p c and ⇡ ⇤ : = fS ( p ) 2 argmax ⇡2f ( p ) V ⇡ , po . The WSE can be constructed by breaking the ties against the configurator . In the rest of the paper , we employ the concept of SSE ; however , every result can be applied to any deterministic tie-breaking rule . We call ⇡⇤p the application of the best response function f S to a transition model p. Notice that the goal of the configurator is well-defined , whenever deciding the function fS . From an online learning perspective , this goal is to minimize the expected regret : E [ Regret ( K ) ] = E X k2 [ K ] max p2P V ⇡p , pc V ⇡pk , pkc . ( 1 ) To lighten the notation , in the following , we will denote with ⇡i the agent ’ s best response policy to the configuration pi , i.e. , ⇡⇤pi and with V i the configurator expected returned attained with configuration pi and policy ⇡i , i.e. , V ⇡i , pic . Finally , we denote with V ⇤ = maxi2 [ M ] V i . Agent ’ s Feedback The configurator knows its reward rc , but it does not know the agent reward ro . At each episode k 2 [ K ] , the configurator selects a configuration pIk 2 P and observes a trajectory of H steps generated by the agent ’ s best response policy ⇡Ik . We study two types of feedback : • Action-feedback ( Af ) . The configurator observes the states and the actions played by the agent ( s 1 , a 1 , . . . , sH 1 , aH 1 , sH ) , where ah ⇠ ⇡Ik , h ( sh ) . • Reward-feedback ( Rf ) . The configurator observes the states , the actions played by the agent , and a noisy feedback of the agent reward function ( s 1 , er 1 , a 1 , . . . , sH 1 , erH 1 , aH 1 , sH , erH ) , where ah ⇠ ⇡Ik , h ( sh ) and erh is sampled from a distribution with mean ro ( s ) and support [ 0 , 1 ] .2 The Rf models situations in which the agent ’ s reward is known under uncertainty , or it is obtained in an approximate way through Inverse Reinforcement Learning [ 33 ] . Connections with Bandit Algorithms The online problem that we are facing can be seen as a stochastic multi-armed bandit [ 25 ] , in which the arms are configurations , and the configurator receives a random realization of its expected return at every episode . Thus , in principle , it can be solved by standard algorithms for bandit problems , such as UCB1 [ 1 ] . These algorithms are computationally less demanding than those we will present in the next sections . On the other hand , they suffer regret that grows logarithmically , i.e. , indefinitely , with the number of episodes . Indeed , they do not exploit either the information regarding the agent ’ s policy or the structure induced by the agent ’ s reward function . We will prove that , instead , the proposed algorithms , which use the problem structure , suffer bounded regret . Furthermore , our algorithms are combined with UCB1 confidence intervals , so their regret , at finite time , is never worse than the one of UCB1 .

Summary: The authors provide an algorithm for online learning in a non-cooperative configurable MDP. They propose a setting where a configurator can choose a transition model for the MDP from a finite set of models. The goal is then to observe the agent's behaviour (either the state-action sequences or a noisy reward) and to learn an optimal configuration that results in maximum configurator reward given that the agent behaves optimally with respect to its own reward function.