Learning in Non-Cooperative Configurable Markov Decision Processes

The Configurable Markov Decision Process framework includes two entities : a Reinforcement Learning agent and a configurator that can modify some environmental parameters to improve the agent ’ s performance . This presupposes that the two actors have identical reward functions . What if the configurator does not have the same intentions as the agent ? This paper introduces the Non-Cooperative Configurable Markov Decision Process , a framework that allows modeling two ( possibly different ) reward functions for the configurator and the agent . Then , we consider an online learning problem , where the configurator has to find the best among a finite set of possible configurations . We propose two learning algorithms to minimize the configurator ’ s expected regret , which exploit the problem ’ s structure , depending on the agent ’ s feedback . While a naïve application of the UCB algorithm yields a regret that grows indefinitely over time , we show that our approach suffers only bounded regret . Furthermore , we empirically validate the performance of our algorithm in simulated domains . 1 Introduction . The standard Reinforcement Learning [ RL , 40 ] framework involves an agent whose objective is to maximize the reward collected during its interaction with the environment . However , there exist real-world scenarios in which the agent itself or an external supervisor ( configurator ) can partially modify the environment . In a car racing problem , for example , it is possible to modify the car setup to better suit the driver ’ s needs . Recently , the Configurable Markov Decision Processes [ Conf-MDPs , 29 ] were introduced to model these scenarios and exploit the configuration opportunities . Solving a Conf-MDP consists of simultaneously optimizing a set of environmental parameters and the agent ’ s policy to reach the maximum expected return . In many scenarios , however , the configurator does not know the agent ’ s reward , and their intentions are different , leading to new forms of interaction between the two actors . For instance , imagine we are the owner of a supermarket , and we have to arrange the products on the shelves . Our objective is to increase the company ’ s final profit ; on the other hand , a customer aims to spend the shortest time possible inside the supermarket and buy the indispensable products only . Since we do not know the customer reward function , the only possibility is to try different dispositions and observe the customers ’ reactions . What if we knew what buyers ⇤ Work done when Giorgia Ramponi was at Politecnico di Milano . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . are most interested in ? In this case , we can strategically decide how to position other products close to the popular ones to induce the customer in a more profitable behavior for the supermarket owner . In this paper , we model these scenarios introducing the Non-Cooperative Markov Decision Processes ( NConf-MDP ) . This novel framework handles the possibility of having different reward functions for the agent and the configurator . While Conf-MDP assumes that the configurator acts to help the agent to optimize its expected reward , an NConf-MDP , instead , allows modeling a wider set of situations , including the cases in which agent and configurator display a non-cooperative behavior . Obviously , this setting can not be addressed with straightforward application of the algorithms designed for cooperative Conf-MDP . In fact , if the configurator and the agent optimize separately different objectives , they might not converge to an equilibrium strategy [ 52 , 12 , 51 , 13 ] . In this novel setting , we consider an online learning problem , where the configurator has to select a configuration , within a finite set of possible configurations , in order to maximize its own return . This framework can be seen as a leader-follower game , in which the follower ( the agent ) is selfish and optimizes its own reward function , and the leader ( the configurator ) has to decide the best configuration , based on its reward . Clearly , to adapt its decisions , the configurator has to receive some form of feedback related to the agent ’ s behavior . We analyze two settings based on whether the configurator observes just the agent ’ s actions or , in addition , a noisy version of the agent ’ s reward . Contributions In this paper , we extend the Configurable Markov Decision Process setting to deal with situations where the configurator and the agent have different reward functions . We call this new framework the Non-Cooperative Markov Decision Process ( NConf-MDP , Section 3 ) . Then , we formalize the problem of finding the best environment configuration , according to the configurator ’ s reward , as a leader-follower game , in which the agent ( follower ) reacts to each presented configuration with its best response policy ( Section 4 ) . We provide a first algorithm , Action-feedback Optimistic Configuration Learning ( AfOCL ) , to tackle this problem under the assumption that the configurator observes the agent ’ s actions only ( Section 5.1 ) . We show AfOCL achieves finite expected regret , scaling linearly with the number of admissible configurations . As far as we know , this represents the first problem-dependent regret analysis in a Multi-Agent RL setting . Then , we introduce a second algorithm , Reward-feedback Optimistic Configuration Learning ( RfOCL ) , that assumes the availability of a noisy version of the agent ’ s reward , in addition to the agent ’ s actions ( Section 5.2 ) . We prove that , under suitable conditions , RfOCL further exploits the structure underlying the decision process , removing the dependence on the number of configurations . The analysis use novel ideas , combining the suboptimality gaps of the configurator with those of the agent . Finally , we provide an experimental evaluation on benchmark domains , inspired by scenarios that motivate the NConf-MDPs framework ( Section 7 ) . The proofs of the results presented in the paper are reported in Appendix B . A preliminary version of this work was presented at “ AAAI-21 Workshop on Reinforcement Learning in Games ” [ 36 ] . 2 Preliminaries . A finite-horizon Markov Decision Process [ MDP , 35 ] is a tuple M = ( S , A , p , µ , r , H ) where S is a finite state space ( S = |S| ) , A is a finite action space ( A = |A| ) , p : S ⇥A⇥ S ! [ 0 , 1 ] is the transition model , which defines the density p ( s0|s , a ) of state s0 2 S when taking action a 2 A in state s 2 S , µ : S ! [ 0 , 1 ] is the initial state distribution , r : S ! [ 0 , 1 ] is the reward function , and H 2 N 1 is the horizon . A stochastic decision rule ⇡h : S ⇥A ! [ 0 , 1 ] with h 2 [ H ] prescribes the probability ⇡h ( a|s ) of playing action a 2 A in state s 2 S. A stochastic policy ⇡ = ( ⇡ 1 , · · · , ⇡H ) 2 ⇧H is a sequence of decision rules , where ⇧H is the set of stochastic policies over the horizon H . A finite-horizon Configurable Markov Decision Process [ Conf-MDP , 29 ] is defined as CM = ( S , A , P , µ , r , H ) and extends the MDP considering a configuration space P instead a single transition model p. The Q-value of a policy ⇡ 2 ⇧H and configuration p 2 P is the expected sum of the rewards starting from ( s , a ) 2 S ⇥A at step h 2 [ H ] : Q⇡ , ph ( s , a ) = r ( s ) + Esh0⇠p , ⇡ '' HX h0=h+1 r ( sh0 ) |sh = s , ah = a # , denoting with Esh0⇠p , ⇡ the expectation w.r.t . the state distribution induced by ⇡ and p at step h0 . The value function is given by V ⇡ , ph ( s ) = Ea⇠⇡h ( ·|s ) [ Q ⇡ , p h ( s , a ) ] and the expected return is defined as V ⇡ , p = Es⇠µ [ V ⇡ , p 1 ( s ) ] . In a Conf-MDP the goal consists in finding a policy ⇡⇤ together with an environment configuration p⇤ so as to maximize the expected return , i.e. , ( ⇡⇤ , p⇤ ) 2 argmax⇡2⇧H , p2P V ⇡ , p . 3 Non-Cooperative Conf-MDPs . The definition of Conf-MDP allows modeling scenarios in which agent and configurator share the same objective , encoded in a single reward function r. In this section , we introduce an extension of this framework to account for the presence of a configurator having interests that might differ from those of the agent . Definition 3.1 . A Non-Cooperative Configurable Markov Decision Process ( NConf-MDP ) is defined by a tuple NCM = ( S , A , P , µ , rc , ro , H ) , where ( S , A , P , µ , H ) is a Conf-MDP without reward and rc , ro : S ! [ 0 , 1 ] are the configurator and agent ( opponent ) reward functions , respectively . Given a policy ⇡ 2 ⇧H and a configuration p 2 P , for every ( s , a ) 2 S ⇥A and h 2 [ H ] we define the configurator and agent Q-values as : Q⇡ , pc , h ( s , a ) = rc ( s ) + Esh0⇠p , ⇡ '' HX h0=h+1 rc ( sh0 ) |sh = s , ah = a # , Q⇡ , po , h ( s , a ) = ro ( s ) + Esh0⇠p , ⇡ '' HX h0=h+1 ro ( sh0 ) |sh = s , ah = a # . We denote with V ⇡ , pc , h ( s ) = Ea⇠⇡h ( s ) [ Q ⇡ , p c , h ( s , a ) ] and V ⇡ , p o , h = Ea⇠⇡h ( s ) [ Q ⇡ , p o , h ( s , a ) ] the value functions and with V ⇡ , pc = Es⇠µ [ V ⇡ , p c,1 ( s ) ] and V ⇡ , p o = Es⇠µ [ V ⇡ , p o,1 ( s ) ] the expected returns for the configurator and the agent respectively . 4 Problem Formulation . While for classical Conf-MDPs [ 29 , 27 ] a notion of optimality is straightforward as agent and configurator share the same objective , in an NConf-MDP , they can display possibly conflicting interests . We assume a sequential interaction between the configurator and the agent that resembles the leaderfollower protocol [ 10 , 6 , 34 , 38 ] . First , the configurator ( leader ) selects an environment configuration p 2 P , where P is a finite set made of M stochastic transition models P = { p 1 , . . . , pM } . Then the agent ( follower ) plays a policy chosen by a best response function f : P ! ⇧H , such that : f ( p ) 2 argmax⇡2⇧H V ⇡ , po . The solution concept that we use is the well-known Stackelberg equilibrium [ 43 , 15 , 30 , 32 , 19 ] . It captures the outcome in which the configurator ’ s transition model is optimal , under the assumption that the agent will always respond optimally [ 26 ] . However , this definition includes the possibility of ties , i.e. , situations in which multiple agent optimal policies exist , with possibly different performance for the configurator . Therefore , it is necessary to employ a tie-breaking rule , i.e. , a criterion to select one agent best response . Different tie-breaking rules lead to different Stackelberg equilibria , and the two prevailing solution concepts in the literature are the Strong Stackelberg Equilibrium ( SSE ) and the Weak Stackelberg Equilibrium ( WSE ) . A policy-transition model pair ( ⇡⇤ , p⇤ ) forms an SSE if ties are broken in favor of the configurator : p⇤ 2 argmax p2P V f S ( p ) , p c and ⇡ ⇤ : = fS ( p ) 2 argmax ⇡2f ( p ) V ⇡ , po . The WSE can be constructed by breaking the ties against the configurator . In the rest of the paper , we employ the concept of SSE ; however , every result can be applied to any deterministic tie-breaking rule . We call ⇡⇤p the application of the best response function f S to a transition model p. Notice that the goal of the configurator is well-defined , whenever deciding the function fS . From an online learning perspective , this goal is to minimize the expected regret : E [ Regret ( K ) ] = E X k2 [ K ] max p2P V ⇡p , pc V ⇡pk , pkc . ( 1 ) To lighten the notation , in the following , we will denote with ⇡i the agent ’ s best response policy to the configuration pi , i.e. , ⇡⇤pi and with V i the configurator expected returned attained with configuration pi and policy ⇡i , i.e. , V ⇡i , pic . Finally , we denote with V ⇤ = maxi2 [ M ] V i . Agent ’ s Feedback The configurator knows its reward rc , but it does not know the agent reward ro . At each episode k 2 [ K ] , the configurator selects a configuration pIk 2 P and observes a trajectory of H steps generated by the agent ’ s best response policy ⇡Ik . We study two types of feedback : • Action-feedback ( Af ) . The configurator observes the states and the actions played by the agent ( s 1 , a 1 , . . . , sH 1 , aH 1 , sH ) , where ah ⇠ ⇡Ik , h ( sh ) . • Reward-feedback ( Rf ) . The configurator observes the states , the actions played by the agent , and a noisy feedback of the agent reward function ( s 1 , er 1 , a 1 , . . . , sH 1 , erH 1 , aH 1 , sH , erH ) , where ah ⇠ ⇡Ik , h ( sh ) and erh is sampled from a distribution with mean ro ( s ) and support [ 0 , 1 ] .2 The Rf models situations in which the agent ’ s reward is known under uncertainty , or it is obtained in an approximate way through Inverse Reinforcement Learning [ 33 ] . Connections with Bandit Algorithms The online problem that we are facing can be seen as a stochastic multi-armed bandit [ 25 ] , in which the arms are configurations , and the configurator receives a random realization of its expected return at every episode . Thus , in principle , it can be solved by standard algorithms for bandit problems , such as UCB1 [ 1 ] . These algorithms are computationally less demanding than those we will present in the next sections . On the other hand , they suffer regret that grows logarithmically , i.e. , indefinitely , with the number of episodes . Indeed , they do not exploit either the information regarding the agent ’ s policy or the structure induced by the agent ’ s reward function . We will prove that , instead , the proposed algorithms , which use the problem structure , suffer bounded regret . Furthermore , our algorithms are combined with UCB1 confidence intervals , so their regret , at finite time , is never worse than the one of UCB1 .

Summary: The paper studies learning in configurable MDPs with two-agents, a reinforcement learning agent and a configurator that can modify the environment parameters. In contrast to prior work, the paper considers a setting in which the configurator and the agent can have misaligned objectives. The setting is formally modeled as a Stackelberg game, in which the configurator is the leader that  selects a transition model and the agent is the follower that plays the best response policy under this model. The goal is to design a learning algorithm for the leader with provable regret guarantees. The paper provides two learning algorithms for the configurator and formally analyzes their regret properties, which turn out to be better than those of a naive solution based on UCB1. The paper supports its theoretical claims using simulation-based experiments.