DATA-DRIVEN EVALUATION  OF TRAINING ACTION SPACE FOR REINFORCEMENT LEARNING

1 INTRODUCTION . A reinforcement learning ( RL ) agent learns how to map states to actions in order to maximize a long-term cumulative award signal in a given environment . Figure 1 shows the various artifacts of an RL algorithm . An RL problem is defined by a quartet of ( S , A , Pa , Ra ) , where S is a set of states or the state space ; A is a set of actions or the action space available to influence S ; Pa ( s , s ′ ) = Pr ( st+1 = s ′|st = s , at = a ) is the transition probability which is the probability that action a in state s at time t will lead to state s′ at time t+ 1 ; and finally , Ra ( s , s′ ) is the immediate reward signal received after transitioning from state s to state s′ , due to action a . An RL agent training involves recognizing an optimal policy function , π∗ : a← s , from a corpus of { ( si , ai ) } to maximize the long-term cumulative reward , ∑ Ra . The reward function , Ra , is defined ab initio for an efficient goal accomplishment . The transition probability or state-action mapping is defined by the environment dynamics . In many real-life use cases , the agent can not directly sense the effect of its actions on the environment . This is particularly true when the state-action relationship can not be modeled by either closed-form analytical expressions [ 26 , 29 , 58 ] or explicit rules as in the popular games such as Chess [ 28 ] , Go [ 47 , 46 ] , and Atari [ 34 ] . This challenge is well documented in the literature [ 51 , 57 ] . To address this challenge for RL model training [ 52 , 33 ] , simulation-based action models [ 50 ] play a pivotal role . The optimal choice of simulation parameters such as the training action space is a non-trivial challenge because of the curse of dimensionality [ 17 ] , non-linearity [ 11 ] , and non-uniform action set [ 25 ] . Training data valuation [ 22 , 10 ] and associated artisanal software engineering efforts constitute a large part of the machine learning ( ML ) life cycle ( or MLOps ) . Yet , most research and development efforts [ 36 ] focus on algorithms and infrastructure . Production-grade MLOps needs to handle data lifecycle management ( DLM ) [ 40 ] challenges including : fairness and bias in labeled datasets [ 16 ] , data quality [ 9 ] , limitations of benchmarks [ 42 ] , and reproducibility concerns [ 39 ] . For RL , the DLM challenges are further compounded due to complexities arising from non-linear state-action interactions , partiallyobservable processes , non-isometric action spaces [ 32 ] , and strong domain specificity [ 21 ] of action models . With the recent emphasis on ML explainability , traditional supervised learning and deep learning research communities are actively working on systematic data-driven frameworks [ 14 , 53 ] for training data valuation . We need equivalent frameworks for RL to streamline conflict-prone training action selection [ 8 , 37 , 18 ] . Depending on the agent-environment interactions , different training actions have different relative contributions to the RL agent performance . Some training actions are indispensable because of their unique positions in the parametric space . Other dispensable actions have different relative contributions to the reward function . Remarkably , a high-cardinality training action space in many cases leads to lower cumulative reward than a well-designed training action space . To the best of authors ’ knowledge , there is hardly any data-driven tool for the evaluation of training action space for RL . Such a tool leads to superior RL agent performance , lower model training and maintenance cost , and strong multi-disciplinary collaboration [ 35 ] . This paper proposes a shapley-inspired [ 54 ] algorithm to categorize and rank training action sets . It also assists in recognizing cut-off cardinality for the training action space to reduce unnecessary exploration and ensure polynomial time complexity . While Section 2 describes the algorithms for RL training action space evaluation , Section 3 illustrates the effectiveness of the algorithms in a specific case study . Section 4 discusses the relevant work . Finally , Section 5 presents a summary with possible future directions . 2 ALGORITHM . The proposed framework provides a shapley-inspired methodology [ 49 ] to efficiently filter out unnecessary training actions and discover a near-optimal action set . Figure 2 shows the framework consists of two algorithms and how these two algorithms interact with each other to design a nearoptimal RL model by training action space filtering and evaluation . The first algorithm computes cutoff cardinality for the training action space . The cut-off cardinality is defined as the minimium number of action points needed in a training action set to enable high-fidelity predictions within an acceptable computational complexity and an error bound . The second algorithm takes action points only above the cut-off cardinality and categorize action points into two classes : { dispensable , indispensable } and ranking dispensable training action points based on the corresponding cumulative rewards . Indispensable action points are absolute essential action points for the RL training to achieve the desired goal . On the other hand , dispensable action points are non-essential action points that can be discarded . However , the RL agent might perform sub-optimally without a disposable action point . By ranking disposable training action points based on the corresponding cumulative rewards , the relative contributions of different training action points are estimated . 2.1 CUT-OFF CARDINALITY COMPUTATION FOR TRAINING ACTION POINTS . RL deals with compilation of a quartet ( S , A , Pa , Ra ) and computation of an optimal policy , π∗ which maximizes the long-term cumulative reward . The search space for a shapley-inspired technique needs to span across the power set for the training action space which has exponential complexity with training action points . To circumvent this problem , the proposed algorithm , as discussed in Algorithm 1 , uses Monte Carlo method to compute the cut-off cardinality . The inputs to this algorithm include an RL algorithm , the power set for the training action space , the corresponding state space , the transition probability , an acceptable performance margin , and an acceptable number of iterations . The output is the cut-off cardinality . The algorithm uses an iterative procedure to train the RL agent with different numbers of training actions , starting from all available training actions to only one training action . If for a certain cardinality , all the cumulative rewards do not improve by within n iterations , then we set the status to failure , and assign the cut-off cardinality to that cardinality plus one . The cut-off cardinality signifies computational efficiency — a larger cut-off cadinality means lower computational overhead and vice versa . Algorithm 1 Monte Carlo method for cut-off cardinality computation for the training action space Input : The RL algorithm , R , the power set for the training action space , P ( A ) , the corresponding state space , P ( S|A ) , transition probability P , an acceptable performance margin , , and an acceptable number of iterations , n. Output : Cut-off cardinality , |A|cutoff for k ← |A| to 1 do repeat π∗ ← R ( S , A , Pa , Ra ) if the cumulative reward improves by with n iterations then status← success else status← failure end if until all necessary training action sets for a given cardinality is exhausted . if all statuses for a given cardinality , k , are failures then |A|cutoff ← k + 1 end if end for 2.2 CATEGORIZATION AND RANKING OF TRAINING ACTIONS . Algorithm 2 offers a principled procedure for training action selection . As for inputs , it takes the combinations of training state-action pairs above the cut-off cardinality , determined by Algorithm 1 , the RL algorithm , the reward function , the threshold condition , and the acceptable number of iterations . As for outputs , it predicts a categorization vector which classifies training action points into two classes : { dispensable , indispensable } and ranks dispensable training actions based on the corresponding cumulative reward . At its core , Algorithm 2 determines whether the policy determined from the given state-action pairs achieves the threshold condition within the acceptable number of iterations . The algorithm uncovers the indispensability of some training action sets for a given RL task . A training action set is deemed to be indispensable : iff upon removal of that training action point , the RL agent fails to accomplish the goal within a finite number of iterations [ 24 , 56 , 23 ] . All other action points are dispensable . The dispensable action points are rank listed in L based on the corresponding cumulative reward . Essentially , L informs the performance impact of different disposable action points on the RL agent . Algorithm 2 Data-driven categorization and ranking for training action space evaluation Input : All combinations of training state-action pairs upto cut-off cardinality : { ( si , ai ) } , the RL algorithm , R , the reward function , the threshold condition , and the acceptable number of iterations . Output : Categorization vector , V , which classifies training action points into two classes : { dispens- able , indispensable } . Rank order list of dispensable training actions based on the corresponding cumulative reward . repeat for < si , ai > do π∗ ← R ( S , A , Pa , Ra ) if Threshold condition is never satisfied within the acceptable number of iterations then V ( A− ai ) ← indispensable else V ( A− ai ) ← dispensable end if end for until all necessary training action sets are evaluated . Rank List , L ← dispensable actions sorted in order of the corresponding cumulative rewards 2.3 ACTION SHAPLEY . The proposed framework adopts Action Shapley which introduces a formulation for the problem of equitable training action valuation in reinforcement learning . Reinforcement learning has three key building blocks . The first building block is the training action space , A , the corresponding state space S , and a pre-assigned reward function , Ra . The second building block is a reinforcement learning algorithm , R , which is treated as a black box . It takes ( S , A , Pa , Ra ) and computes an optimal policy function , π∗ . The third building block is the performance measure or the cumulative sum of reward , ∑ Ra . The goal of the framework is to compute the valuation for each training action point , φi ( A , R , ∑ Ra ) as a function of three building blocks . A shapley technique is useful for action valuation because it satisfies nullity , symmetry , and linearity [ 45 ] . From game theory , φ can be expressed as shown in Equation 1 : φi = C ∑ S⊆A− { i } ∑ Ra ( S ∪ { i } ) − ∑ Ra ( S ) ( n−1 |S| ) ( 1 ) where , the sum is over all subsets of A not containing i and C is an arbitrary constant . We call φi the Action Shapley value of action ai . Equation 1 suggests Action Shapley computation is with exponential time complexity w.r.t . the training action points . This exponential computational complexity warrants the need for the cut-off cardinality discussed in Algorithm 1 .

Summary: The paper addresses the problem of action set selection, i.e. identifying which actions should be available to a RL agent, during training. The set of available actions can influence the RL agent's performance or even hinder it to reach its goal. A method to evaluate action sets is introduced and a case study is performed on a resource tuning example on cloud infrastructure.