DATA-DRIVEN EVALUATION  OF TRAINING ACTION SPACE FOR REINFORCEMENT LEARNING

1 INTRODUCTION . A reinforcement learning ( RL ) agent learns how to map states to actions in order to maximize a long-term cumulative award signal in a given environment . Figure 1 shows the various artifacts of an RL algorithm . An RL problem is defined by a quartet of ( S , A , Pa , Ra ) , where S is a set of states or the state space ; A is a set of actions or the action space available to influence S ; Pa ( s , s ′ ) = Pr ( st+1 = s ′|st = s , at = a ) is the transition probability which is the probability that action a in state s at time t will lead to state s′ at time t+ 1 ; and finally , Ra ( s , s′ ) is the immediate reward signal received after transitioning from state s to state s′ , due to action a . An RL agent training involves recognizing an optimal policy function , π∗ : a← s , from a corpus of { ( si , ai ) } to maximize the long-term cumulative reward , ∑ Ra . The reward function , Ra , is defined ab initio for an efficient goal accomplishment . The transition probability or state-action mapping is defined by the environment dynamics . In many real-life use cases , the agent can not directly sense the effect of its actions on the environment . This is particularly true when the state-action relationship can not be modeled by either closed-form analytical expressions [ 26 , 29 , 58 ] or explicit rules as in the popular games such as Chess [ 28 ] , Go [ 47 , 46 ] , and Atari [ 34 ] . This challenge is well documented in the literature [ 51 , 57 ] . To address this challenge for RL model training [ 52 , 33 ] , simulation-based action models [ 50 ] play a pivotal role . The optimal choice of simulation parameters such as the training action space is a non-trivial challenge because of the curse of dimensionality [ 17 ] , non-linearity [ 11 ] , and non-uniform action set [ 25 ] . Training data valuation [ 22 , 10 ] and associated artisanal software engineering efforts constitute a large part of the machine learning ( ML ) life cycle ( or MLOps ) . Yet , most research and development efforts [ 36 ] focus on algorithms and infrastructure . Production-grade MLOps needs to handle data lifecycle management ( DLM ) [ 40 ] challenges including : fairness and bias in labeled datasets [ 16 ] , data quality [ 9 ] , limitations of benchmarks [ 42 ] , and reproducibility concerns [ 39 ] . For RL , the DLM challenges are further compounded due to complexities arising from non-linear state-action interactions , partiallyobservable processes , non-isometric action spaces [ 32 ] , and strong domain specificity [ 21 ] of action models . With the recent emphasis on ML explainability , traditional supervised learning and deep learning research communities are actively working on systematic data-driven frameworks [ 14 , 53 ] for training data valuation . We need equivalent frameworks for RL to streamline conflict-prone training action selection [ 8 , 37 , 18 ] . Depending on the agent-environment interactions , different training actions have different relative contributions to the RL agent performance . Some training actions are indispensable because of their unique positions in the parametric space . Other dispensable actions have different relative contributions to the reward function . Remarkably , a high-cardinality training action space in many cases leads to lower cumulative reward than a well-designed training action space . To the best of authors ’ knowledge , there is hardly any data-driven tool for the evaluation of training action space for RL . Such a tool leads to superior RL agent performance , lower model training and maintenance cost , and strong multi-disciplinary collaboration [ 35 ] . This paper proposes a shapley-inspired [ 54 ] algorithm to categorize and rank training action sets . It also assists in recognizing cut-off cardinality for the training action space to reduce unnecessary exploration and ensure polynomial time complexity . While Section 2 describes the algorithms for RL training action space evaluation , Section 3 illustrates the effectiveness of the algorithms in a specific case study . Section 4 discusses the relevant work . Finally , Section 5 presents a summary with possible future directions . 2 ALGORITHM . The proposed framework provides a shapley-inspired methodology [ 49 ] to efficiently filter out unnecessary training actions and discover a near-optimal action set . Figure 2 shows the framework consists of two algorithms and how these two algorithms interact with each other to design a nearoptimal RL model by training action space filtering and evaluation . The first algorithm computes cutoff cardinality for the training action space . The cut-off cardinality is defined as the minimium number of action points needed in a training action set to enable high-fidelity predictions within an acceptable computational complexity and an error bound . The second algorithm takes action points only above the cut-off cardinality and categorize action points into two classes : { dispensable , indispensable } and ranking dispensable training action points based on the corresponding cumulative rewards . Indispensable action points are absolute essential action points for the RL training to achieve the desired goal . On the other hand , dispensable action points are non-essential action points that can be discarded . However , the RL agent might perform sub-optimally without a disposable action point . By ranking disposable training action points based on the corresponding cumulative rewards , the relative contributions of different training action points are estimated . 2.1 CUT-OFF CARDINALITY COMPUTATION FOR TRAINING ACTION POINTS . RL deals with compilation of a quartet ( S , A , Pa , Ra ) and computation of an optimal policy , π∗ which maximizes the long-term cumulative reward . The search space for a shapley-inspired technique needs to span across the power set for the training action space which has exponential complexity with training action points . To circumvent this problem , the proposed algorithm , as discussed in Algorithm 1 , uses Monte Carlo method to compute the cut-off cardinality . The inputs to this algorithm include an RL algorithm , the power set for the training action space , the corresponding state space , the transition probability , an acceptable performance margin , and an acceptable number of iterations . The output is the cut-off cardinality . The algorithm uses an iterative procedure to train the RL agent with different numbers of training actions , starting from all available training actions to only one training action . If for a certain cardinality , all the cumulative rewards do not improve by within n iterations , then we set the status to failure , and assign the cut-off cardinality to that cardinality plus one . The cut-off cardinality signifies computational efficiency — a larger cut-off cadinality means lower computational overhead and vice versa . Algorithm 1 Monte Carlo method for cut-off cardinality computation for the training action space Input : The RL algorithm , R , the power set for the training action space , P ( A ) , the corresponding state space , P ( S|A ) , transition probability P , an acceptable performance margin , , and an acceptable number of iterations , n. Output : Cut-off cardinality , |A|cutoff for k ← |A| to 1 do repeat π∗ ← R ( S , A , Pa , Ra ) if the cumulative reward improves by with n iterations then status← success else status← failure end if until all necessary training action sets for a given cardinality is exhausted . if all statuses for a given cardinality , k , are failures then |A|cutoff ← k + 1 end if end for 2.2 CATEGORIZATION AND RANKING OF TRAINING ACTIONS . Algorithm 2 offers a principled procedure for training action selection . As for inputs , it takes the combinations of training state-action pairs above the cut-off cardinality , determined by Algorithm 1 , the RL algorithm , the reward function , the threshold condition , and the acceptable number of iterations . As for outputs , it predicts a categorization vector which classifies training action points into two classes : { dispensable , indispensable } and ranks dispensable training actions based on the corresponding cumulative reward . At its core , Algorithm 2 determines whether the policy determined from the given state-action pairs achieves the threshold condition within the acceptable number of iterations . The algorithm uncovers the indispensability of some training action sets for a given RL task . A training action set is deemed to be indispensable : iff upon removal of that training action point , the RL agent fails to accomplish the goal within a finite number of iterations [ 24 , 56 , 23 ] . All other action points are dispensable . The dispensable action points are rank listed in L based on the corresponding cumulative reward . Essentially , L informs the performance impact of different disposable action points on the RL agent . Algorithm 2 Data-driven categorization and ranking for training action space evaluation Input : All combinations of training state-action pairs upto cut-off cardinality : { ( si , ai ) } , the RL algorithm , R , the reward function , the threshold condition , and the acceptable number of iterations . Output : Categorization vector , V , which classifies training action points into two classes : { dispens- able , indispensable } . Rank order list of dispensable training actions based on the corresponding cumulative reward . repeat for < si , ai > do π∗ ← R ( S , A , Pa , Ra ) if Threshold condition is never satisfied within the acceptable number of iterations then V ( A− ai ) ← indispensable else V ( A− ai ) ← dispensable end if end for until all necessary training action sets are evaluated . Rank List , L ← dispensable actions sorted in order of the corresponding cumulative rewards 2.3 ACTION SHAPLEY . The proposed framework adopts Action Shapley which introduces a formulation for the problem of equitable training action valuation in reinforcement learning . Reinforcement learning has three key building blocks . The first building block is the training action space , A , the corresponding state space S , and a pre-assigned reward function , Ra . The second building block is a reinforcement learning algorithm , R , which is treated as a black box . It takes ( S , A , Pa , Ra ) and computes an optimal policy function , π∗ . The third building block is the performance measure or the cumulative sum of reward , ∑ Ra . The goal of the framework is to compute the valuation for each training action point , φi ( A , R , ∑ Ra ) as a function of three building blocks . A shapley technique is useful for action valuation because it satisfies nullity , symmetry , and linearity [ 45 ] . From game theory , φ can be expressed as shown in Equation 1 : φi = C ∑ S⊆A− { i } ∑ Ra ( S ∪ { i } ) − ∑ Ra ( S ) ( n−1 |S| ) ( 1 ) where , the sum is over all subsets of A not containing i and C is an arbitrary constant . We call φi the Action Shapley value of action ai . Equation 1 suggests Action Shapley computation is with exponential time complexity w.r.t . the training action points . This exponential computational complexity warrants the need for the cut-off cardinality discussed in Algorithm 1 .

Summary: The paper proposed a data-driven method for optimal action space selection in a reinforcement learning problem. Given a set of training state, action pair, the proposed approach first filters out the set of indispensable action set and then rank the other action set according to their cumulative reward values. To further improve the efficiency, a Monte Carlo sampling method is proposed for cut-off cardinality computation for action space. An action update rule is devised by computing optimal step size. Finally, a case study on a cloud environment is illustrated to select the optimal set of resources (#vCPUs and Memory size) to optimize the CPU utilization rate. The case study demonstrates that the Monte Carlo sampling based algorithm reduces action search space by 81% and then it creates a list of ranked action set. It is shown that a large action space does not necessarily lead to better performance for an RL agent.