CrystalBox: Efficient Model-Agnostic Explanations for Deep RL Controllers
Authors: Sagar Patel
Sangeetha Abdu Jyothi
Nina Narodytska
Venue: Submitted to ICLR 2023 (2023)
Keywords: explainability
reinforcement learning

Abstract: Practical adoption of Reinforcement Learning (RL) controllers is hindered by a lack of explainability. Particularly, in input-driven environments such as computer systems where the state dynamics are affected by external processes, explainability can serve as a key towards increased real-world deployment of RL controllers. In this work, we propose a novel framework, CrystalBox, for generating black-box post-hoc explanations for RL controllers in input-driven environments. CrystalBox is built on the principle of separation between policy learning and explanation computation. As the explanations are generated completely outside the training loop, CrystalBox is generalizable to a large family of input-driven RL controllers.To generate explanations, CrystalBox combines the natural decomposability of reward functions in systems environments with the explanatory power of decomposed returns. CrystalBox predicts these decomposed future returns  using on policy Q-function approximations. Our design leverages two complementary approaches for this computation: sampling- and learning-based methods.  We evaluate CrystalBox with RL controllers in real-world settings and demonstrate that it generates high-fidelity explanations.


Under review as a conference paper at ICLR 2023
CRYSTALBOX: EFFICIENT MODEL-AGNOSTIC EXPLANATIONS FOR DEEP RL CONTROLLERS
Anonymous authors Paper under double-blind review
ABSTRACT
Practical adoption of Reinforcement Learning (RL) controllers is hindered by lack of explainability. Particularly, in input-driven environments such as com- puter systems where the state dynamics are affected by external processes, plainability can serve as a key towards increased real-world deployment of RL controllers. In this work, ating black-box post-hoc explanations for RL controllers in input-driven environ- ments. CrystalBox is built on the principle of separation between policy Iearn- ing and explanation computation. As the explanations are generated completely outside the training loop, CrystalBox generalizable to a large family of input- driven RL controllers. To generate explanations, CrystalBox combines the natural decomposability of reward functions in systems environments with the explana- tory power of decomposed returns. CrystalBox predicts these decomposed future returns using on-policy Q-function approximations. Our design leverages two complementary approaches for this computation: sampling- and learning- baaninggbbssed methods. We evaluate CrystalBox with RI. controllers in real-world settings and demonstrate that it generates high-fidelity explanations.
propose novel framework, CrystalBox, for gener-
INTRODUCTION
Deep Reinforcement Learning (DRL) based solutions outperform manually designed heuristics in many computer systems and networking problems in lab settings. DRL agents have been successful in wide variety of areas, such as Adaptive Bitrate Streaming (Mao al. 2017). congestion con- trol (Jay et al. 2019). cluster scheduling (Mao et al.] 2019b)) and network traffic optimization (Chen al. 2018). However, because DRL agents choose their actions in a black-box manner, systems operators are reluctant to deploy them in real-world systems (Meng et al. 2020). Hence, similar to many ML algorithms, the lack explainability and interpretability of RL agents has triggered a quest for eXplainable Reinforcement Learning algorithms and techniques (XRL).
There are two major research directions in explainability of deep RL, The first line of work, which can be described feature- based methods, transfer established XAI results developed for supervised learning algorithms to deep RL settings. They focus on tailoring commonly used post-hoc explainers for classification and regression tasks, such saliency maps (Zahavy et al.) 2016 [yerr al. 2018
Greydanus et al. 2018 Puri et al.) 2019) or model distillation (Bastani et al. 2018 Verma et al. 2018 Zhang et al. (020). While such adapted techniques work well for some RL applications, is becoming apparent that these types of explanations are not sufficient to explain the behavior of complex agents in many real-world settings (Puiutta & Veith 2020; Madumal et al. 2020). For example, the inherent time- denendepndd characteristic of RL's decision making process can not be easily captured by feature-bast based methods. In the second line of work, XRL techniques help the user to understand the agent's dynamic behavior (Yau et al.] 2020 Cruz et al.] [0211 Juozapaitis (et al. 2019). The main underlying idea of this class of XRL methods is to reveal to the user how the agent "views the future' most algorithms compute an explanation using various forms of the agent's 's future beliefs like future rewards, goals, etc. For example, (Juozapaitis et al., (019)) proposed to modify DQN agent decompose its Q-function into interpretable components. (van der Waa ct al. 2018) introduce the concept of explaining two actions by explaining the differences between their future consequences.

---

Under review as a conference paper at ICLR 2023
In this work, we present CrystalBox, novel framework for extracting post-hoe black-box expla- nations. CrystalBox is designed to work with input-driven RL environments which is a rich class of RL environments, including systems or networking domains. Input-driven environments have two distinctive characteristics compared to standard RI. settings. These environments operate over input data traces (where trace can be sequence of network conditions measurements), and often have a decomposable reward function. Traces are difficult to model, and make both policy learning and explainability more challenging: learning sell-explainable policy can lead significant per- formance degradation. Hence, we build CrystalBox on the principle of separation between policy learning and explanation computation. Our next key observation is that thanks to the decomposable reward property, we can adapt the idea of decomposable returns (Anderson et al. 2019) as the basis for explanations. Below, we summarize our main contributions.
1. We propose the first post-hoc black box explanation framework for input-driven RL envi- ronments.
2. We demonstrate that decomposable return-based explanations (Anderson et al. 2019) are a good fit for input-driven RI. environments and propose a novel method for generating decomposed future returns using on-policy Q-function.
3. We design two complementary approaches to compute on-policy Q-function approxima- tions outside of the RL agent's training loop: sampling- and learning- based methods.
4. We implement CrystalBox and evaluate it on input-driven RL environments. We demon- strate that CrystalBox produces high-fidelity explanations in real-world settings.
SYSTEMS ENVIRONMENTS
Systems environments are a rich class of environments that represent dynamics in computer systems, which are fundamentally different from traditional RL environments. We provide an overview of our representative examples, Adaptive Bitrate Streaming and Congestion Control, and various other systems environments. For a thorough discussion on these environments, please see Appendix A.2 In this section, we highlight the characteristics that we leverage in our explainer, decomposability of reward functions and the notion of traces in these settings.
Adaptive Bitrate Streaming, (ABR) In adaptive video streaming, there are two communicating entities: a client, such as Netflix subseriber, who is streaming video over the Internet from server, such as a Netflix server. In video streaming, the video is typically divided into small seconds- long chunks and encoded, in advance, at various discrete bitrates. The goal of the ABR controller is to maximize the Quality of Experience (QoE) of the client by choosing the most suitable bitrate for the next video chunk based on the network conditions. The controller ensures that the client receives an uninterrupted high-quality video stream while minimizing abrupt changes in the video quality and stalling. QoE in this setting is typically defined as a linear combination that awards higher quality and penalizes both quality changes and stalling (Mok et al. 2011).
Note that network conditions are non-deterministic and constitute the main source of uncertainty in this setting. For example, the time taken to send chunk depends on the network throughput. These network conditions are defined as the trace ABR. More concretely, a trace is a sequence of network throughput values over time in ABR. Thus, environment in ABR is modeled using network traces that represent network conditions.
Congestion Control (CC) Congestion control protocols running on end-user devices are responsi- ble for adaptively determining the most suitable transmission rate for data transfers over shared, dynamic network. When a user transmits data at rate that the network cannot support, the user experience high queuing delays and packet losses. Deep RL-based solutions have shown superior performance in this setting (Jay et al.), 2019; Abbasloo et al., (020). Similar to the ABR environ- ment, traces in this setting also constitute a timeseries of throughput values. The reward function in congestion control incentivizes higher sending rates and penalizes delay and loss.
Other Systems Environments. Deep RL offers high performance in cluster scheduling (Mao) ctt al. 2019b), network planning (Zhu et al. 202I), network traffic engineering (Chen et al. 2018). database query optimization (Mareus et al. 2019) and several other systems control problems. A
2

---

Under review as a conference paper at ICLR 2023
common theme across these deep RL-based systems controllers is the decomposable reward fune- tion. The reason for that is that control in systems settings involves optimization across multiple objectives which are typically represented as the various reward components.
3 FORMALIZATION OF EXPLANATIONS
Preliminaries. In systems environments, consider Input-Driven Markov Decision Pro- cess (Mao et al., 2018), which a special class Markov Decision Processes where the envi- ronment transitions depend
outside process called trace. Formally, an Input-Driven MDP defined by the tuple (S, A, Z, P.. P: 7;) where is the set of states, A is the set of actions, Z the set of time-variant traces, is the reward function, and y is the discount. P. ( [st, aty ze) is the transition function of the environment that outputs the distribution of the next state, given the current state st, the action ae, and the value of the current trace 24. Finally, P2 |2e) ) the tran- sition function of the traces, which outputs the distribution over the next value of the trace given the current one. For example, ABR, Pa a model which determines how the Internet link between the viewer and the platform behaves over time.
Explainablity. We take the perspective systems operators. is important to gain an understand- ing of controller's decision-making process. Some of the common questions may be 'Why does the controller pick action A?', 'Why is action A better than action B?', and 'What are the mea- surable consequences of picking action A?'. Note that these questions span from explanations about a single action to explanations that require reasoning about multiple actions. To answer these questions, we need to define structure of explanations that is (a) succinct and (b) expressive.
Decomposed future returns (Anderson et al. 2019) is a category of explanations that satisfies these requirements. When each return component meaningful to the user and the number of return components is small, decomposed future returns provide concise and expressive explanation. This technique was demonstrated to be effective in learning self-explainable agent in game environ- ments (Juozapaitis et al. 2019). In systems environments, since the reward functions are naturally decomposable, the future returns are decomposable as well. Moreover, each component represents an aspect of performance or cost that is meaningful to the operator. Thus, decomposed future returns are an apt choice as the units of explanation in this setting. The core challenge then is to generate these decomposed future returns accurately and efficiently.
Future returns. To build our explanations we require an oracle to compute future returns of a given state St, action at, and policy л. We note that this problem is equivalent to computing the decomposed on-policy Q" (st, ax) function, which calculates the expected future returns for taking acting action a, state 8, and following the policy æ thereafter.
We propose to directly approximate this decomposed on-policy version of the Q-function, Q", out- side of the policy training process. This separation allows us build post-hoc explanations for any fixed policy π. even if π is non-deterministic or has continuous action space. We only require to be able to query this policy, without ever having modify or know internal structure.
Following @uozapaitiss et al. 2019), we define the explainability problem estimating the decom- posed components the on on-policy action-value function Q" (st, az) = Eesc Q: (st,4t), where C is the set of reward components. For example, in ABR, the components are Quality, Quality Change, and Stalling. Each component Q (st, at) computes the expected return that component for taking action in state 8, and following policy thereafter. It is formally defined as:
(s,, az) ro(st, az) + B OIT
b Al
(I)
We can obtain empirical samples of this function for all of the different components e by Monte Carlo rollouts. We refer to these Monte Carlo samples of the ground truth as Q" for convenience.
We define an explanation for a given state, action, and fixed policy as a tuple of return components:
X St, ar) 12", .... Qa CL.... C₁ C
(2)
3

---

Under review as a conference paper at ICLR 2023
state
CrystalBox
action
Sampling
Post- Processing (Optional)
decomposed returns: R,, Rg.
Explanation
policy
Learned Predictor
simulator
Figure Overview CrystalBox.
In general, one can consider more complex explanations that are functions over the return compo- nents. The function may depend concrete environments and user preferences.
CRYSTALBOX
In this section, we present our novel framework, CrystalBox, We start with high-level descrip- tion. CrystalBox consists two main components (Figure [T]. The first component the future returns predictor. It takes as inputs a state, an action, simulation environment, and policy. We present two ways to build this component, a sampling- bassdd approach ($ 4.122 and a learning- based approach ($ 4.2) The predictor produces expected returns that fed to optional post- processing module which generates casy-to-understand explanations. As an example, we present post-processing approach summarize the returns in Section 5.3
We discuss a few assumptions we make about available data. The framework requires four inputs: a state, an action, a policy, and simulation environment. The first two inputs, a state, and an action, form pair that we want to explain. The next input, policy, is treated as a black-box that we can only query. We never assume access the model of the environment or future information such as 81+1
The only assumption we make is that we have access to a simulation environment, the last input. Note that for most input-driven driven RL environments, these simulation environments are publicly available, e.g., ABR (Mao et al. 2017), CC (Jay et al., 2019), network scheduling (Mao et al. 2019a).
Let us highlight several features of CrystalBox. First, it does implement our design principle which the separation of policy learning and explanation computations. Second, it is flexible and al- lows the user to plug and play different environments and policies. To the best of our knowledge, CrystalBox is the first framework that provides such capabilities among the class of reward-based explainers. In the next section, present a few approaches to design future returns predictors.
4.1 SAMPLING-BASED APPROACH
Quality
Stalling
3.0
WER
2.5
Prodiction
20
Procictio
2.0
Melhod
Sempling
Learned Precictor Ground Truth
20000
States
States
Figure 2: Examples of future return predictions: Examples of Quality and Stalling return predictions by Naive Sampling and Learned approaches, plotted alongside samples of ground truth returns. We see that Naive sampling fails to capture stalls or quality drops.
Our first approach to designing a predictor sampling sampling-based: we estimate the individual compo- nents of (s ax) empirically by averaging over the outcomes of running simulations starting from 8, and taking the action a,. However, in practice, (= ($t, a,) cannot be computed exactly. Thus, to get empirical approximation of Q it is necessary to bound the infinite horizon by fixed length
4

---

Under review as a conference paper at ICLR 2023
Phase
Phase
CrystalBox
rollout
policy
Trajectory
(s, )
Learned Predictor
Predictor
simulator
(Baay Fih (5y091)2)
Preprocess
(5. a, (
Supervised Learning
Training Data
Figure 3: Overview of Learning-Band -Baee Approach.
Emaz- Enforcing this bounded horizon approximates the true O with a commonly used truncated version where the rewards after (maz are effectively assumed to be zero (Sutton & Barto 2018).
To approximate Q (8t. 0z). we need to sample potential futures of state 8, for tmaz steps. If we have a model of P- available, may simply use it to obtain samples of 2t, and in turn 8t- In this case, our sampling would be equivalent to the MC simulation method. However, in input-driven environments, P. is available (Mao al. 2018 2019a). Therefore, obtain potential futures of state we have no choice but to sample traces from Z. Evidently, it is not MC simulation anymore, as these potential futures are 'guessed' by our sampling procedure rather than given to us. Z can be sampled using different strategies and we discuss two possible strategies.
Before diving into sampling strategies, we consider how caapling-basede approach would work on ABR. Suppose we need an explanation for drop in bitrate in ABR. In this case, we roll out the policy the environment and consider set of states with a drop in bitrate for the next chunk. Our goal is to approximate Q (st, az) in these states using our sampling strategies. Note that also continue rollouts from this point onwards and compute Q" ($ of each of the future return components. We can use these to gain an initial understanding of how good our approximations are.
4.1.1 NAIVE SAMPLING
simple strategy for sampling involves uniformly random sampling. Given a state 8t, we randomly sample traces from Z to guess potential futures and compute approximations of Q Now we can compare computed approximations of Q (st, 0z) values with Q Figure 2 shows results of the comparison for the ABR example above focusing on two return components: quality and stalling. We observe that naive sampling- paasing predictions have low accuracy, especially for stalling predic- tions. To analyze the poor performance of the naive sampling approach, we took a close look at the sampling procedure. Recall that we randomly sample Z obtain potential futures, so our estimates depend on the distribution of Z. We observe that the distribution of traces is very unbalanced (see Fig. in Appendix [A.I]. The dominant traces do not sufficiently represent all relevant scenarios. One remedy to solve this issue is to make our sampling produce distribution aware, e.g. we could weight potential futures that we get from
4.1.2 DISTRIBUTION SAMPLING APPROACH
We propose an improved sampling-based method. As mentioned in the previous section, sampling-based approach can benefit from a smarter weighting of potential futures that we obtain from Z. To do so, take advantage the features of s; and condition our future values by calcu- lating P(ze| In practice, this probability distribution cannot be easily computed because of the complexity of the underlying environment. We propose method to approximate this conditioning. We assume that traces have underlying natural clustering, e.g. clusters may correspond to set of regions, clients, time, etc. Hence, we cluster all traces in and provide a procedure map the state to its closest cluster. Finally, we randomly sample trace within that cluster. In an experimental evaluation, we demonstrate that such conditioning does improve the campiing-based method.
4.2 LEARNING-BASED APPROACH
Our second approach is learning-based This approach is based the insight that future returns components of (st o,) form a function that can be directly parameterized and learned in a model- free manner by function approximator.
5

---

Under review as a conference paper at ICLR 2023
The proposed learning procedure consists of two phases (see Figure In the first phase, we take a policy and a simulation environment and collect trajectories by rolling out policy π in the simu- lation environment. Next, we pre-process the trajectories to create dataset of (s,, at, Q (St, a.)) tuples. In the second phase, we learn our predictor Qº,0 for each component, where is a set of neural network parameters. We emphasize that we employ deep supervised learning to find the final parameters 0 by iteratively updating the function approximator to better approximate the sam- ples of Q Qr, (St, at) t (,t ae) + aQQ (St, a4) - Qr (S,, at)). Here, Q2,0( Qe) is the prediction of the neural network, and Qº (st, ax) is a sample of the true value of the Q-function, calculated in the first phase by looking at the trajectory of states and actions after st and ag. As in the sampling-basee approach, we use the truncated version of the Q function.
This formulation is a special case of the function approximation version of the Monte Carlo Policy Evaluation algorithm (Silver 2013 Sutton & Barto 2018) for estimating Q3- In our case, Q is further broken down into smaller return components Qg.c that can be added up to the original value. Therefore, the standard proof of correctness of the Monte Carlo Policy Evaluation applies. Thus, our method converges to the true Q² function and captures how the policy performs.
4.3 QUALITY OF EXPLANATIONS
Next, we discuss evaluation metries for explanations. First, briefly overview commonly used evaluation criteria for explanations: the fidelity metric. In standard explainability workflow, an explainer takes as input a complex function f() and produces an interpretable approximation g(x) as output. For example, g(z) can be a decision tree that explains a neural network fx)) To measure the quality of the approximation, the fidelity metric FD (=) - g(z))l, x E DD measures how closely the approximation follows the original function under an input region of interest D.
Let us consider how these evaluation criteria are applied to our RL settings to evaluate CrystalBox explanations. It turned out that such translation is rather direct. As above, we have the complex function 2. one per each component e (defined Section CrystalBox outputs approximation, i.e. a predictor (Qed(Q"), that also serves an explanation. Hence, the fidelity metric is defined as a norm between a complex function and its approximation:
FD. ||Q% - Pred(Q V.
(3)
In our experiments, we use L2 norm. However, there is one distinction to discuss. Unlike standard settings, º is neither explicitly given to us as input nor can be efficiently extracted in any realistic environment, e.g. systems environments described in Section 2 Hence, the best we can do is to obtain estimates of Q using Monte Carlo rollouts.
EVALUATION
We present an experimental evaluation of CrystalBox that consists of two parts. First, we eval- uate the fidelity of the returns predictors described in Sections 4.1.1 (..2 Next, we focus on the explainability capabilities of CrystalBox.
We perform our experiments on two systems environments: ABR and CC. In ABR, the controller decides the video quality of an online stream show client. The controller receives a reward equal to the quality of experience of the client, measured as a weighted sum of three components: quality, quality change, and stalling. In CC, the controller manages the Internet traffic of connec- tion between a sender and receiver by adjusting the sending rate of the outgoing traffic from the sender. Here, the controller receives reward that is a weighted sum of three components: through- put, latency, and loss. For a detailed overview of these environments, see $
We consider the three return predictors we presented earlier: the naive-sampling based approach ($ 4.I.I. the distribution-awarr sampling approach ($ 4.1.2) and the learning-based approach ($ 4.2). Moreover, in some applications, we may have partial access to the policy. For example, we may have access to the embeddings of the states. In this case, it is important for explainability frameworks to take advantage of this additional knowledge to improve explanations. We demon- strate that our approach can do this without any major modifications. Therefore, in addition to the "black-bor box" setting, we consider a "gray-box" " variant of our learned predictor where our predictor reuses the embedding (sx) of state 8, from the policy.

---

Under review as a conference paper at ICLR 2023
Nalve Sampling
Distribution Aware Sampling
Learned Predktor Blackbox
.... Learned Predictor Graybox
Quality
Quality Change
Stalling
0.95
Better
0.93
0
0.53
0.75
0.04
0.0
Squared
Squared Frmer
Squared Finorr
Adaptive Bitrate Streaming
Throughput
Latency
Loss
0.8
Better
0.6
CDF
0.4
0.2
0.0
0.1
6.2
0.0
0.008
0.017
0.025
0.0
0.1 Scuared Error
0.3
Squared Error
Squered Error
(b) Congestion Control
Distribution Squared Error of different methods to Monte Carlo samples of the ground truth in Adaptive Bitrate and Congestion Control. Here, we focus on traces that can potentially experience stalling and discuss results on all traces in Figure (Appendix The Learned approach offers predictions with the lowest both environments. Note that the values of all the returns are scaled measured for error. The y-axis in results for ABR is adjusted due to the inherent tail-ended nature ABR's optimization.
Figure 4: Evaluation of CrystalBox for factual actions:
to the ground truth all three return components of
in the range zero to one before being
5.1 FIDELITY EVALUATION
We recall that decomposable future returns form the basis for CrystalBox explanations, so it is criti- cally important for us produce accurate predictions. To measure the quality of these predictions, we turn to the fidelity metric we introduced earlier ($ 4.3) and measure the eтror between the pre- dictions of different approaches and samples of the true Q function. We generate these samples by rolling out the policy on held-out set of traces Z' to ensure that these samples have not been seen by any of the approaches before.
In Figure 4
that the learned predictor outperforms both of the sampling approaches in pro- ducing high-fidelity predictions of all three of the return components both of the environments. The gray-box predictor narrowly beats the black-box approach at predicting the returns in Adaptive Bitrate Streaming, while achieving similar performance Congestion Control. Next, we analyze the performance of two sampling-based methods. We see that Distribution-Aware sampling pro- vides dramatic performance improvements over the standard sampling approach, especially, in CC. These results provide additional evidence confirm observation that exploiting the information in state s, can be vital to producing high-fiiclity return predictions.
5.2 EXPLAINABILITY ANALYSIS
Having established the fidelity our return predictors, we now turn to evaluating the explanatory power of these predictors. We focus our analysis the predictors' ability to answer contrastive questions such as "Why action instead action B?". Recent work (Doshi-Veled et al., 2017] van der Waa et al.) 2018 Mittelstadt al. 2019 Miller 2019) has highlighted the importance of such questions for human interpretability. Contrastive queries allow the user to differentiate between multiple possible actions take, e.g. for debugging purposes.

---

Under review as a conference paper at ICLR 2023
Naive Sampling
Distribution-kke Aware Sampling
Learndd Predictor Blackbox
Learned Predictor Graybox
Quality
Quality Change
1.00
Better
0.93
0.85
0.75
0.36
0.33
Squered Error
Squared Error
Squered Error
(a) Adaptive Bitrate Streaming: Counter Factual Actions
Throughppt
Latency
Loss
1.0
Better
0.5
0.4
0.2
0.0
0.008
0.C17
0.025
0.05
0.25
Squared
Squared Frror
Squered
(b) Congestion Control: Counter Factual Actions
Distribution of Squared Error to samples of the ground truth decomposed return for ABR and CC. We see that the Learned approach offers the most accurate predictions for both factual and counter-factual actions in all of the different return components. Note that values of returns are again sealed to be in the range zero to one before being measured for error. The y-axis results for ABR is adjusted due to the inherent tail-ended nature of ABR's optimization.
Figure 5: Evaluation of CrystalBox for Counter-factual actions:
CrystalBox supports answering this type question. Given two actions and B, we simply need to compute explanation for A and one for B. The intuition is that by looking at explanations the domain expert can gain insight into why one action, e.g. the action that the policy suggests, preferred over alternative action. For example, consider ABR environment. Suppose that are streaming a high-quality video. In state 81, the policy unexpectedly drops to sending medium- quality video (action A). An alternative action B is keep the same bitrate value which might be seen as a better action by the operator. To resolve this discrepancy, the user requests explanations for A and from CrystalBox that we show in the following table:
Explanation (future returns per component)
Action
quality
quality change stalling
A
16.23
0.85
0.0
B
16.31
2.11
0.41
Table 1: Example of predictions of returns
By comparing future returns, we see that action B is more preferable to action A in terms of video quality but it loses to A in terms of quality change and stalling ( we want these values to be as low as possible for QoE). These indications should convince the operator that action A is more reasonable in this situation. We would like to note that while our explanations operate in terms of very high- level notions for given environment, like video quality for ABR, however, we do expect the systems operator to have the basic domain knowledge to draw conclusions given our explanations.
To quantitatively measure the quality of counterfactual explanations we again use the same fidelity metric. However, there is difference between factual and counterfactual explanations that we need to take into consideration. Consider set of trajectories generated in Phase of the learning based approach (see Figure 3). They come from running given policy in the environment. Thus, only actions taken by the policy are recorded. However, envision range of cases, like policy
8

---

Under review as a conference paper at ICLR 2023
debugging, where the user might be interested in actions that policy does not frequently take. In this case, such counterfactual actions might be underrepresented in these trajectories leading to poor future returns estimates. To resolve this issue, we again exploited our separation principle between policy training and learning predictor. Namely, we augmented our dataset by generating additional trajectories where we add explorative action the beginning of the trajectory. We use the augmented dataset to train a single predictor. To clarify, the same predictor was used for factual (in Figure 4 and counterfactual explanations. In Figure 5 we observe the distribution of squared error of the different approaches to samples of the true Q function where all the actions a are counterfactual. We emphasize that counterfactual actions can be seen as difficult-to, ceafifeulltt--rediie to-predict scenarios for the reasons we just explained. First, we see that the learned predictor outperforms sampling- based approaches in almost all cases. Moreover, they provide high-fieelity return predictions for counterfactual actions. Another interesting conclusion is that we see the advantage of the gray- box over the black-box learned predictor in the same cases that were not that prominent for factual actions. Consider, for example, results for stalling in factuals in Figure 4a and counterfactuals in Figure 5a The gray-box learned predictor significantly outperforms all other predictors in the latter plot.
5.3 EVENT DETECTION
CrystalBox provides an optional post-processing capability on top of original future return- stuurn-based explanations. Consider again the example in Table [] In many applications, eeturn-based explana- tions can be sufficient. However, we believe that domain-specific post-processing can be very useful in practice. In some cases, it might not be obvious how to compare two contrastive actions in a state 81 from just their numerical returns. For example, if the stalling return value under action B is 0.1 then is unclear whether we should interpret it as a sufficient indication of stalling that not present under action A. Rather than making the user wonder about how to compare these future returns, can post-process them in a form of binary events, e.g. if a stalling happens or not in the near future.
We introduce the notion of threshold for demarcating the boundary between binary events along each return component. For example, in ABR environment, we use the 0.3 threshold for stalling. If the return value is greater than 0.3 then the explanation signals that a stalling occurs in the future. Thresholds can be determined based on a variety of factors such as risk tolerance, recovery cost for certain events, etc. In Appendix A.5 we show experimental evaluation of this technique, where we analyze two comparative actions: factual and counter factual from the same state, Overall, we show that all predictors are capable to detect large portion of events, while the learning- based predictors have better recall of events.
6 DISCUSSION AND FUTURE WORK
We start our discussion with an applicability scope of CrystalBox and, then, discuss its possible extensions. In this work, CrystalBox targets systems-related applications. However, input-driven environments are not limited to this class of applications. For example, there is rich class of game- based environments that are also input-driven (Mao et al., 2018). CrystalBox can be potentially extended to game-basee environments, however, such extension is non-trivial. For example, in our learned approach, we used Monte Carlo returns as estimates of the ground-truth QT function. However, in games where rewards be extremely delayed (only the end of the episode) or attributed to a large sequence of actions, these returns can be extremely high variance. Such high variance can lead to poor estimates of future returns, hence, low-fidelity explanations. To overcome
this variance, several strategies can be explored (Mao et al.) 2018 Hessel et al. 2018 Silver et al., 2017). We believe that it is interesting future direction on own.
One interesting direction to explore is whether we can use model distillation techniques to extract an interpretable model of future returns predictors. Another potential avenue to explore whether we can employ future return predictors during policy learning to facilitate understanding and debugging for human-in in-the-loop frameworks.
9

---

Under review as a conference paper at ICLR 2023
REFERENCES
Soheil Abbasloo, Chen- Yu Yen, and H Jonathan Chao. Classic meets modern: A pragmatic learning- based congestion control for the Internet. Proceedings of the Annual conference of the ACM Special Interest Group Data Communication on the applications, technologies, architectures, and protocols for computer communication, pp. 632-647, 2020.
Andrew Anderson, Jonathan Dodge, Amrita Sadarangani. Zoe Juozapaitis, Evan Newman, Jed Irvine, Souti Chattopadhyay, Alan Fern, and Margaret Burnett. Explaining reinforcement learning to mere mortals: An empirical study, arXiv preprint 19XX::990300900. 2019.
Osbert Bastani, Yewen Pu, and Armando extraction. Advances in neural information processing systems, 31, 2018.
Verifiable reinforcement learning via policy
Li Chen, Justinas Lingys, Kai Chen, and Feng Liu. Auto: Scaling deep reinforcement learning for datacenter-scale automatic traffic optimization. In Proceedings of the 2018 conference of the ACM special interest group on data communication, pp. 191-205, 2018.
Francisco Cruz, Richard Dazeley, Peter Vamplew, and Ithan Moreira. Explainable robotic systems: Understanding goal-driven actions a reinforcement learning scenario. Neural Computing and Applications, pp. 1-18, 2021.
Finale Doshi- Velez, Mason Kortz, Ryan Budish, Chris Bavitz, Sam Gershman, David O' Brien, Kate Scott, Stuart Schieber, James Waldo, David Weinberger, al. Accountability of ai under the law: The role of explanation. arXiv preprint arXiv: 134, 2017.
Samuel Greydanus, Anurag Koul, Jonathan Dodge, and Alan Fern. Visualizing and understanding atari agents. In International conference on machine learning, pp. 1792-1801. PMLR, 2018
Matteo Hessel, Joseph Modayil, Hado Van Hasselt, Tom Schaul, Georg Ostrovski, Will Dabney, Dan Horgan, Bilal Piot, Mohammad Azar, and David Silver. Rainbow: Combining improvements in deep reinforcement learning. In Thirty-second AAAI conference artificial intelligence, 2018.
Rahul Iyer, Yuezhang Li, Huao Li, Michael Lewis, Ramitha Sundar, and Katia Sycara. Transparency and explanation in deep reinforcement learning neural networks. In Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society, pp. 154-150, 2018.
Nathan Jay, Noga Rotman, Brighten Godfrey, Michael Schapira, and Aviv Tamar. A deep reinforce- ment learning perspective on internet congestion control. In International conference machine learning, pp. 3050- 300000099..00 PMLR, 2019.
Zoe Juozapaitis, Anurag Koul, Alan Fern, Martin Erwig. and Finale Doshi Velez. Explainable rein- forcement learning via reward decomposition. In IJCAI/ECAI Workshop on explainable artificial intelligence, 2019.
Adam Langley, Alistair Riddoch, Alyssa Wilk, Antonio Vicente, Charles Krasie, Dan Zhang, Fan Yang, Fedor Kouranov, Ian Swett, Janardhan Iyengar, et al. The quic transport protocol: Design and internet-scale deployment. In Proceedings of the conference of the ACM special interest group on data communication, pp. 183-196, 2017.
Prashan Madumal, Tim Miller, Liz Sonenberg, and Frank Vetere. Explainable reinforcement learn- ing through causal lens. In Proceedings of the AAAI conference artificial intelligence, vol- ume 34, pp. 2500,2500, 2020.
Hongzi Mao, Ravi Netravali, and Mohammad Alizadeh. Neural adaptive video streaming with pensieve. In Proceedings of the Conference of the ACM Special Interest Group on Data Commu- nication, pp. 197-210, 2017.
Hongzi Mao, Shaileshh Bojja Venkatakrishnan, Malte Schwarzkopf, and Mohammad Alizadeh. Variance reduction for reinforcement learning in input-driven environments. arXiv preprint arXiv:1807.022 2018.
10

---

Under review as a conference paper at ICLR 2023
Hongzi Mao, Parimarjan Negi. Akshay Narayan, Hanrui Wang, Jiacheng Yang. Haonan Wang, Ryan Marcus, Mehrdad Khani Shirkoohi, Songtao He, Vikram Nathan, et al. Park: An open platform for
computer systems. Advances in Neural Information Processing Systems,
32, 2019a.
Hongzi Mao, Malte Schwarzkopf, Shaileshh Bojja Venkatakrishnan, Zili Meng, and Mohammad Alizadeh. Learning scheduling algorithms for data processing clusters. In Proceedings of the ACM special interest group on data communication, pp. 270-288. 2019b.
Ryan Marcus, Parimarjan Negi, Hongzi Mao, Chi Zhang, Mohammad Alizadeh, Tim Kraska,
Olga Papaemmanouil, and Nesime Tatbul. Neo: A learned query optimizer. arXiv preprint arXiv:1904.037/1 03771111 2019.
Zili Meng, Minhu Wang, Jiasong Bai, Mingwei Xu, Hongzi Mao, and Hongxin Hu. Interpreting deep bearning-based networking systems. In Proceedings of the Anmual conference of the ACM Special Interest Group on Data Communication on the applications, technologies. architectures, and protocols for computer communication, pp. 171, 2020.
Tim Miller. Explanation in artificial intelligence: Insights from the social sciences. Artificial intel- ligence, 267:1-38, 2019.
Brent Mittelstadt, Chris Russell, and Sandra Wachter. Explaining explanations in ai. In Proceedings of the conference on fairness, accountability, and transparency, pp. 279-288, 2019.
Volodymyr Mnih, Koray Kavukcuoglu. David Silver, Alex Graves, Ioannis Antonoglou, Daan Wier- stra, and Martin Riedmiller. Playing atari with deep reinforcement learning. arXiv preprint arXiv: 13.56002, 2013.
Ricky KP Mok, Edmond WW Chan, and Rocky KC Chang. Measuring the quality of experience of http video streaming. In 12th IFIP/EEE International Symposium on Integrated Network Management (IM 2011) and Workshops, pp. 485-492 492. IEEE, 2011.
Tobias Pohlen, Bilal Piot, Todd Hester, Mohammad Gheshlaghi Azar, Dan Horgan, David Budden, Gabriel Maron, Hado Van Hasselt, John Quan, Mel Večerík, et al. Observe and look further: Achieving consistent performance on atari. arXiv preprint arXiv: 1593, 2018.
Erika Puiutta and Eric Veith. Explainable reinforcement learning: A survey. In International cross- domain conference for machine learning and knowledge extraction, pp. 77-95. Springer, 2020.
Nikaash Puri, Sukriti Verma, Piyush Gupta, Dhruv Kayastha, Shripad Deshmukh, Balaji Krishna- murthy, and Sameer Singh. Explain your move: Understanding agent actions using specific and relevant feature attribution. arXiv preprint a91iv: 12/91, 2019.
David Silver. Lectures on reinforcement learning. URL: http:://wwww.davddslleeeer uk/
teaching/ 2015.
David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Mare Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, et al. Mastering chess and shogi by self-play with a general reinforcement learning algorithm. arXiv preprint 171100/712.01//5 2017.
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
Jasper van der Waa, Jurriaan van Diggelen, Karel van den Bosch, and Mark Neerincx. Con- trastive explanations for reinforcement learning in terms of expected consequences. arXiv preprint arXiv:1807.08706, 2018
Abhinay Verma, Vijayaraghavan Murali, Rishabh Singh, Pushmeet Kohli, and Swarat Chaudhuri. Programmatically interpretable reinforcement learning. In International Conference on Machine Learning, pp. 5045-5054. PMLR, 2018.
Francis Y Yan, Hudson Ayers, Chenzhi Zhu, Sadjad Fouladi, James Hong, Keyi Zhang, Philip Levis, and Keith Winstein. Learning in situ: a randomized experiment in video streaming. In 17th USENIX Symposium on Networked Systems Design and Implementation (NSDI20) pp. 495-511, 2020.
11

---

Under review as a conference paper at ICLR 2023
Henman Yau, Chris Russell, and Simon Hadfield. What did you think would happen? explaining agent behaviour through intended outcomes. Advances in Neural Information Processing Systems, 33:18375-18386 183866 2020.
Tom Zahavy, Nir Ben-Zrihem, and Shie Mannor. Graying the black box: Understanding dqns. In International conference on machine learning, pp. 1899-1908. PMLR, 2016.
Hengzhe Zhang, Aimin Zhou, and Xin Lin. Interpretable policy derivation for reinforcement learn- ing based on evolutionary feature synthesis. Complex Intelligent Systems, 6(3):741-753, 2020.
Hang Zhu, Varun Gupta, Satyajeet Singh Ahuja, Yuandong Tian, Ying Zhang, and Xin Jin. Network planning with deep reinforcement learning. In Proceedings of the 2021 ACM SIGCOMM 2021 Conference, pp. 251-271, 2021.
12

---

Under review as a conference paper at ICLR 2023
3.5
Throughput (Mbp5)
12
Through huug (Mbрs)
Throughput (Mbps)
3.0
10
2.5
3
2.0
2
1.5
1.0
0.5
0.0
20
40
80
80
100
60
100
20
40
60
80
100
Time
(s)
12
Throughout (Mbps)
Throughput (Mbps)
Throughout (Mbps)
MMM M
6
0
20
40
80
80
100
20
40
60
80
100
20
40
60
80
100
Time
Time
Time
(6)
Throughput (Mbps)
M
Throughput (Pibps)
10
Throughput (Mibps
Wmm
M Um
M
MM MMM
6
4
2
0
100
100
20
60
80
100
Time
Time (5)
Time
(5)
3.0
Throughput (Mbpa)
20
Throughput (Mbps)
5
Trm mm
Throughput (Mbsp)
2.5
M
4
2.0
1.3
2
1.0
0.5
Time (s)
Time (8)
Time (6)
Figure 6: Examples Traces Adaptive Bitrate Streaming, In ABR, a trace the over-time throughput of the internet connection between a viewer and a streaming platform. In this figure, we present a visualization of a few of those traces for the first 100 seconds. Note that the y-axis is different on each plot due to inherent differences between traces.
A
APPENDIX
A.1 TRACES
In this section, visualize representative traces in Figure aadd Figure for ABR and CC applica- tions, respectively.
In ABR, trace is the over-time throughput of the internet connection between viewer and streaming platform. We obtain a representative set traces by analyzing the logged data of a public live-streaming platform (Yan al. 2020). In Figure we present visualization of few of
13

---

Under review as a conference paper at ICLR 2023
Effective Throughput (Mbps)
100
mmm
Effective Throughput (Mbps)
115
MMM
Effective Throughput (Mbps)
130
M
M
110
90
125
85
105
120
80
100
75
11.5
Time: (#)
Time (s)
Time (s)
200
Effective Throug hpuu (Mbps)
Effective Throughput (Mbps)
MW M. MMWM
Effective Throughput (Mbps)
150
M/
170
190
180
170
140
140
160
130
120
140
0
20
40
0
20
40
10
20
30
40
Time (s)
Time (s)
Time
(s)
Effective Throughput (Mbps)
230
Effective Throughput (Mbps)
260
Effective Throughput (Mbps)
300
280
225
255
260
220
240
215
220
210
200
240
180
20
10
20
Time (s)
Time
(s)
Time (s)
345
Effective Throughpus hput (Nbps)
Effective Throughput (Mbps)
390
Effective Throughout (Mbps)
450
340
M
440
335
370
430
330
360
420
350
320
410
340
315
400
330
10
20
10 15
Time (s)
Time (s)
Time (s)
Figure 7: Examples of Traces in Congestion Control. In CC, trace defined the internet network conditions between sender and receiver over time. These conditions can characterized by many different metrics such as throughput, latency, or loss. In this figure, we represent these traces by the sender's 's effective throughput over time. Note that both the x-axis and y-axis are different on each plot due to the inherent differences between traces.
those traces for the first 100 seconds. Note that the y-axis is different on each plot due to inherent differences between traces. However, even with naked eye, we can see that some traces are high- throughput traces, e.g. all traces in the third row, while other traces are slow throuhhruughput, e.g. the first plot in the second row. To further analyze these inherent differences, mean and coefficient of variance of througbput within each trace. In Figure we see that a majority of traces have mean throughput well above the requirements of the highest quality video. When
analyze the distribution of
14

---

Under review as a conference paper at ICLR 2023
Coufiisent of Vemimmoe Threctghppp
1.0
Avg. bitrate
highast
0.75
qualtty
0.6
0.6
0.5
0.4
0.4
0.2
0.2
0.25
0.0
0.0
Mean Throughput (MEps)
Confficient Variance Throughout
0.75
0.01
0.1
Throughout (MOps))
50
100
mean throughput in traces. Note that X-axis is log-scaled due to the large differences between all the clients of this server. Middle: distribution of coofficient of variance of the throughput within each trace. Right: The joint distribution of mean and cocfficient of variance of throughput. The traces logged over the course of couple of months from online public live-streaming Puffer (Yan et al. 2020). We find that majority of traces have mean throughput well above bitrate highest quality video. Only a small percentage traces represent poor network conditions such as low throughput, high variance, etc.
Figure 8: Distribution of Traces ABR. Left: distribution
17.5
3.0
20
15.0
2.5
15
22.5
Percers
20.0
300
20B
300
400
500
0.05
0.10
0.15
3.20 Letency
0.25
0.30
10
20
30
400
30
0,000
2,005
0.013
2015
0.00
Througrput (Pockets/s)
Max. Queue Size
Loss Rote
Figure 9: Distribution of Traces in CC: We analyze the distribution of traces in CC by analyzing the distri- bution by four key metrics: throughput, latency. maximum queue size and loss. These traces are synthetically generated by sampling from range of values, similar to the technique employed observe that traces with especially poor network conditions such as high loss rate or high queuing delay are small number.
(Jay et al. 2019). We
analyze this jointly with the distribution of throughput coefficient of variance, see that majority of those traces also have smaller variance. Only small number of traces represent poor network conditions such as low throughput high throughput variance. These observations are consistent with a recent Google study (Langley et al. 2017) that showed that more than 93% of YouTube streams never come to stall.
In CC, a trace is the over-time network conditions between a sender and receiver. We obtain representative set of traces by following (Jay et al., 2019) and synthetically generating them by four key values: mean throughput, latency, queue size, and loss rate. In Figure 7 we demonstrate how these traces may look like from the sender's perspective by looking at the effective throughput over time. Similar to the traces in ABR, we can visually see that the traces can be greatly different from one another. Figure 9 the distribution isn't nearly unbalanced as is in ABR, there are still only small number of traces that have exceedingly harsh network conditions.
we analyze the effective distribution of these traces. We observe that while
15

---

Under review as a conference paper at ICLR 2023
A.2 ENVIRONMENTS IN THE SYSTEMS DOMAINS
In system domains, traditional environment simulators are not available due to the complexity of the underlying input-process 2. To circumvent this issue, the state-of-the-art training environments replay specific logged runs of the system called "traces". The training environment selects a specific input-trace 2 from the set of given traces, and generates the successive state St by simply looking up the logged value 2g to compute P(s az 2t). This replaying process circumvents the need to explicitly calculate or approximate the behavior of the trace P(z 2t-1). However, this replaying process is not available when the policy is deployed in the real world. In the real world, we do not
know the future value of the input-trace 24, nor do we have a way to approximate it using žt-1-
To summarize, we highlight the key differences between traditional simulators and "simulators" in input-driven environments.
Rollout mechanism during training: In traditional simulators, future states can be sam- pled. In trace-based simulators, it is not obvious how to obtain future states in similar manner because they depend on an underlying process for which we lack model. To over- come this issue, the -tate-of-the-art solution is to select a specific trace and replay it. This is a reasonable substitute for having a model of the underlying process and is efficient.
Complexity of rollouts: Rollouts can be expensive in traditional simulators. However, in trace-based simulators, given specific trace, trace replay in the environment is relatively inexpensive, we just replay this trace.
Rollout mechanism during evaluation: In traditional simulators, this mechanism is not different from the one during training. Therefore, Monte Carlo sampling of the future states is trivial. By contrast, in trace-based environments, the rollout mechanism from point cannot be used, because we are not given the future nor do we have a method to model it.
A.3 EXAMPLES OF FUTURE RETURN PREDICTIONS (ADDITIONAL FIGURES)
We visualize all three components of future returns in Figure 10 which is an extension of Figure
Method
Naive Sampling
Learned Predktoo
Ground Truth
Que Sty
Quality Change
Stalling
3.0
20
Prediction
Prediction
10.0
Prediction
2.0
1.0
5.0
2.5
0.5
0.0
0.0
States
States
States
Figure 10: Examples of future return predictions: We visualize the Quality Change component of the future return predictions we presented in Fig. 2 We present Quality and Stalling here again for clarity. Unlike the other two return components where Naive sampling achieves dramatically poor performance, we see that Naive Sampling can detect quality changes to a certain degree.
A.4 RUNTIME ANALYSIS
We visualize the latency of Sampling- based and Learning- based techniques to generate an explana- tion. In sampling-basee techniques, we empirically estimate the return components Q by rolling out the policy for Emaz steps under a number of sampled traces 21, R2, #3. During this rollout process, we repeatedly query the simulator and policy at each step. By contrast, in Learning-based techniques, we directly generate the return components by querying a neural network once. In Fig- ure we the impact of this reduced overhead: learning- based techniques offer drastically lower latency than sampling-basee based techniques, reducing the computation time from 50-250 ms to just 5-6 ms.
16

---

Under review as a conference paper at ICLR 2023
200
Runtime (ms)
150
Method Sampling
100
Learned
50
ABR-Sempling
CC Cc-Sampling
ABR-Learned Predictor
CC-Leadeed Predictor
Domain
Figure 11: Runtime Analysis: We visualize the amount of time taken by Sampling-basse -ased and Learning- Learning-based techniques to generate an explanation in hoth ABR and CC environments. We see that the Learning-hat hased techniques significantly reduce the compute time necessary generate an explanation, reducing by an order of magnitude on average.
A.5 EVENT DETECTION
We present evaluation of event- best- post-processing technique. We recall that the goal of the post-procsssing technique is to make an explanation easier for the operator to interpret. To do so, introduce notion of threshold convert numerical returns to event-based returns. For example, if the value of future return for quality change is greater than 0.2, we say that an event of quality change is going be seen in the near future.
To evaluate the effectiveness of this post-processing technique, we consider the scenario where the operator may want understand the impact of comparative actions in the same state. We select a trace 2, rollout the policy in that trace, and select state to focus on. Within that state, we simulate taking actions, Qe, and aeys and continue rolling out the policy using 2 under both the actions to obtain two Q vectors.
Our event-based post-processing allows to treat the explainer as event predictor model. Hence, we can evaluate it using standard metrics. The ground truth events are obtained from the two Q vectors using the same threshold. Note that threshold values are parameters this post-processing method. We manually tried out a few threshold values and choose one that balances recall and false positives among those we tried on the training set.
Figure 12 shows results for ABR (the first row) and CC (the second row). The first plot in each row shows results for factual and the second for counterfactual explanations. For ABR we used the following event threshold values: (quality below 0.55, quality change above 0.2, stalling above 0.3)- For CC we used the following event threshold values: values:(throuu (throughput below 0.4, latency above 051,, loss above 0.025). On the y-axis, we show the percentage of events that were correctly detected by each explanation generation method. For example, if there are 10 quality drop events and naive sampling detects 5 of them then the percentage value 50 as this method detects 50% of events. Consider ABR results first. For factual explanations, sampling sampling-based based methods are better in detecting large quality change events. However, learning-baa based methods are better detecting quality drop and long stall events. In fact, sampling-based method misses all long stall events. If we consider results for counterfacuuals we see that learning- based methods outperform sampling methods. We observe slightly different picture in CC environment. While the learned-basee approaches outperform all other at detecting high latency, all the predictors provide high performance in other scenarios.
Next, we compute the false positive rate for each predictor. Figure 13 shows these results for ABR and CC environments. By false-positive we mean a situation when a predictor signals that an event will happen but this is not the case w.r.t. Q" As can be seen from the plot, the false- calse-positive rate is low for all predictors overall. The only exception that stands out is the high loss events in CC. Note that learning-basse based methods mostly manage to avoid false positives on factual explanations in CC. However, they do have high false positive rate on counterfactuals.
17

---

Under review as a conference paper at ICLR 2023
Method
Sampling
Sampling
Blackbox
Prediter Graybox
100
100
80
80
Pecontagg
Better
Perontagee
60
40
20
20
Quality Drogs
Large Qual ty Changes Events
Long Stalls
Quality Drops
Lai Qual ty Changes Evants
Long Stalls
(a) ABR: Factual Actions
(b) ABR: Counter Factual Actions
100
100
80
80
Better
Percertage
Percentagee
60
20
20
Throughput Draps
High Letency Events
High Loss
Throughput Draps
High Letency Events
High Loss
(c) CC: Factual Actions
(d) CC: Counter Factual Actions
Figure 12: Event Detection: analyze efficacy of different predictors detecting events. We identify events happening by detecting if samples of the ground-truth return exceed a threshold. For a detailed discus- sion, see Section 5.3 We evaluate their efficacy by analyzing their recall under both factual and counterfactual actions.
Method
Nelve Sampling
Distrbuzionn Aware Sampling
Learned Predictor Bleckbox
Learned Predictor Greybox
100
80
80
Better
PerCnnaage
Aecanntage
60
60
40
20
Quality Drogs
Large Changes
Long Stalls
Quality
Large Quality Changes Events
Long Stalls
Evants
(a) ABR: Factual Actions
(b) ABR: Counter Factual Actions
100
100
Better
Purcankege
Butantage
60
60
40
40
20
Throughput Draps
High Letency
High Loss
Throughput Draps
High
High Loss
Events
Events
(c) CC: Factual Actions
(d) CC: Counter Factual Actions
Figure 13: False Positive Rates on Event Detections: The false positive rate of the different predictors under both factual and counterfactual actions. We consider predictions false positive when ground truth samples of returns not exceed the threshold, but the predictions exceed
18

---

Under review as a conference paper at ICLR 2023
A.6 FIDELITY EVALUATION (ADDITIONAL RESULTS FOR ABR)
We present our evaluation of CrystalBox explanations on all traces. Figure 14 shows our results. We can see that all predictors perform really well. The learned-basee based predictors do slightly outperform sampling-bant but the difference is not that prominent compared to results on traces that might experience stalling (see Figure 4ap. For high throughput traces, the optimal policy for the controller simple: send the highest bitrate. Therefore, all predictors do well on these traces.
Method
Neive Sampling
Distribution-Awwrr Sampling
Learned Predictor Blackbox
Learned Predictor Greybox
Quality
Quality Change
Stalling
1.00
0.95
Better
0.80
0.75
0,000
0,025 0.350 0.075 0.100
0.0
0.16
0.33
0.5
0.0
0.33
0.66
1.0
Squared Error
Squared Error
Squared Error
(a) Factual Actions
Quality
Quality Change
Stalling
0.95
Better
CDF
0.90
0.75
0.00
0.05
0.15
0.0
0.16
0.33
0.5
0.0
0.33
Squared Error
Squared Error
Squared Error
(b) Counter Factual Actions
Figure 14: Evaluation of CrystalBox in ABR across all traces. Distribution Squared Error to samples of the ground truth decomposed return predictions for all traces in ABR. We observe that the differences in distribution of error for all of the return predictors shrink, but the relative ordering remains the same. Learned approach offers the best predictions for both factual and counterfactual actions.
A.7 MONTE CARLO ROLLOUTS
We collect samples of the ground truth values of the decomposed future returns by rolling out the policy in simulation environment. That is, we let the policy interact with the environment under an offline set of traces Z, and observe sequences of the tuple (s, a, F). With these tuples, we can calculate the decomposed return ( a;) for each timestamp. However, for given episode, these states and returns can be highly correlated (Mnih et al.) 2013). Thus, to efficiently cover wide variety of scenarios, we do not consider the states and returns Q (st, at) after s for tmaz steps. Moreover, when attempting to collect samples for a counterfactual action a', we ensure the rewards and actions from timestamp onwards are not used in calculation for any state-action pair before (st, af). This strategy avoids adding any additional noise to samples of Q due to exploratory actions.
imaz is hyper-paramedt for each environment. In systems environments, usually observe an effect of each action within a short time horizon. For example, if controller drops bitrate, then the user experiences lower quality video in step. Therefore, it is only required to consider rollouts of a few steps to capture consequences of each action, so tmax equals to five is sufficient for our environments.
19

---

Under review as a conference paper at ICLR 2023
A.8 BEARNING-BASED APPROACH
Preprocessing. We employ Monte Carlo Rollouts to get samples of Q for training our learned predictor. By themselves, the return components can vary across multiple orders magnitude. Thus, similar to the standard reward clipping (Mnih et al. (013) and return normalization (Pohlen et al. 2018) techniques widely employed in Q-learning, we normalize the returns be in range [0, 1].
Neural Architecture Design. We design the neural architecture of our learned predictors to be compact and sample efficient. We employ shared layers that feed into separate fully connected 'tails" that then predict the return components. We model the samples of Qr as samples from a Gaussian distribution, and predict the parameters (mean and standard deviation) to this distribution in each tail. To learn to predict these parameters, we minimize the negative log likelihood loss of each sample Q
For the fully connected layers, we perform limited tuning to choose the units of these layers from (64, 128, 256, 512]. We found that a smaller number of units is enough both of our environments. We present visualization of our architectures in Figures 15 18
Learning Parameters. We learn our predictors in two stages. In the first stage, we train our net- work end-to-end. In the second stage, we freeze the shared weights in our network, and fine-tune our predictors with a smaller learning rate. We use an Adagrad optimizer, and experimented with learning rates from 6 le-4, with decay from 1e-10 1e-9. We tried batch sizes from (50, 64, 128, 256, 512}. We found that small batch sizes, learning rates and decay work best.
20

---

Under review as a conference paper at ICLR 2023
Quality Neturns: ol,,
Come
Conn
PC
Quality Change Returns: (u,ar"
Stalline Returns: 3 M(,,o))
Figure 15: Neural Architecture of the Black-box Learned Predictor in ABR.
Quality Returns: ((L
Quality Change Returns: 3V (j.
rC
TC
PC
Stalling (a.
Figure 16: Neural Architecture of the Gray-box Learned Predictor ABR.
Throuphput Returns: ()
FC
FC
Latancy Retums: NV (P)
Loss Returns: (
Figure 17: Neural Architecture of the Black-box Learned Predictor in CC.
Throuphput Returns: N(
FC
FC
Laterncy Returns: o)
Loss Returns: (().F))
concat
Figure 18: Neural Architecture of Gray-box Learned Predictor CC.
21


Reviews:
[{"review_id": "ylPHv8pcDZ", "reviewer": "Reviewer_NU4N", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a method for explaining black-box RL models called CrystalBox, developed primarily for Systems Environments (such as adaptive bitrate streaming and congestion control). CrystalBox involves using a simulator and trained policy to rollout different actions and predict reward-component values and produce explanations by presenting humans with decomposed return values for a given action selection. The proposed method is evaluated on fidelity/faithfulness to ground-truth reward component values, and an example explanation is shown to demonstrate how the system would work.", "summary_of_the_review": "CrystalBox is an interesting idea and provides useful explanations, though the assumptions inherent to the method are difficult to overlook. The approach seems to be accurate in the domains in which it was evaluated, though there is no true evaluation with humans (who would be the actual end-users of explanations) and the failure cases of the explanations are not clearly explored. If the assumptions of having access to a simulator to generate counterfactuals could be relaxed, I would increase my score significantly, but at the moment the requirement of a simulator is too restrictive for the method to be more widely useful.", "strength_and_weaknesses": "Strengths:\n* The paper clearly defines the environment, problem setting, and explanations in the given context.\n* The proposed method works well in the given settings with decomposable rewards, and discussions of extension to future work are readily addressed in the end of the paper.\n* CrystalBox is evaluated for faithfulness to ground-truth reward values and achieves high correlation to ground-truth data. The system's explanations are sensible and feasibly useful.\n\nWeaknesses:\n* There is not an evaluation of how such an evaluation could be used with a human in the loop (e.g., [1, 2]). While fidelity/faithfulness is useful, it would be more meaningful to understand how these explanation are received by humans in relation to other explanations (e.g., counterfactuals without CrystalBox [3], or decision-tree approximations [4, 5] of the policy)\n* The assumptions of CrystalBox are quite strict (e.g., requiring a simulator and decomposable reward function + labels for the Q function)\n* There is no evaluation of runtime, though the method seems to potentially be quite slow (as there must be several simulated forward passes and reward-component predictions).\n* CrystalBox fundamentally assumes that the policy is taking steps to maximize reward components in the given time-bound of the simulators rollouts, but this assumption may not always be valid (e.g., if the agent encounters out-of-distribution states and takes sub-optimal actions). For cases in which the policy may not be completely trustworthy, it seems that CrystalBox may also not be completely trustworthy.\n\n[1] Silva, Andrew, Mariah Schrum, Erin Hedlund-Botti, Nakul Gopalan, and Matthew Gombolay. \"Explainable Artificial Intelligence: Evaluating the Objective and Subjective Impacts of xAI on Human-Agent Interaction.\" International Journal of Human–Computer Interaction (2022): 1-15.\n\n[2] Andrew Anderson, Jonathan Dodge, Amrita Sadarangani, Zoe Juozapaitis, Evan Newman, Jed Irvine, Souti Chattopadhyay, Matthew Olson, Alan Fern, and Margaret Burnett. 2020. Mental Models of Mere Mortals with Explanations of Reinforcement Learning. ACM Trans. Interact. Intell. Syst. 10, 2, Article 15 (June 2020), 37 pages. https://doi.org/10.1145/3366485\n\n[3] Karimi, Amir-Hossein, Gilles Barthe, Borja Balle, and Isabel Valera. \"Model-agnostic counterfactual explanations for consequential decisions.\" In International Conference on Artificial Intelligence and Statistics, pp. 895-905. PMLR, 2020.\n\n[4] Bastani, Osbert, Yewen Pu, and Armando Solar-Lezama. \"Verifiable reinforcement learning via policy extraction.\" Advances in neural information processing systems 31 (2018).\n\n[5] Wu, Mike, Michael Hughes, Sonali Parbhoo, Maurizio Zazzi, Volker Roth, and Finale Doshi-Velez. \"Beyond sparsity: Tree regularization of deep models for interpretability.\" In Proceedings of the AAAI conference on artificial intelligence, vol. 32, no. 1. 2018.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "LbE_f9GqiFP", "reviewer": "Reviewer_24j8", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work performs reward decomposition with a focus on Input-Driven MDPs.\nIt presents CrystalBox, which creates post-hoc explanations for blackbox RL agents in input-driven environments.\nThis method is evaluated on two tasks.", "summary_of_the_review": "This work focuses on Input-Driven MDPs, but it ultimately presents a method that is not limited to Input-Driven MDPs.\nAs a result, this work is effectively performing reward decomposition except with approximators learned after-the-fact.\nThis aspect risks generating explanations that differ from the agent's own estimates.\nFinally, this method is evaluated only in its ability to obtain estimates of future reward components.\nThe ability to use these decompositions in answering specific questions is not evaluated.\n\nI have read the author responses. I appreciate the clarifications, but key concerns remain: my recommendation is unchanged. ", "strength_and_weaknesses": "This work uses reward decomposition, which is a promising subtype of XRL methods.\nIt highlights specific questions that the user seeks to address and returns to these questions in evaluation!\n\nThe presented approaches require a simulator and do not learn the decomposed reward function from replay buffer data nor during training.\nThe authors highlight the separation of learning and explaining, but:\n1. This separation is not a desired property, as it removes the causal nature of explanations: the learned Q is no longer the reason for the policy behavior.\n   With this method, one can get explanations where the sum of Q components would suggest an alternate action (relative to what agent performs) or an action ordering based on one of the reward components may differ from one that would be obtained during training.\n   Note that an estimator will exhibit more error in rare states (whether rare in environment or ones where agent makes an unexpected choice).\n   These situations are exactly the ones where an operator would like to see an explanation. As a result, this post-hoc approximation will disproportionately affect meaningful queries.\n   This aspect of the method is not evaluated.\n2. The transition to a post-hoc approximation of Q via traces is not difficult nor non-obvious.\n\nThis work focuses on input-driven environments, which are defined as using traces and typically have decomposable reward.\nHowever, it is unclear why there are complexities with P_z. The method assumes that a simulator is available during one portion, but then P_z is said to be inaccessible at a different point.\nSpecifically, it is stated that the \"framework requires four inputs: a state, an action, a policy, and a simulation environment ... [and we assume] we have access to a simulation environment, the last input.\"\nIt is unclear how the lack of a P_z simulator fits with this characterization of the required inputs and the assumptions made.\nWhy is Z directly used for traces rather than gathering samples during training?\n\nA post-processing approach is presented in Section 5.3.\nThis is an important step in tailoring this approach to the specific problem type considered.\nDeveloping these ideas further would strengthen this work.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "L_lVMHB4lZ", "reviewer": "Reviewer_iPCV", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work considers the problem of providing interpretability for RL agents in input-driven environments. Specifically, this work proposes CrystalBox, a method for decomposing expected future returns into multiple components, when the reward itself is composed of multiple components. This work presents two approaches for providing the decomposed expected future returns: 1) A sampling-based approach that estimates the future returns with Monte Carlo simulation, assuming access to a simulator that enables multiple roll-outs from the current state; 2) A learning-based approach, which implements Monte Carlo policy evaluation.", "summary_of_the_review": "Overall, I am mainly concerned about the relevance and the impact of this work for the ML community. Therefore, I cannot recommend acceptance.", "strength_and_weaknesses": "## Strengths\n\nThis work's main strengths are that it studies an interesting and important problem of providing interpretability to RL policies, which are generally difficult to interpret, and that the presentation is quite clear.\n\n## Weaknesses\n\n- My main concern is that there is very limited technical contribution of interest to the ML community. Ultimately, the two proposed variants of CrystalBox leverage extremely well-understood classic techniques in fairly straightforward ways that can be found in RL textbooks, e.g., Sutton & Barto. I appreciate that these techniques are being used to decompose a value function into components, which is somewhat novel, but also still explored in prior work, e.g., Anderson et al., '19.\n- The sampling-based method is also quite expensive and relies on fairly strong assumptions in general. Specifically, it requires many Monte Carlo roll-outs (which are expensive), and also requires the ability to reset to a given state, which is generally not possible across RL tasks, although it is true that in the considered input-driven tasks, there is access to a simulator that enables this. Yet, I assume that a downstream goal is to apply this in production systems, rather that merely on simulators, in which case, it would not be possible.\n- Though this work claims that directly decomposing the rewards in the policy can \"lead to significant performance degradation,\" it's not actually clear to me that this is the case. It seems quite possible that directly learning a separate value function for each reward component, or predicting different components as an auxiliary task could yield a better policy. At least, this requires some empirical investigation, and raises the question of why we need a separate framework to estimate these returns.\n- Finally, a major concern is that it's difficult to assess the practical utility of the system from the current set of experiments. While the cdf error curves are interesting and show an ordering between the different proposed variants, it's unclear what amount of squared error is acceptable and useful for actually using this system for interpretability. Compounding this issue is the fact that even the computed squared errors are only estimates because the \"ground-truth\" values themselves are MC estimates. This work proposes reasonable ways that such a system might be used, but the key question remains: Is this system in fact useful for e.g., network engineers? A user study could potentially be useful here, although I do not have the expertise to understand how a network engineer might want to change the ABR or CC policy or its decisions based on the outputs of the interpretability system. However, determining how the interpretability system can beneficially impact future decision making is crucial -- if no decisions can practically be made from its outputs, then how is the interpretability helping?\n- Minor comment: Section 4 states that CrystalBox consists of 2 components, but only describes 1 component.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject