Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This work considers the problem of providing interpretability for RL agents in input-driven environments. Specifically, this work proposes CrystalBox, a method for decomposing expected future returns into multiple components, when the reward itself is composed of multiple components. This work presents two approaches for providing the decomposed expected future returns: 1) A sampling-based approach that estimates the future returns with Monte Carlo simulation, assuming access to a simulator that enables multiple roll-outs from the current state; 2) A learning-based approach, which implements Monte Carlo policy evaluation.
strength_and_weaknesses: ## Strengths

This work's main strengths are that it studies an interesting and important problem of providing interpretability to RL policies, which are generally difficult to interpret, and that the presentation is quite clear.

## Weaknesses

- My main concern is that there is very limited technical contribution of interest to the ML community. Ultimately, the two proposed variants of CrystalBox leverage extremely well-understood classic techniques in fairly straightforward ways that can be found in RL textbooks, e.g., Sutton & Barto. I appreciate that these techniques are being used to decompose a value function into components, which is somewhat novel, but also still explored in prior work, e.g., Anderson et al., '19.
- The sampling-based method is also quite expensive and relies on fairly strong assumptions in general. Specifically, it requires many Monte Carlo roll-outs (which are expensive), and also requires the ability to reset to a given state, which is generally not possible across RL tasks, although it is true that in the considered input-driven tasks, there is access to a simulator that enables this. Yet, I assume that a downstream goal is to apply this in production systems, rather that merely on simulators, in which case, it would not be possible.
- Though this work claims that directly decomposing the rewards in the policy can "lead to significant performance degradation," it's not actually clear to me that this is the case. It seems quite possible that directly learning a separate value function for each reward component, or predicting different components as an auxiliary task could yield a better policy. At least, this requires some empirical investigation, and raises the question of why we need a separate framework to estimate these returns.
- Finally, a major concern is that it's difficult to assess the practical utility of the system from the current set of experiments. While the cdf error curves are interesting and show an ordering between the different proposed variants, it's unclear what amount of squared error is acceptable and useful for actually using this system for interpretability. Compounding this issue is the fact that even the computed squared errors are only estimates because the "ground-truth" values themselves are MC estimates. This work proposes reasonable ways that such a system might be used, but the key question remains: Is this system in fact useful for e.g., network engineers? A user study could potentially be useful here, although I do not have the expertise to understand how a network engineer might want to change the ABR or CC policy or its decisions based on the outputs of the interpretability system. However, determining how the interpretability system can beneficially impact future decision making is crucial -- if no decisions can practically be made from its outputs, then how is the interpretability helping?
- Minor comment: Section 4 states that CrystalBox consists of 2 components, but only describes 1 component.
clarity,_quality,_novelty_and_reproducibility: This work is extremely clear and seems to be reproducible. The quality is fairly high in that the proposals are technically sound, though the experimental evaluation could be tightened. The novelty is on the lower side, as the core techniques and ideas have been explored previously, though not in conjunction.
summary_of_the_review: Overall, I am mainly concerned about the relevance and the impact of this work for the ML community. Therefore, I cannot recommend acceptance.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This work performs reward decomposition with a focus on Input-Driven MDPs.
It presents CrystalBox, which creates post-hoc explanations for blackbox RL agents in input-driven environments.
This method is evaluated on two tasks.
strength_and_weaknesses: This work uses reward decomposition, which is a promising subtype of XRL methods.
It highlights specific questions that the user seeks to address and returns to these questions in evaluation!

The presented approaches require a simulator and do not learn the decomposed reward function from replay buffer data nor during training.
The authors highlight the separation of learning and explaining, but:
1. This separation is not a desired property, as it removes the causal nature of explanations: the learned Q is no longer the reason for the policy behavior.
   With this method, one can get explanations where the sum of Q components would suggest an alternate action (relative to what agent performs) or an action ordering based on one of the reward components may differ from one that would be obtained during training.
   Note that an estimator will exhibit more error in rare states (whether rare in environment or ones where agent makes an unexpected choice).
   These situations are exactly the ones where an operator would like to see an explanation. As a result, this post-hoc approximation will disproportionately affect meaningful queries.
   This aspect of the method is not evaluated.
2. The transition to a post-hoc approximation of Q via traces is not difficult nor non-obvious.

This work focuses on input-driven environments, which are defined as using traces and typically have decomposable reward.
However, it is unclear why there are complexities with P_z. The method assumes that a simulator is available during one portion, but then P_z is said to be inaccessible at a different point.
Specifically, it is stated that the "framework requires four inputs: a state, an action, a policy, and a simulation environment ... [and we assume] we have access to a simulation environment, the last input."
It is unclear how the lack of a P_z simulator fits with this characterization of the required inputs and the assumptions made.
Why is Z directly used for traces rather than gathering samples during training?

A post-processing approach is presented in Section 5.3.
This is an important step in tailoring this approach to the specific problem type considered.
Developing these ideas further would strengthen this work.
clarity,_quality,_novelty_and_reproducibility: My impression is that the presented method is not limited to Input-Driven MDPs.
This is a positive in that this method is more applicable, but this also means that this method is (1) framed in an unnecessarily narrow way and (2) not as novel as initially suggested.
The need for P_z is seemingly resolved in 4.2, but "the trajectory of states and actions" is used for each estimate.
If these trajectories are available, what is the issue with P_z access in 4.1?
My understanding is that 4.2 only differs in using a neural network to generalize.
Once the P_z difficulty is resolved, state-action-state pairs are obtained as with a typical MDP; the difference from typical MDPs disappears.
From this point, how do the traces play into the explainability method?
What is the benefit of splitting out Z from S in this setting?
What are the advantages of this approach over other reward decomposition methods?

The evaluation should be improved. Using fidelity does not convey the information necessary.
Beyond that, using Monte Carlo rollouts for the ground-truth is odd. The presented experiment is comparing the error between two estimators that both should converge.
Why are the Monte Carlo rollouts not used directly as a method?
If a ground truth is needed, then consider making an environment where one can be computed.
If seeking to convey that a good estimate can be reached efficiently, then results for different numbers of samples should be presented.
Fidelity would be useful to measure if learning Qs during training (and comparing to Qs obtained after training that are derived from many more rollouts).

For Section 5.2, claims are made about using CrystalBox for answering comparative questions, but the evaluations use fidelity here, too.
To support the use of CrystalBox in comparing actions, pairwise evaluations (best action vs second-best, for example) are needed.

Other Comments:
- Figure 2 is not legible as a scatterplot. Also, a quantitative metric should be reported rather than simply visualizing the distributions.
- Figure 4 is oriented in an odd way (better being "further left").
summary_of_the_review: This work focuses on Input-Driven MDPs, but it ultimately presents a method that is not limited to Input-Driven MDPs.
As a result, this work is effectively performing reward decomposition except with approximators learned after-the-fact.
This aspect risks generating explanations that differ from the agent's own estimates.
Finally, this method is evaluated only in its ability to obtain estimates of future reward components.
The ability to use these decompositions in answering specific questions is not evaluated.

I have read the author responses. I appreciate the clarifications, but key concerns remain: my recommendation is unchanged.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a method for explaining black-box RL models called CrystalBox, developed primarily for Systems Environments (such as adaptive bitrate streaming and congestion control). CrystalBox involves using a simulator and trained policy to rollout different actions and predict reward-component values and produce explanations by presenting humans with decomposed return values for a given action selection. The proposed method is evaluated on fidelity/faithfulness to ground-truth reward component values, and an example explanation is shown to demonstrate how the system would work.
strength_and_weaknesses: Strengths:
* The paper clearly defines the environment, problem setting, and explanations in the given context.
* The proposed method works well in the given settings with decomposable rewards, and discussions of extension to future work are readily addressed in the end of the paper.
* CrystalBox is evaluated for faithfulness to ground-truth reward values and achieves high correlation to ground-truth data. The system's explanations are sensible and feasibly useful.

Weaknesses:
* There is not an evaluation of how such an evaluation could be used with a human in the loop (e.g., [1, 2]). While fidelity/faithfulness is useful, it would be more meaningful to understand how these explanation are received by humans in relation to other explanations (e.g., counterfactuals without CrystalBox [3], or decision-tree approximations [4, 5] of the policy)
* The assumptions of CrystalBox are quite strict (e.g., requiring a simulator and decomposable reward function + labels for the Q function)
* There is no evaluation of runtime, though the method seems to potentially be quite slow (as there must be several simulated forward passes and reward-component predictions).
* CrystalBox fundamentally assumes that the policy is taking steps to maximize reward components in the given time-bound of the simulators rollouts, but this assumption may not always be valid (e.g., if the agent encounters out-of-distribution states and takes sub-optimal actions). For cases in which the policy may not be completely trustworthy, it seems that CrystalBox may also not be completely trustworthy.

[1] Silva, Andrew, Mariah Schrum, Erin Hedlund-Botti, Nakul Gopalan, and Matthew Gombolay. "Explainable Artificial Intelligence: Evaluating the Objective and Subjective Impacts of xAI on Human-Agent Interaction." International Journal of Human–Computer Interaction (2022): 1-15.

[2] Andrew Anderson, Jonathan Dodge, Amrita Sadarangani, Zoe Juozapaitis, Evan Newman, Jed Irvine, Souti Chattopadhyay, Matthew Olson, Alan Fern, and Margaret Burnett. 2020. Mental Models of Mere Mortals with Explanations of Reinforcement Learning. ACM Trans. Interact. Intell. Syst. 10, 2, Article 15 (June 2020), 37 pages. https://doi.org/10.1145/3366485

[3] Karimi, Amir-Hossein, Gilles Barthe, Borja Balle, and Isabel Valera. "Model-agnostic counterfactual explanations for consequential decisions." In International Conference on Artificial Intelligence and Statistics, pp. 895-905. PMLR, 2020.

[4] Bastani, Osbert, Yewen Pu, and Armando Solar-Lezama. "Verifiable reinforcement learning via policy extraction." Advances in neural information processing systems 31 (2018).

[5] Wu, Mike, Michael Hughes, Sonali Parbhoo, Maurizio Zazzi, Volker Roth, and Finale Doshi-Velez. "Beyond sparsity: Tree regularization of deep models for interpretability." In Proceedings of the AAAI conference on artificial intelligence, vol. 32, no. 1. 2018.
clarity,_quality,_novelty_and_reproducibility: Clarity:
* The system is clearly described and the paper reads well.

Quality:
* The work is of high quality, but the experiments are lacking in evaluation of the system for true interpretability/explainability in the loop with humans. The proposed method is sensible, but requires access to a high-fidelity simulator for the given problem, which may be a severely limiting assumption.
* The failure cases of CrystalBox are not discussed or evaluated. For example, how does the explanation handle out-of-distribution states? When the policy degrades, can explanations highlight this degradation or will they deceive humans into over-trusting the policy [2].

Originality:
* Counterfactuals are not a new idea, and related work on counterfactuals has fewer limiting assumptions than CrystalBox. However, CrystalBox provides explanations (including counterfactuals) through reward decomposition, and the reward decomposition comes from high-accuracy reward predictors, offering an advantage over prior work when reward decomposition is available.
summary_of_the_review: CrystalBox is an interesting idea and provides useful explanations, though the assumptions inherent to the method are difficult to overlook. The approach seems to be accurate in the domains in which it was evaluated, though there is no true evaluation with humans (who would be the actual end-users of explanations) and the failure cases of the explanations are not clearly explored. If the assumptions of having access to a simulator to generate counterfactuals could be relaxed, I would increase my score significantly, but at the moment the requirement of a simulator is too restrictive for the method to be more widely useful.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper addresses the problem of explainability of reinforcement learning based system controllers. This is motivated by the need for explanations in real-world applications of RL controllers. The proposed approach, CrystalBox, learns a separate explanation given a policy. The approach is evaluated in trace-based simulated real world settings of adaptive bitrate streaming (ABR) and congestion control (CC).

Initial reviews highlight the interesting, important and difficult nature of the problem addressed here. Giving useful explanations to users of such a system could genuinely advance real-world RL applications. In addition, an approach based on reward decomposition is considered promising.

A range of concerns and questions were raised in initial reviews, some of which were addressed during the rebuttal period. Novelty is considered limited, as the approach uses well-known techniques. Developing novel insights for new uses of well-known techniques could be a valuable contribution, but would require greater depth of insight than developed here. Reviewers are not convinced by the motivation for training an explanation model separately from the policy, especially given that such an explanation model could contradict the system's policy. Evaluation is not sufficient. Reviewers note that the provided metrics provide a valuable starting point, but the authors do not systematically show the value of generated explanations to human users. The discussion during the rebuttal phase did clarify questions about the nature of the trace-based simulator setup that is used here, and that is common in systems research.

Some of the raised concerns were not conclusively addressed during the rebuttal, with evaluation being a major example here. A revised version of the paper would need to systematically show how explanations generated by the proposed approach would provide value to users of such a system. Given the remaining concerns, the consensus is not to recommend acceptance at this stage.
justification_for_why_not_higher_score: The reviewers raise valid concerns that have not been fully addressed in the rebuttal. I consider the lack of human evaluation as a major limitation of the present work.
Decision: Reject