Assessing the Interpretability of Programmatic Policies using Large Language Models

Authors: Zahra Bashir, Michael Bowling, Levi Lelis
Primary area: interpretability and explainable AI
Keywords: Programmatic Policies, Interpretability, Program Synthesis

Abstract:
Programmatic representations of policies for solving sequential decision-making problems often carry the promise of interpretability. However, previous work on programmatic policies has only presented anecdotal evidence of policy interpretability. The lack of systematic evaluations of policy interpretability can be attributed to user studies being time-consuming and costly. In this paper, we introduce the LLM-based INTerpretability (LINT) score, a simple and cost-effective metric that uses large-language models (LLMs) to assess the interpretability of programmatic policies. To compute the LINT score of a policy, an LLM generates a natural language description of the policy's behavior. This description is then passed to a second LLM, which attempts to reconstruct the policy from the natural language description. The LINT score measures the behavioral similarity between the original and reconstructed policies. We hypothesized that the LINT score of programmatic policies correlates with their actual interpretability, and evaluated this hypothesis in the domains of MicroRTS and Karel the Robot. Our evaluation relied on a technique from the static obfuscation literature and a user study, where people with various levels of programming proficiency evaluated the interpretability of the programmatic policies. The results of our experiments support our hypothesis. Specifically, the LINT score decreases as the level of obfuscation of the policies increases. The user study showed that LINT can correctly distinguish the ``degree of interpretability'' of programmatic policies generated by the existing algorithms. Our results suggest that LINT can be a helpful tool for advancing the research on interpretability of programmatic policies.

Decision: reject
Reviewer scores: 3, 5, 3, 6, 5

Reviews and discussion:
<thread>
creation date:
1730142193123

thread_type:
official_review

<thread_object>
creator: reviewer_deqy

<content>
summary:
The paper proposes the LLM-based Interpretability score as a method for evaluating the interpretability of programmatic policies. The approach leverages LLMs to predict human judgments of interpretability. Experiments are conducted in two domains, MicroRTS and Karel the Robot, with the goal of demonstrating LINT's effectiveness in assessing how easily humans can understand policies.

strengths:
- The idea of using LLMs to evaluate policy interpretability is creative and shows promise in leveraging existing language models for new applications.

- Interdisciplinary Relevance: The integration of LLMs into policy interpretability assessment is an exciting intersection of interpretability research and natural language processing, which could be beneficial to both communities.

- The paper is well-structured and the experiments are explained in detail, which makes it easy to follow the motivation and method behind LINT.

weaknesses:
- The LINT score's novelty would be more compelling if it were benchmarked against existing interpretability metrics. The absence of such comparisons leaves questions about its relative performance. I am confused about why the authors propose using LLMs for evaluation. What are the advantages compared to previous methods? I believe that LLMs are essentially black boxes, which could potentially increase the lack of explainability.

- The user studies are conducted with small sample sizes (n = 60 for MicroRTS, n = 33 for Karel). This limits the statistical power and generalizability of the findings.

- The experiments are confined to just two simple domains, which may not translate well to more complex or diverse real-world scenarios. Broader testing would help in validating the generalizability of LINT. 

- The paper does not adequately address how biases inherent in LLMs might affect the validity of LINT as an interpretability measure. A deeper analysis of LLM bias would help in evaluating the reliability of this method.

- LINT struggles to differentiate policies with small differences in interpretability. Enhancing this sensitivity is crucial for practical applications where subtle interpretability changes matter.

questions:
Please refer to weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730270116695

thread_type:
official_review

<thread_object>
creator: reviewer_gzii

<content>
summary:
This paper introduces the LINT score, a metric for measuring the interpretability of a programmatic reinforcement learning policy. The idea is based on the intuition that the more interpretable a program is, the easier it is to describe it in natural language. The authors first use an LLM to explain a program in natural language, then ask another LLM to reconstruct the program from the explanation, measuring the similarity between the two versions. Their framework also includes a verifier to ensure that the explanation does not provide line-by-line details for program reconstruction.

They evaluated their method on MicroRTS and Karel the Robot, demonstrating that the more obfuscated the code, the less similar the two versions of the program will be. They also conducted a user study to test whether the proposed metric aligns with human preference.

strengths:
Recent advancements in programmatic reinforcement learning (PRL) primarily focus on improving performance, despite interpretability being a foundational goal of PRL. This makes the motivation for measuring how interpretable a programmatic policy is well-founded, as a program’s interpretability can naturally vary between its language and underlying code. The approach of asking an LLM to explain and then reconstruct the program is both innovative and intuitively sound.

weaknesses:
**The problems of evaluating the results on Karel.** In the Karel the Robot environment, there should be ten tasks, including six from Karel and four from Karel-Hard. What is the reason for selecting the five tasks in the experimental results and not the others? I see you chose Seeder as one of the test beds from Karel-Hard, so why didn’t you include the other tasks as well? Another issue is that there is only one policy for each task, which seems insufficient for a comprehensive evaluation of the results. Additionally, the action metric appears unclear. In Karel the Robot, the action an agent performs depends on both the grid it occupies and the code blocks it is executing. In other words, the agent may perform different actions even when on the same grid. How do you define "state" in this context?

**The experimental evidence is hard to confirm.** In Table 3, the paper lists only the average results for all policies, without including the per-task results. Regarding the level of obfuscation, it appears that you did not specify how you inserted the useless code blocks. Was this done through a rule-based method or with human involvement? Additionally, what are the standards for levels 1 and 2? In Line 168, you mentioned that you only invoked the verifier once, so how can you claim that "all approved explanations satisfied the constraints"?

questions:
- As you mentioned in Section 4, the time spent pondering may affect the results. So what is the time limit for users to answer the questions?
- Given that users rate program interpretability on a scale from 1 to 5, can you instruct the LLM to answer the same questions and calculate the correlation with users' responses?
- I noticed that you included the prompts for Karel in the appendix, but I’m still curious whether the LLM has access to task information when generating both the explanation and the reconstructed program. It seems intuitive that understanding a program would be easier when the purpose of the code blocks is provided.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730528486487

thread_type:
official_review

<thread_object>
creator: reviewer_n7uq

<content>
summary:
This authors propose a metric called LINT score. This metric leverages LLMs to evaluate the interpretability of programmatic policies used in sequential decision-making tasks by reconstructing a policy based on its natural language description.

The proposed method includes using one LLM describe a policy's behavior in natural language, followed by a second LLM attempting to reconstruct the policy based on this description.  The similarity between the original and reconstructed policies serves as the interpretability measure. The authors hypothesize that higher similarity indicates greater interpretability. The metric was tested in two domains: MicroRTS and Karel. The experiments involve increasing the obfuscation level of policies to simulate varying degrees of interpretability, with user studies to validate the correlation between LINT scores and human perceptions of interpretability

strengths:
- The paper presents the LINT score methodology with clarity, detailing each component the explainer, reconstructor, and verifier and its purpose within the interpretability assessment pipeline.
- Although initial results suggest areas for improvement, the study opens up valuable avenues for refining interpretability metrics, setting a constructive groundwork for further development in areas of robotics and RL.
- The use of varying obfuscation levels and structured user studies across domains demonstrates a good experimental design.

weaknesses:
- Since the original policy is directly provided to the explainer, there’s a risk that the natural language description generated is highly specific and detailed, making it relatively straightforward for the reconstructor to recreate the policy. This dependency raises questions about whether the LINT score truly reflects interpretability or merely the ability of another LLM to follow a sufficiently descriptive prompt.
- The paper defines interpretability narrowly as the ability to reconstruct a policy from a natural language description, which may not capture the full human-centric aspects of interpretability.
- The LINT score is tested only on specific domains (MicroRTS and Karel the Robot), both of which are relatively contained environments with domain-specific languages. It’s unclear how well the LINT score would generalize to more complex or diverse environments.
- The method relies on the assumption that a policy that is interpretable should be reconstructable in a way that behaves similarly to the original policy. However, minor discrepancies in LLM-generated reconstructions, even if they don’t reflect actual interpretability, might impact the LINT score due to the stochastic nature of LLMs. Because LLMs are stochastic and may produce different outputs for the same prompt, achieving consistent LINT scores might require multiple runs, which complicates reproducibility.
- The paper relies solely on the LINT score for interpretability assessment without integrating other interpretability measures, such as feature importance or visualization techniques, that might provide a more comprehensive view of a policy's interpretability.
- Although the LINT score aims to be cost-effective compared to user studies, the computational expense of using multiple LLM instances (Explainer, Verifier, Reconstructor) may still be high, especially for larger-scale deployments.
- While I do feel this area of study is important, the methodology in this paper felt more like a pipeline of multiple LLMs strung together, which limits its methodological novelty.

questions:
- Since programmatic policies provide explicit code with defined logic, wouldn’t they be inherently interpretable by examining their internal workings? Could you clarify why an additional interpretability metric, like LINT, is necessary when the policy’s behavior can ostensibly be understood directly from its code structure?
- Given that LLMs themselves are black boxes, what is the rationale behind using LLMs as proxies for interpretability? Wouldn't alternative methods, such as rule-based systems offer more transparency in the interpretability assessment?
- For variability in LINT scores across multiple runs? Did you consider using temperature or other settings to stabilize outputs?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730579967746

thread_type:
official_review

<thread_object>
creator: reviewer_tpe6

<content>
summary:
Programmatic policies for sequential decision-making problems often promise interpretability, but measuring and evaluating this promise can be challenging. User studies that assess interpretability tend to be subjective and costly. This work introduces LINT, a straightforward solution based on large language models (LLMs) to evaluate the interpretability of programmatic policies. LINT employs an LLM explainer to translate the program into natural language text, adhering to specific constraints, and then uses an LLM reconstructor to regenerate the original program from that text. If the original and reconstructed policies are similar, the original is considered interpretable. The authors demonstrate LINT’s effectiveness as an interpretability evaluator through obfuscation experiments and user studies in two domains: MicroRTS and Karel the Robot.

strengths:
* The authors have pinpointed a significant challenge in the realm of programmatic policies for sequential decision-making: evaluating the interpretability of these policies. They propose a straightforward yet effective algorithm that addresses this issue, with their methodology and experimental design presented clearly. The prompts used for the LLMs and the specifics of the user studies are well detailed.
* The design choices for LINT—including the three-model setup featuring an explainer, verifier, and reconstructor, as well as the constraints for generation—are straightforward and logical. While the selected metrics, specifically the action and return heuristics, can be debated (e.g., why not consider state visitation?), they serve as defensible and practical starting points.
* The authors demonstrate a strong correlation between the LINT score and the level of obfuscation in programs, indicating that more obfuscated programs are less interpretable and thus receive lower LINT scores. This finding is promising for utilizing LLMs in code interpretability, applicable not only to programmatic policies but to any computational program.
* The user study suggests that LINT has potential as an effective tool for assessing the interpretability of programmatic policies.

weaknesses:
I have three main concerns with this paper that prevent me from giving it a higher score, which I outline below:
* The primary factor that would make LINT a compelling algorithm and metric is its correlation with user interpretability. While the authors demonstrate a positive correlation through their experiments, the metrics used to assess user interpretability are unconvincing and potentially noisy. Users viewing a programmatic policy alongside the relatively straightforward domain-specific languages (DSLs) in the benchmarks might feel they understand the program and assign a high I-score, even if their understanding is superficial. Although the V-score attempts to address this, the authors acknowledge that it is a noisy metric; designing an evaluation setup with four clear options is challenging, and users can often make educated guesses. This complicates drawing conclusions from the user study results. Additionally, the scores in the table for I-scores and V-scores suggest that obfuscated programs may have scores very close to those of more interpretable ones (as seen in Table 5).
* A second challenge in evaluating LINT is the absence of baseline comparisons. Even if LINT is the first algorithm focused on evaluating the interpretability of programmatic policies, it would be beneficial to compare it against simple baselines using code heuristics—such as the number of lines, loops, and conditional clauses—to gauge interpretability. Such comparisons would strengthen the validity of LINT, particularly in light of the noisy I-scores and V-scores. Without these baselines, it raises the question of whether simple heuristic-based methods could also effectively distinguish between obfuscated and straightforward programs.
* Lastly, the LINT score itself is difficult to interpret. For instance, in Section 6.3, a policy receives a LINT score of (0.626, 0.57). This score leaves me uncertain about the interpretability of the policy; while 0.626 doesn’t seem impressive out of 1, the action score is at least bounded between 0 and 1. The return score, however, is unnormalized and unbounded, making it very domain-specific and hard to interpret. As a user, I am left unsure of how good or bad the policy truly is. In this case, the policy appears quite interpretable to me, but I would not be able to ascertain that just from the LINT score.

questions:
* When I run LINT and receive scores for Action and Return, how should I interpret them? The Action score is somewhat straightforward since it ranges from 0 to 1 although there are challenges as described in the Weaknesses section. However, in grid world scenarios, multiple aliased actions for policies $\pi$ and $\pi^{'}$ can unfairly penalize the Action score. Additionally, if policy $\pi_1$ has an Action score of 0.8, $\pi_2$ has 0.7, and $\pi_3$ has 0.6, can we confidently say that $\pi_2$ is better than $\pi_3$ by the same margin that $\pi_1$ is better than $\pi_2$? Being able to make such claims would be valuable, especially when balancing interpretability against performance.
* It would also be beneficial to include a brief description of the Karel and MicroRTS environments for users who may not be familiar with them.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730672334613

thread_type:
official_review

<thread_object>
creator: reviewer_vvdr

<content>
summary:
The author proposes a novel metric, LLM-based INTerpretability (LINT) score, which uses large-language models (LLMs) to quantify the interpretability of programmatic policies. The method leverages three LLMs to act as explainer (to generate natural language description), reconstructor (to generate policy based on language description from explainer) and verifier (to check if the explainer has not provided any unwanted information about policy). The authors validated the usability of LINT across two domains, MicroRTS and Karel the robot, demonstrating its correlation with human interpretability assessments.

strengths:
- The proposed solution for quantifying the interpretability of programmatic policies is innovative but depends heavily on LLMs, which are inherently stochastic. This reliance introduces potential variability in results, but the authors address this concern by implementing measures such as multi-evaluation runs and verification steps to mitigate inaccuracies and enhance reliability.

- The authors validate the effectiveness of LINT scores across two distinct domains, MicroRTS and Karel the Robot, demonstrating a strong correlation between LINT scores and human interpretability assessments.

weaknesses:
**Major:**
- **Importance of proposed method:** Currently, it is not much clear why there is a need for interpretability score for programmatic policies. From the examples shown in paper (table 1 and 2), these policies are in itself interpretable and might not warrant the need for the proposed score. Have there been any study to support the cause, authors should highlight that in introduction.
- **No theoretical guarantee:** Although the approach proposed by authors is interesting, they don't provide any theoretical guarantee as the scores they achieve is in actuality explains the policy.
- **Lack of baseline evaluation:** Although the authors conducted human study to support the method, they lack comparisons with other baseline methods. Just by looking at results from table 3 and 4 it is hard to conclude if the proposed method is an improvement or not.

**Minor:**
- Adding a brief introduction about programmatic policies in the problem definition will help in better understanding of the paper.
- The authors have described about the limitations of their approach at multiple places (lines 163 - 172 and section 4). Maybe these can be moved towards the end as a Limitations section.
- **Bad representation of data:** The letter superscript presented in table 4 and 5 for column I-score and V-score is really confusing. There should be a better way to represent that data.
- **Typo (line 300):** IFor &#8594; For

questions:
1. What specific challenges in programmatic policy interpretability justify the need for a dedicated metric like LINT?

2. Are there theoretical guarantees that the LINT score reliably reflects policy interpretability?

3. How does LINT compare to other interpretability metrics, and have any baseline comparisons been considered?
</content>
</thread_object>

</thread>
<thread>
creation date:
1731805149641

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Summary of Revisions

comment:
We thank the reviewers for their feedback on our work. We have revised the paper according to their feedback and have uploaded it on OpenReview. You will see the changes in blue in the revised manuscript. In order to fit all the changes in the main text, we moved the representative reconstruction example we had in the main text to Appendix C of the revised paper. 

We believe the paper has improved substantially after dealing with the questions and suggestions of the reviewers. In particular, the addition of baselines certainly strengthened the contributions of our work. Please do not hesitate to ask further clarifying questions or suggest further changes to our paper.

In addition to clarifying several aspects around the user study (please see the individual answers below), we made the following changes to the paper.

- Added two baselines to our experiments. In one of the baselines, we use LLMs to assign an interpretability score to the programs we evaluate. However, in contrast with LINT, this LLM baseline does not use LINT's natural language bottleneck (see Section 4 of the revised version). We also added a comparison of LINT with simple structural metrics such as number of lines of code, number of loops and conditionals (see Appendix E of the revised version). None of these baselines correlate as strongly as the LINT scores to the I and V-Scores collected in our user study. See Section 5.2 and Appendix E of the revised paper for a longer discussion of these results (**Reviewers vVdR, tPe6, gZii, and deQy**).
- We added in Appendix E empirical results on C programs from an obfuscation competition. (**Reviewers vVdR, tPe6, n7Uq and deQy**)
- Added a brief introduction to programmatic policies at the end of Section 2 (**Reviewer vVdR**).
- Appendix L has been added, containing four tables that detail the statistical differences in the I-Score and V-Score columns, providing clarification on the superscripts used in Tables 4 and 5 for both Karel and MicroRTS (**Reviewer vVdR**).
- Answered all questions the reviewers asked (please see **individual comments** below).
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper introduces the LINT (LLM-based INTerpretability) score, a novel metric for evaluating the interpretability of programmatic policies in reinforcement learning. The methodology involves using large language models (LLMs) in three roles: (1) generates a natural language description of the policy, (2) reconstructs the policy from the explanation, and (3) ensures the explanation doesn't leak unnecessary details for reconstruction. The LINT score quantifies interpretability based on the similarity between the original and reconstructed policies. The method is tested in two domains—MicroRTS and Karel the Robot—using obfuscation experiments and user studies to validate its correlation with human interpretability assessments.

Reasons to accept
- The use of LLMs to quantify interpretability in programmatic policies is interesting and creative.
- The proposed pipeline (explainer, reconstructor, verifier) is well-structured, with explicit design choices and motivations for each step.
- Testing across two domains (MicroRTS and Karel) with obfuscation experiments and user studies demonstrates thoughtful experimental design.
- The authors are transparent about the limitations of their approach, including the stochastic nature of LLMs and the domain-specificity of their experiments.


Reasons to reject
- The paper does not convincingly establish why an interpretability score is necessary for programmatic policies, which are often inherently interpretable by examining their code.
- The methodology does not provide formal guarantees that the LINT score accurately reflects policy interpretability.
- The paper does not benchmark LINT against existing interpretability metrics or heuristic-based methods. This omission raises doubts about whether LINT offers any improvement over simpler approaches.
- The experiments are confined to two relatively simple domains, MicroRTS and Karel, with small sample sizes in user studies. This limits the generalizability of the findings to more complex or real-world scenarios.
- The LINT score (e.g., action and return components) is difficult to interpret, especially without normalization or domain-independent scaling. 
- The stochastic nature of LLMs can introduce variability in results, complicating reproducibility. Also, the black-box nature of LLMs undermines the interpretability of the LINT score itself.
- The reliance on multiple LLMs (explainer, reconstructor, verifier) can be computationally expensive, challenging the claim of cost-effectiveness compared to user studies.

While this paper studies a promising research direction (the interpretability of programmatic RL policies) and proposes an interesting approach, I believe its weaknesses outweigh its strengths. Consequently, I recommend rejecting the paper.