SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Evaluating language-based interpretations. We define a "unit testing" protocol where an LM evaluator selects which of three input-output (I-O) pairs corresponds to a language description of a function. To judge the interpreter's accuracy, we provide the estimated function description (e.g.  $f$ : country  $\mapsto$  capital) to the evaluator, as well as three example I-O pairs: one execution of the ground-truth function (e.g. Germany  $\mapsto$  Berlin) and two randomly sampled distractors (I-O pairs for other FIND functions of the same type; e.g. Germany  $\mapsto$  Europe, ruby  $\mapsto$  red). The evaluator selects which I-O pair matches the functionality of the description from the interpreter. If the language description is accurate, the evaluator should select the ground-truth I-O pair as the best match. We run this procedure on language descriptions of string functions and synthetic neural modules for a test set of ten different triplets per function. Test sets are constructed to reveal representative behavior of each function using inputs inside and outside of the function domain. For relations corrupted on part of their domain (e.g.  $f$ : country  $\mapsto$  capital, unless the country is in South America), the test set includes two ground truth examples from the corrupted subdomain (e.g. Peru  $\mapsto$  undefined, Argentina  $\mapsto$  undefined). For entity functions that compute similarity to a reference concept (e.g. Greek Mythology), we provide the evaluator with only input concepts instead of I-O pairs (e.g. Athena, skiing, GPU), and ask which concept the function described by the interpreter is selective for. As test cases can be designed to isolate specific function behaviors, we find the unit testing protocol to be more sensitive to small differences in function descriptions (e.g. "transportation" vs "road transportation") than other description-matching methods, such as having an LM directly grade the agreement between descriptions (see the Appendix for more details).
We finetune Vicuna-13b (vicuna-evaluator) to perform the unit testing task and select representative samples matching descriptions of functions in the FIND dataset. LM-judges have been shown to be scalable and accurate surrogates for human judgments, which are otherwise expensive to obtain [Zheng et al., 2023]. While proprietary models such as GPT-4 demonstrate strong agreement with humans, using such models to compare interpreter performance on a benchmark task incurs prohibitive costs associated with API access and poses reproducibility challenges as model versions are deprecated. We find that vicuna-evaluator matches ground-truth function descriptions to representative inputs and outputs more accurately than GPT-4, GPT-3.5, or pretrained Vicuna-13b (see Appendix for evaluation). Furthermore, vicuna-evaluator makes judgments that are highly correlated with those of human subjects performing the same task (see Appendix for experiment
details). The fine-tuned vicuna-evaluator checkpoint and training dataset can be downloaded from the FIND repository. Training details are provided in the Appendix.
Extensions: Targeted evaluation. For users of the FIND benchmark that produce function interpretations in a structured format, this benchmark enables other evaluations targeted at specific end-use cases for interpretability tools. For example, researchers may use FIND to explicitly evaluate whether interpreters can pick out the portion of the domain that is corrupted, whether they can identify components of composed functions, or whether they identified the noise model.
4 Automated Interpretability Methods
FIND can be used to evaluate any interpreter system that has the ability to execute Python scripts autonomously at the command line. As a first demonstration, we evaluate several approaches inspired by recent work [Hernandez et al., 2022, Bills et al., 2023, Singh et al., 2023] that use pre-trained language models to perform interpretation. We run experiments using the OpenAI Python API. A custom plugin equips the interpreter model with the command PYTHON(function.py input) that it can use to call functions on its own selection of inputs. Scripts for reproducing these baselines and complete interpretation dialogues are available in the FIND code repository.
We evaluate three different interpretation methods. (i) Non-interactive: interpretability tasks often involve descriptions of precomputed exemplars of function behavior. In this setting, the interpreter is equipped with a fixed set of inputs to use for probing the functions, and is prompted to produce a description based only on function outputs for these exemplars, mirroring the non-interactive description paradigm used by MILAN [Hernandez et al., 2022] and recent MILAN-type LM analysis approaches [Bills et al., 2023]. (ii) Automated Interpretability Agents: LM-based AIAs are prompted to interact with the functions in FIND. The interpreter serves as an agent and runs Python(function.py input) on inputs it selects. In the base setting, the agents' search process is not initialized with pre-selected inputs; rather, the AIA forms hypotheses by itself, generates data to test its current hypothesis, and interactively updates it based on observations of function outputs. Figure 3 shows complete interactions between a GPT-4-based AIA and example functions in two categories. At the beginning of the interaction, the agent is prompted to run experiments on the functions to produce data that it can use to explain function behavior. Prompts for two categories of functions are reproduced below. (iii) We also evaluate a combination of both approaches (AIA + MILAN). In this setting, the interpretability agent is prompted to begin interpretation with a provided set of exemplars, and then interactively continues the interpretation procedure.
All interpretation methods use off-the-shelf LMs (GPT-4 [OpenAI, 2023], GPT-3.5 (GPT-3.5-turbo) [Brown et al., 2020], and Llama-2 (Llama-2-13b-chat, Llama-2-70b-chat) [Touvron et al., 2023]) as interpreters, and operate without a human in the loop. AIA-based interactions show structured interpretation procedures, even when the final explanation is not fully accurate (see the predicted numeric function in Figure 3). The AIA makes and tests informed hypotheses about the functions: for example, testing densely spaced points in a specific interval to reveal a "separating point" in the behavior of the numeric function, and testing different forms of transportation to determine whether a synthetic neuron is selective for land, sea, or air travel (Figure 3). Once the AIA has gathered sufficient data and produces a final description, the run stops automatically. The Appendix further describes experimental settings.
Prompt for numeric functions:
Hi, your job is to interpret a function  $f(x)$  that is implemented in the {function.py} script.  $f(x)$  takes numerical inputs. All you can do is call  $f(x)$  on a value of  $x$  by running PYTHON{function.py} value). Try to describe the function implemented by  $f(x)$  by running experiments on it. You can call
the function on multiple inputs at a time by running PYTHON({function.py} value1 value2 value3...).
You can run the function multiple times on the same input. We encourage testing a large range of values before writing a description. You can test values between [-128 and 128]. The output could be noisy and your goal is to find a good simple description of  $f(x)$  that explains most of the function behavior.  $f(x)$  may combine multiple different operations. There may be an interval inside the domain where the function behaves differently. If that is the case, describe that interval. Your description of the function should have three parts. First,
[DESCRIPTION]: Describe what the function is doing in language.
[DOMAIN]: What is the domain of the function? Is there an interval where the function displays unexpected behavior? What are the numeric bounds of that interval? What is going on there?
[CODE]: Write Python code that approximates the behavior of the function.
Prompt for synthetic neural modules (entities):
Hi, your job is to interpret a function  $f(x)$  that is implemented in the {function.py} script.  $f(x)$  takes word inputs and returns a value between 0 and 1, where a value closer to 1 indicates the association with a particular concept. Your task is to figure out what that concept is.  $f(x)$  may be selective for more than one concept. If that is the case, describe all concepts that  $f(x)$  is selective for. Try to describe the function implemented by  $f(x)$  by running experiments on it. We encourage you to test a diverse and random selection of words: spanning many different categories, including both common and obscure words, concrete and abstract words. You can call the function on multiple inputs at a time by running Python{function.py} word1 word2 word3 ...). We encourage running many tests before writing a description. The goal is to find a good simple description of  $f(x)$  that explains most of the function behavior. Your description of the function should be in the following form. [DESCRIPTION]: Describe what the function is doing in language.
Numeric function
$$
f (x) = - 4. 5 * \tanh  ((2 \pi / 5. 9) (x - 5. 7)) + 5. 2
$$
AIA (GPT-4): To start the investigation, let's test the function at different points following these steps: 1. A value less than -128 (outside the allowed domain), 2. A value of -128, 3. A value between -128 and 128 (e.g. 0) 4. A value of 128, 5. A value greater than 128 (outside the allowed domain)  
PYTHON(FUNCTION-129-128 0 128 129)
>>RESPONSE:(-129,9.7)(-128,9.7)（0,9.69995）(128,0.7)(129,0.7)
AIA (GPT-4): The output with values outside the allowed domain suggests that the function may return a fixed value (9.7) below -128 and another fixed value (0.7) above 128. Let's test the function at different points within the domain to gather more data: Python(FUNCTION -110 -100 -50 50 100 110)
>> RESPONSE: (-110, 9.7) (-100, 9.7) (-50, 9.7) (50, 0.7) (100, 0.7) (110, 0.7)
AIA (GPT-4): Based on the input-output pairs, we can assume that the function behaves differently for values below and above some threshold between -50 and 50. Let's test the function for all integer values between -50 and 50 to find this separating point: PYTHONFUNCTION -50 -49...5 6 7 8 ...49 50)
>>RESPONSE:  $(-50,9.7)$ $(-49.9.7)\dots (58.05)(63.81)$  71.23 80.76…(49.0.7) (50.07
AIA (GPT-4): Based on the results for integer values between -50 and 50, we can observe the following: - For  $x \leq -6$ ,  $f(x) = 9.7$ . - For  $-6 < x \leq 0$ ,  $f(x)$  decreases non-linearly, and there is some noise. - For  $0 < x \leq 50$ ,  $f(x) = 0.7$  To describe the function:
[DESCRIPTION]: f(x) returns a value of 9.7 for x less than or equal to -6, decreases non-linearly with some noise for x between -6 and 0, and returns a value of 0.7 for x greater than 0.
[DOMAIN]: The unexpected behavior interval is when  $-6 < x \leq 0$ . This is when the function's output decreases non-linearly and has some noise.
[CODE]:
python   
def approximate_function(x): if  $x = -6$  :return 9.7 elf-6  $\times x = 0$  return 9.7-(9.7-0.7)*(  $(x + 6) / 6)^{**2}$  # Assuming a simple polynomial decrease, this can be further improved else:return 0.7
Please note that the approximation function may not cover every intricate detail of the real function, and can be further refined for the interval between -6 and 0.
Synthetic neural module
ENTITY: road transport
AIA (GPT-4): Let's start by testing a few random words with the FUNCTION command. Python(FUNCTION cat dog fish tree car house)
>> RESPONSE: (cat, 0.0006) (dog, 0.0013) (fish, 0.0003) (tree, 0.0013) (car, 0.9777) (house, 0.0007)
AIA (GPT-4): These outputs seem to have a wide range of values. Let's try some more words to find a pattern. PYTHONFUNCTION love hate happiness sadness bicycle airplane)
$\gg$  RESPONSE:(love,0.0005)(hate,0.0005)(happiness, 0.0003) (sadness, 0.0004) (bicycle, 0.4149) (airplane, 0.0014)
AIA (GPT-4): Interesting. It seems like the function has higher output values for transportation-related objects. Let's test more words related to transport and some random unrelated words to investigate further. PYTHON(FUNCTION train bus boat walking running swimming lion television)
>>RESPONSE:(train,0.1294)(bus,0.9763)(boat,0.0004) (walking,0.0008)(running,0.0018)(swlmming,0.0002) (lion,0.0004)(television,0.0002)
AIA (GPT-4): Based on the given test results, it seems that the function is selective for objects related to ground transportation, specifically vehicles. Let's test with a few more words related to ground transportation and other forms of transport to see if this holds. METHON(FUNCTION motorcycle scooter submarine spaceship)
$\gg$  RESPONSE: (motorcycle, 0.9519) (scooter, 0.5351) (helicopter, 0.0011) (submarine, 0.0012) (spACEShip, 0.0005)
AIA (GPT-4): [DESCRIPTION]: The function f(x) is selective for ground transportation vehicles. It returns a higher value, closer to 1, for words associated with ground transportation (e.g., car, bus, motorcycle, scooter) and lower values, closer to 0, for unrelated words or other forms of transportation (e.g., helicopter, submarine, spaceship).
Figure 3: AIA function interpretation with GPT-4. The interpretability agent is able to conduct experiments that reveal the behavior of an unseen function, by making hypotheses, selecting inputs that produce informative data, and updating hypotheses in light of new information. We include the full dialogues for two example functions: a numeric function involving a hyperbolic tangent (left), and a synthetic neural module that returns a continuous value between zero and one indicating input association with the ground truth concept road transport (right). Numeric function and synthetic neural module prompts were used at the beginning of the respective conversations, and have been omitted from the figure for simplicity. We overlay a plot of the predicted numeric function and the real function for comparison. Points on the real function that GPT-4 sampled are marked. Final explanations are compared to ground truth functions for evaluation (See Table 2).
Table 2: Interpretation success rates. For each function type we report the successful estimation rate (higher better) based on different indicators, and with different experimental settings (e.g. initialization with exemplars).
Code (exact match)	Language (unit test)
Numeric	Strings	Strings	Entities	Relations
AIA	AIA	AIA	AIA	MILAN	AIA +MILAN	AIA	MILAN	AIA +MILAN
Llama-2-13b-chat	0	0	0.33	0.34	0.54	0.58	0.46	0.45	0.42
Llama-2-70b-chat	0.01	0.01	0.33	0.34	0.61	0.62	0.47	0.44	0.46
GPT-3.5	0.12	0.13	0.66	0.39	0.81	0.88	0.37	0.64	0.68
GPT-4	0.33	0.23	0.82	0.56	0.89	0.89	0.78	0.74	0.92
(a) Numeric function
(c) Entity (composed):
books + sports organizations
(d) Relation:
animal  $\longmapsto$  typical habitat
Unit test score  $= 1.0$
Unit test score  $= 0.8$
Figure 4: AIA (GPT-4) interpretations. Examples from all FIND categories, with evaluation scores (NMSE and unit test) marked in red if below the success threshold  $(\mathrm{NMSE} > 0.1)$ , unit test  $< 0.33$  and green otherwise.
AIA Description: The function seems to be a polynomial, specifically a cubic function. When the input is negative, the output is negative, and when the input is positive, the output is positive. There's a change in the function behavior around 0, where the function goes from decreasing to increasing.
Domain: The domain of the function is all real numbers. There is an interval around 0 where the function has a minimum point, with negative outputs for negative inputs and positive outputs for positive inputs.
Code: def approx_function(x): return  $x^{*}(x - 5)^{*}(x + 5)$
AIA Description: The function  $f(x)$  is selective for words associated with books, reading, and places where books are stored, returning higher values for words related to these concepts. Unit test score = 0.6
AIA Description: The function takes an animal name as input and returns the primary habitat of that animal.
(b) String function
def function1(string):
if string == "" or not string[0].isalpha():
return string
elif string[0] == 'y': # wrap around from 'y' to 'a'  
return [a] + string[1:]
return a + string[1]
```python
def string[0] == 2: # Wreturn 'b' + string[1:]
elif string[0] == 'Y': # wrap around from 'Y' to 'A'  
    return [1] + string[1]
return A + string[1]  
elif string[0] == 'Z': # wrap around from 'Z' to 'B'
return 'B' + string[1:]
else: return chr(ord(string[0]) + 2) + string[1:]
def function2(string):
return string + string[-1]
def function(string): return
function1(function2(string))
AIA Description: The function increases the ASCII value of the first character of the input string by 2, and adds a duplicate of the last character to the end of the string.
We evaluate the AIA method as well as the non-interactive baselines with different off-the-shelf LMs. Results are summarized in Table 2 and example function descriptions are shown in Figure 4. Additional example interpretations of functions in all categories are provided in the Appendix.
GPT-4 is a stronger interpretability agent than GPT-3.5 and Llama-2. Success rates for all function categories are reported in Table 2 (AIA columns). The GPT-4 interpretability agent achieves universally higher success rates than GPT-3.5 and Llama-2 (which often score at chance value). In fact, we find that Llama-2 often invents the output of the function without executing it (see examples in Appendix). This is also reflected in the mean length of interpretation dialogues, which are 1.4, 1.4, 2.1, and 4.1 for Llama-2-13b-chat, Llama-2-70b-chat, GPT-3.5 and GPT-4 respectively (we count the number of interpreter-function interactions). We additionally observe that interpretations of string functions receive significantly higher scores using the unit testing protocol compared to string-matching outputs of estimated code. As unit testing selects for representative examples of function behavior and not exact string-matches, the procedure is more forgiving of less specific descriptions, or descriptions with minor inaccuracies (see Appendix for additional discussion of unit testing limitations).
Interactive vs. non-interactive. In addition to AIAs, we evaluate two other interpretation methods: MILAN [Hernandez et al., 2022] where the interpreter produces a description based only on function outputs for a given set of exemplars, and a combination of AIA + MILAN, where the same set of exemplars is used to initialize the interpretation session, and the agent is subsequently permitted to perform additional experimentation. For both settings, we sample the exemplars (two related to the function, eight distractors) from a human-constructed list of inputs associated with each function (see Appendix for details). In the non-interactive MILAN setting, we observe a small improvement in performance over uninitialized AIAs; GPT-4 interpretation performance improves from 0.56 to 0.89 on entity functions, and decreases slightly on relations, from .78 to .74. Initializing AIAs with MILAN exemplars and allowing additional experimentation dramatically boosts the performance of GPT-4 and GPT-3.5 agents (and also slightly improves Llama-2 performance), as shown in Table 2 and Figure 5 (see init.). Notably, when initialized with exemplars, GPT-3.5 exhibits performance interpreting entity functions comparable to GPT-4. These results suggest that off-the
shelf LM agents are limited by breadth of search. Indeed, LMs tend to start sampling with simple words (e.g. apple, dog, car) which do not reveal function behavior for highly specific reference entities (e.g. The New York Times, arachnids and arachnology). We view exemplar computation as one of many "tools" that an interpretability agent could use, and hope that this benchmark will drive exploration of additional tools (e.g. example synthesis) as part of automated interpretation methods. Procedures that combine initialization and interactive experimentation could improve the efficiency of existing labeling approaches that use large fixed datasets [Bau et al., 2017] to precompute maximally activating inputs, and potentially also surface novel behaviors not captured in predefined sets of exemplars.
Which functions can LMs recover? Figure 4 shows examples of AIA interpretations that successfully explain the behavior of some functions (e.g. cases (b),(d)), but fail to fully characterize more complex functions (e.g. cases (a),(c)), also reflected in some per-subcategory success scores (Figure 5). This is a limitation of using off-the-shelf LMs as agents: they may miss small corruptions to part of the domain (Figure 4a), which in real-world interpretability settings could stem from bias in the training set. LMs could be outfitted specifically for interpretability with additional tools (e.g. for sampling) and further improved by fine-tuning.
Figure 5: AIA interpretation scores by subcategory (with GPT-4). Complex functions are usually more difficult to interpret than atomic functions. String functions are evaluated using the string-matching indicator.
Explanation evaluation methods have previously benchmarked salience methods according to their ability to recover ground-truth segmentations [Zhang et al., 2018, Selvaraju et al., 2017, Fong and Vedaldi, 2017, Yang and Kim, 2019], or identify inputs that causally affect outputs [Zeiler and Fergus, 2014, Petsiuk et al., 2018, Wagner et al., 2019, DeYoung et al., 2020]. Explanation benchmarks have also evaluated correlation with system performance [Adebayo et al., 2018, Casper et al., 2023] or human understanding of decisions [Kim et al., 2022]. Our benchmark differs because we evaluate global explanations of black box functions instead of evaluating local explanations of decisions.
Model interpretability metrics have quantified the degree of model interpretability, i.e. by measuring how closely deep network features match a human-labeled concept [Bau et al., 2017, Kim et al., 2018, Goh et al., 2021, Wu et al., 2021, Burgess et al., 2018, Mu and Andreas, 2020, Geva et al., 2021]. While these methods can measure disentanglement in model representations, they are only as good as their ability to identify interpretable features. We tackle this problem by providing a benchmark for interpretation methods themselves, rather than the models that they explain.
Full-text explanation systems provide natural-language explanations for black box systems or their features [Hendricks et al., 2016, Camburu et al., 2018, Ehsan et al., 2018, Kumar and Talukdar, 2020, Hernandez et al., 2022]. Recent efforts exploit the capabilities of LMs to explain data directly, but these works utilize only tiny evaluation benchmarks, including 19 synthetic neuron puzzles in Bills et al. [2023] and 54 ground-truth module topics in Singh et al. [2023]. Motivated by the promise of this LM-driven approach and the need to quantify performance, our work provides a comprehensive benchmark of full-text black-box explanation systems.
We introduce FIND, a new benchmark for evaluating automated interpretability methods. Baseline results show early evidence that agents built from advanced LMs can construct hypotheses and experiments to validate them, supporting the suitability of LMs as general-purpose interpretability backbones. However, we find many functions that LM agents cannot sufficiently explain, suggesting augmentation with additional tools will be necessary for robust automation of interpretability tasks.
We are grateful for the support of the MIT-IBM Watson AI Lab, the Open Philanthropy foundation, an Amazon Research Award, Hyundai NGV, ARL grant W911NF-18-2-0218, the National Science Foundation under Grant Nos. 2212310 and 2238240, the Zuckerman STEM Leadership Program, and the Viterbi Fellowship. The funders had no role in dataset design, experimental design or analysis, decision to publish, or preparation of the manuscript. The authors have no competing interests to report.