FIND: A Function Description Benchmark for Evaluating Interpretability Methods
Authors: Sarah Schwettmann
Tamar Rott Shaham
Joanna Materzynska
Neil Chowdhury
Shuang Li
Jacob Andreas
David Bau
Antonio Torralba
Venue: NeurIPS 2023 Datasets and Benchmarks Poster (2023)
Keywords: Interpretability
Explainability
Dataset
Benchmark
LLMs
LM Agents
Language Model
TL;DR: An interactive dataset of black box functions and benchmark for evaluating interpretability methods, including a new Automated Interpretability Agent (AIA)

Abstract: Labeling neural network submodules with human-legible descriptions is useful for many downstream tasks: such descriptions can surface failures, guide interventions, and perhaps even explain important model behaviors. To date, most mechanistic descriptions of trained networks have involved small models, narrowly delimited phenomena, and large amounts of human labor. Labeling all human-interpretable sub-computations in models of increasing size and complexity will almost certainly require tools that can generate and validate descriptions automatically. Recently, techniques that use learned models in-the-loop for labeling have begun to gain traction, but methods for evaluating their efficacy are limited and ad-hoc. How should we validate and compare open-ended labeling tools? This paper introduces FIND (Function INterpretation and Description), a benchmark suite for evaluating the building blocks of automated interpretability methods. FIND contains functions that resemble components of trained neural networks, and accompanying descriptions of the kind we seek to generate. The functions are procedurally constructed across textual and numeric domains, and involve a range of real-world complexities, including noise, composition, approximation, and bias. We evaluate methods that use pretrained language models (LMs) to produce code-based and natural language descriptions of function behavior. Additionally, we introduce a new interactive method in which an Automated Interpretability Agent (AIA) generates function descriptions. We find that an AIA, built with an off-the-shelf LM augmented with black-box access to functions, can sometimes infer function structure—acting as a scientist by forming hypotheses, proposing experiments, and updating descriptions in light of new data. However, FIND also reveals that LM-based descriptions capture global function behavior while missing local details. These results suggest that FIND will be useful for characterizing the performance of more sophisticated interpretability methods before they are applied to real-world models.

FIND: A Function Description Benchmark for Evaluating Interpretability Methods
Sarah Schwettmann
Tamar Rott Shaham I-
Joanna Materzynska'
Neil Chowdhury? Shuang Li¹ David Ban
Jacob Andreas
Antonio Torralba'
MIT CSAIL
Northeastern University
Abstract
Labeling neural network submodules with human-legible descriptions is useful for many downstream tasks: such descriptions can surface failures, guide interventions, and perhaps explain important model behaviors. To date, most mechanistic descriptions of trained networks have involved small models, narrowly delimited phenomena, and large amounts of human labor. Labeling all human-interpretable sub-computations in models of increasing size and complexity will almost certainly require tools that can generate and validate descriptions automatically. Recently, techniques that use learned models in-the-loop for labeling have begun to gain traction, but methods for evaluating their efficacy are limited and ad-hoc. How should we validate and compare open-ended labeling tools? This paper introduces FIND (Function INterpretation and Description), benchmark suite for evaluating the building blocks of automated interpretability methods. FIND contains functions that resemble components of trained neural networks, and accompanying descrip- tions of the kind we seek to generate. The functions are procedurally constructed across textual and numeric domains, and involve a range real-world complexities, including noise, composition, approximation, and bias. We evaluate methods that use pretrained language models (LMs) to produce code-based and natural language interactive
descriptions of function behavior. Additionally, method in which an Automated Interpretability Agent (AIA) generates function descriptions. We find that AIA, built with off-the-shelf LM augmented with black-bor box access to functions, can sometimes infer function structure acting as a scientist by forming hypotheses, proposing experiments, and updating descriptions in light of new data. However, FIND also reveals that LM-based descriptions capture global function behavior while missing local details. These results suggest that FIND will be useful for characterizing the performance of more sophisticated interpretability methods before they are applied real-world models.
introduce
Introduction
The central task of interpretability research is explain the functions that AI systems learn from data. Investigating these functions requires experimentation with trained models, using tools that incorporate varying degrees of human input. Hand-toole approaches that rely close manual inspection [Zeiler and Fergus, 2014, Zhou et al., 2014, Mahendran and Vedaldi, 2015, Olah et al., 2017, 2020, Elhage al., 2021] search for predefined phenomena [Wang al., 2022, Nanda et al., 2022] are increasingly complemented by more automatic approaches that enable larger-scale analysis [Bau al., 2020, Mu and Andreas, 2020, Hernandez al., 2022, Oikarinen and Weng, 2023,
Indicates equal contribution. Address correspondence to: schwett mit.edu, @amarott@mi.eede
37th Conference on Neural Information Processing Systems (NeurIPS 2023) Track on Datasets and Benchmarks.

---

Define atomic functions
Sample complex functions
Interpret functions
Evaluate interpretations
Numeric
Composition
maz(D),)
Text interpretation
6 mase(0,
o
sifisl)
AUB
The function implement ReLU (Roctified Linear Unit) activation function. it returns the ingur value if it is positive sero, otherwise returns zero.
AIA:
to
Rx
Domain Corruption
Interpreter
Strings
AIA: I will start by calling the fumetion with some input values to understand its behavior. FUNCTION(2, 6,, 8)
abc abd cat tac
Code interpretation
Approximation
Symthettcc modnles
Entities
AIA: def relu(x):
xe Espeetsy
(arr
>> RESPONSE: (U, (U,0,6,7,8)
(Peligioo)))
nax x)
AIA: It seams the function returns the input
(1(r)) f(ar
positive retums if the input negative. Let me run it with some neguiive values and lorger positive valuas confinu.
AIA
Relations
city courtry
Observation Noise
MSE
lergth
[(x) 10 1)
Figure 1: The FIND benchmark. FIND is constructed procedurally: atomic functions are defined across domains including elementary numeric operations (purple), string operations (green), and synthetic neural madules that compute semantic similarity to reference entities (yellow) and implement real-world factual associations (blue). Complexity is introduced through composition, bias, approximation and noise. We provide LM-based interpretation baseline that compares text and code interpretations to ground-truth function implementations.
Conmy et al., 2023]. Selection between approaches is highly application-deeedeenn to date, no single protocol answers all queries users might have about a system [Doshi-Vekke and Kim, 2017, Vaughan and Wallach, 2020]. However, considering the growing body of evidence that LMs are capable of complex reasoning and problem-solving across domains [Wei et al., 2022, OpenAI, 2023, Yao et al., 2023, Lightman al., 2023], we recognize their potential to become backbones of generalized agents for automated interpretability. Indeed, recent open-ended techniques have used pretrained LMs to describe the behavior of black-box text modules (Singh et al., 2023], including individual units inside other LMs [Bills et al., 2023]. As we enter regime where model explanation is performed by models that are themselves uninterpretable, external evaluations of these techniques will be vital. At present, evaluation of network description procedures is limited and bespoke, in part because measuring performance on real-world problems is difficult when ground-truth descriptions of network structure unknown [Doshi-Velee Velez and Kim, 2017, Lipton, 2018, Miller, 2019, Hooker et al., 2019].
This paper introduces FIND (F (Function INteprettaionn and Description), a benchmark suite for eval- uating the building blocks of automated interpretability methods on functions whose structure is known a priori (see Figure 1). FIND is built from over 2000 procedurally generated function inter- pretation problems (e.g. f country + capital, unless country is in South America, in which case s returns indd.. each problem, candidate interpretation procedures (interpreters) are given black-box access to functions, optionally accompanied by metadata (e.g. the domain of the function). After evaluating these functions on chosen inputs (e.g. Japan, Mexico, Peru), interpreters must eventually return structured description that can be evaluated against other descriptions or used to simulate function behavior.
We also introduce new interpretation method that uses Automated Interpretability Agents (AIAS) to interactively probe functions and explain their behavior. We formulate our approach to the interpretation problem an implementation of the scientific method, wherein the goal of the AIA is to describe the process underlying observed input-output relations. In contrast to existing full-text explanation systems that apply automatic captioning methods (e.g. [Hernandez et al., 2022, Bills et al., 2023]) to pre-selected onput-output pairs, the AIA generates the data itself, by running the function on inputs it selects, observing the outputs, and updating hypotheses until it can describe the function to human end-users. We evaluate both AIAS and existing, mon-interactive automated interpretability methods on FIND. While exhibiting sophisticated experimentation strategies and out- out-perfon non-interactive methods, the top- top-performing AIA nonetheless fails to adequately describe 48% of functions. FIND thus shows that, in spite of promising recent results, methods based on current LMs alone are unlikely to robustly automate even high-level interpretability tasks.
FIND focuses on the black-box function description paradigm because black-box description appears as a subroutine (or the sole operation) implemented by almost every existing automated inter- pretation method, spanning label retrieval, program synthesis, and learning- based approaches. The

---

functions that make up FIND are derived from past findings about specific phenomena observed in the wild, including polysemanticity [Olah et al., 2020, Gurnee al., 2023], compositionality [Fong and Vedaldi, 2018, Mu and Andreas, 2020], and task misspecification [Grünwald and Van Ommen, 2017], and thus exercise the kinds of interpretability behaviors that will required for understanding real neural networks. While not the focus of our initial release, FIND is extensible, and designed to eventually support targeted evaluation interpreters' ability to identify disparate model accuracy across input regions (relevant to fairness and accountability) and shortcut solutions to complex algorithmic problems (relevant to reasoning and robustness). We intend for to be a living bench- mark, incorporating new functions, interaction paradigms, and evaluation metrics, interpretability methods grow in sophistication and improve our understanding of real-world model behavior.
FIND functions
FIND is built from diverse set of programs including explicit mathematical equations, string opera- tions, logical composition, lexical relations, and factual associations. Our motivation to construct a set of tasks that assess how well an automated interpreter formulates and tests hypotheses about an opaque system, and based on the results of those experiments, writes a language description or program approximating the system. Using additional automated evaluation procedures that we describe in Section 3.1, language descriptions are compared to ground-trutt function explanations, and code-based descriptions are evaluated for how accurately they reproduce function behavior.
Each function in FIND defined inside an independent Python script that accepts input arguments from the interpreter and returns outputs to the command line. We construct FIND by defining set of atomic tasks in different domains and a set of operators that introduce additional complexity. Numeric functions (Section 2. and string functions (Section 2.2) are expressed under common API that supports sampling parameters and composing, corrupting, and approximating functions. For functions involving lexical semantics and factual associations, we implement synthetic neural modules with an LM backbone (Section 2.3). Table shows example functions from each category.
The FIND API and benchmark have been open-sourced under the MIT License and are available at: https: thub. wtt.hb..ooo.....oooeeeet.e...eeo m/l t mimodal
2.1 Functions with numeric inputs
Numeric functions test variety of capabilities needed interpret basic mathematical operations in neural networks, which can include relations between neurons or a model's representation of numeric
Table 1: Example FIND functions.
Category
Example Atomic Functions
Example Complex Functions
linear x + az +
Composition
periodic x ++ sin ( (x (x-c))
linear absolute + (ar b) [x]
absolute x + [x]
relu 27 + max 0)
Domain corruption
Numeric Functions
sqrt + vx
corruption x H |a|] if x [a, bj else
constant.z rational x + a+c
Observation noise
reciprocal x + 1/2
linear noise
b +
capitalize 'apple" + APPLE reverse 'apple' + 'elppa" replace(a,b) 'apple" + "ppple" shift_ "apple" + 'appld'
Composition
(relace(reeeree 'apple' 'elppb"
String Functions
reverse (shiftsshhit_last 'apple' + flypaa
capitalize 'apple' + BPPLE
Entities
Compasition
time and duration
time and duration OR winter sports
winter sports
winter sports OR plants and botany time and duration OR plants and botany
Neural Modules
plants and botany
Relations
Domain corruption
country + capital city
country + capital EXCEPT in Asia (return 0) gemstone ++ color EXCEPT if red (return 0)
gemstone + color
3

---

inputs. One example is Nanda et al. [2022], where an algorithm using trigonometric manipulations was found to perform modular arithmetic in a toy transformer model. We sample 1000 numeric functions from the FIND API for the benchmark dataset. 85% are parameterized atomic functions under noiseless and noisy conditions, and 15% are compositions.
Atomic functions are defined as explicit functions found in many mathematical and scientific computing libraries such as Python [Van Rossum, 2020], SciPy [Virtanen et al., 2020], and NumPY [Harris et al., 2020], as well as standard neural activation functions such as ReLU [Fukushima, 1975, Nair and Hinton, 2010]. Table shows examples of atomic functions defined in FIND, For each function in the set of atomic functions A. we sample native parameters, scaling factor a, and bias b. Parameters and sampling procedure details are provided in the Appendix and the FIND API.
Composition of atomic functions [(x) and g(x) applies an operator sampled from C = +). where f 9 f(x) g(æ) or f(x:) g(r)). Composed functions sampled from a subset Ac of atomic functions A to limit final complexity. f(x), g()) E Ac are described in the Appendix.
Observation noise added to f(x) tests how well the interpreter is able to estimate an underlying function in the presence of additive noise, and whether it is able to distinguish between different types of noise. 15% of functions f (ax E A in the FIND benchmark are sampled with additive noise f(z) + X, where X follows either a Normal, Uniform, or Poisson distribution.
Domain corruption replaces function values locally with random values. This is done either inside or outside of a sampled interval I of range [aa,b] (a, oo] [-o0, a]), where a ~ 41-100,100) The length of a finite interval is sampled from M[3.20)- Corruption is defined as noise X E [[,,, 0.01) replacing the function values I. We choose μl as the mean value of f(x) on its domain. The interpreter' is prompted to discover the corrupted interval, if any, and to return a and b. 15% of the functions (ax A in the benchmark dataset are corrupted on part of their domain.
Approximation of atomic functions implemented using two-layer neural network (MIP) with a ReLU non-linearity. For 15% of the functions in the dataset, we train MLP for 10k epochs 10k points uniformly sampled on its domain bounded by (-100, 100). Trained MLPs are provided in the benchmark dataset and loaded by the corresponding function during interpretation ?.
2.2 Functions on strings
We build on a long history of using toy problems strings [Hofstadter et al.. 1995, Hofstadter, 1995, Mitchell, 2021] and simple visual matrices [Lovett and Forbus, 2017, Wang and Su, 2015, Carpenter et al., 1990, Chollet, 2019] to test the ability of a system to reverse-engineer underlying symbolic operations. As this release of FIND is designed to evaluate language- we focus functions with text inputs and procedurally generate set of functions strings with different levels of complexity. The benchmark dataset contains 1000 string functions sampled from the FIND API, representing both atomic functions (30%) and compositions (70%).
interpretability systems,
Atomic functions include common string manipulation operations such as concatenate, replace, and reverse, and implementations of copycat problems from Hofstadter et al. [1995], such as shi ft_laastttttt (abe + abd). Example atomic string operations are shown in Table 1, and the full set can be found in the FIND API.
Composition of atomic functions f(x) and g(x) is defined as (g ()x) gf((((((())) The FIND API supports compositions where (g (z) is well-defined (i.e. function pairs where (g f)( = x are excluded). Example composition functions are shown in Table 1.
2.3 Synthetic neural modules
FIND includes set of synthetic neural modules that perform word-level tasks built from Wikidata properties [Vrandečić and Krötzsch, 2014]. We construct two types of text modules: functions involving lexical semantics and functions involving factual relations. FIND implements synthetic neural modules on text inputs using a single pretrained language model ( (Vicuna-133) instructed to apply different rules to inputs in response queries (See Figure 2). Vicuna [Chiang et al., 2023] is finc-tuned LLaMA model licensed under an Apache License 2.0 and open-sourced for non-
We provide an LM baseline where MLPs are treated as black boxes during interpretation, similar to the other functions in the dataset. An alternate paradigm could allow probing of internal activations and parameters.

---

Atomic relation
China
Prompt: am going to give you an input. If the input is a country, return only the capital of that country. the input a country, return undefined.
Interpreter
Vieuna
Beijing
Figure 2: Synthetic neural module implementation. Vicuna acts as backbone that generates the outputs of neural modules. Each function provides instructions to Vicuna that mimic the behavior of either single neuron ("returm an association score with predefined concept") or a more complex module ("map inputs to outputs according to a a predefined mapping"). " The interpreter then interacts with Vicuna as a black-bor box function.
commercial use. Each function in the dataset is built to interact with Vicuna to return the relevant output to the interpreter, which tests the function on different inputs to recover the underlying rule. We find that Vicuna reliably implements functions expressed in prompts (reliability scores are provided for each function type; see Appendix for full evaluation).
Type 1: Entities
Atomic entity functions fa( compare input text to reference concepts (entities) drawn from metaclasses of Wikidata properties related to a particular concept or subject area (e.g. related to food and eating) and return value indicating how associated an input is with the entity. Table 1 shows example entities, and the complete list of 140 atomic entities included in FIND is provided in the Appendix. The task of interpreter evaluating an entity function is recover and describe the underlying concept, or, what all of the inputs that produced high output values had in common. For example, if the entity is plants and botany, fe (garden) and fe (trre) should return high values, while fe (car) should return a low value. This task is relevant to lines of work that automatically summarize maximally activating exemplar sets to characterize neuron function [Bau al., 2020, Mu and Andreas, 2020, Hernandez et al., 2022, Bills et al., 2023, Singh et al., 2023], generalized to setting where the interpreter must also produce the data.
To implement each function, we instruct Vicuna to synthesize a binary response to interpreter queries, corresponding to whether an input word is associated (return 1) or unassociated (return 0) with the reference entity (see the Appendix for Vicuna prompts). The function output 8 = fe( is then a continuous scalar value representing Vicuna's internal probability of being the response token, between a choice of @ and 1. Specifically, s = /Po + P1), where p. = -' and logit, represents Vicuna's 's output logit for token i. To validate that Vicuna can reliably identify associations between inputs and reference entities, we collect buman-labelee each entity ey in the dataset, and compute in FIND, for 10 annottaed concepts per entity, is 0.85. We compute the same score for 10 distractor concepts per entity, sampled from the list of human annotations of other entities. The mean
set of concepts ž, associated with (.,). The mean value of á across all 140 entities
fey((;
score for the distractors is 0.08. See the Appendix for full experiment details.
Composed entity functions mimic the behavior of neurons inside deep networks that selective for multiple concepts [Fong and Vedaldi, 2018, Olah et al., 2020, Mu and Andreas, 2020]. We construct compositions of entity functions by sampling two entities from the dataset and instructing Vicuna to return if the input value to the function is associated with either entity. 60 composed entity functions are included in the FIND benchmark.
Type 2: Relations
More complex functions inside language models learn factual associations [Meng et al., 2022]. To mimic the behavior of these modules, we construct a set of relation functions that map inputs to outputs according to real-world factual association (for instance, river -> length gemstone -> color). Atomic relations are drawn from Wikidata properties included in the Wikidata dump provided by Sorokin and Gurevych [2018]. Table shows example relation functions; a full list is provided in the Appendix. Again we use Vicuna to implement a black box function that applies the rule in the relation to interpreter inputs (we verify that Vicuna returns factually correct answers, see Appendix). Figure 2 shows the prompt template for Vicuna and an example relation function.
Relation functions with bias include corruption to part of the function domain. FIND tests whether an interpretability model can uncover which parts of the function domain have been corrupted. Corrupted relation functions return undefined for a small region of the function domain; for example, we corrupt the country + capital relation on the subdomain Asia by prompting Vieuna to return undefined for inputs in the subdomain.
5

---

3
Evaluation protocol
The FIND protocol evaluates interpreters' natural language descriptions of functions in all cate- gories. In categories where domain corruption is introduced (numeric functions and factual relations), interpreters are evalauted on their ability to return a description of the domain of the function, indicating any subdomain where the function behaves differently. In the mathematical reasoning and strings categories, interpreters are evaluated their ability to output code that approximates the function. Section 4 provides details on the interpretation procedures we evaluate and how they are instructed engage with functions.
3.1 Evaluation metries
To evaluate the accuracy of function descriptions produced by an interpreter, indicators for individual function interpretations and calculate average success rates across all functions. For functions where interpreters write code approximating the function (numeric and string functions), we score the accuracy of the interpretation by running the interpreter's code representative test set and comparing the result to execution of the ground-truth function. To evaluate language descriptions of functions, we judge how well the interpreter's description agrees with ground-truth function behavior (for string functions and synthetic neural modules). Below we describe success indicators for each function category in more detail.
success
Evaluating code-based interpretations. Rumning interpretation the FIND benchmark produces descriptions of numeric and string functions code as well as natural language. We measure explanation accuracy by comparing performance of the interpreter's code to the ground-truth function implementation on a test set. For numeric functions, we compute a normalized mean- squared error [{(((x) g(x))) 2 /E[(f()) for [-128 S S 128] between the ground truth fIND function f and the interpreter's implementation g. and regard a successful estimation as one with Normalized MSE (NMSE) 0.1. For string functions, we run an exact-matching binary test [[(<string>))==ff (<string)))))))) on a set of 10 test inputs per function.
Evaluating language-based interpretations. We define a "unit testing" protocol where an LM evaluator selects which of three input-output (I-O) pairs corresponds to a language description of function. To judge the interpreter's accuracy, provide the estimated function description (e.g. f country + capital) to the evaluator, as well as three example I-O pairs: one execution of the ground-truth function (e.g. Germany + Berlin) and two randomly sampled distractors (I-O pairs for other FIND functions of the same type; e.g. Germany + Europe, ruby H red). The evaluator selects which I-O pair matches the functionality of the description from the interpreter. If the language description is accurate, the evaluator should select the ground-truth I-O pair as language descriptions of string functions and synthetic
the best match. We run this procedure neural modules for a test set of ten different triplets per function. Test sets are constructed to reveal representative behavior of each function using inputs inside and outside of the function domain. For relations corrupted on part of their domain (e.g. f country + capital, unless the country is in South America), the test set includes two ground truth examples from the corrupted subdomain (e.g. Peru + undef ined, Argentina + undefined). For entity functions that compute similarity to a reference concept (e.g. Greek Mythology), we provide the evaluator with only input concepts instead of I-O pairs (e.g. Athena, skiing, GPU), and ask which concept the function described by the interpreter selective for. As test cases can be designed to isolate specific function behaviors, we find the unit testing protocol to be more sensitive small differences in function descriptions (e.g. "transportation" vs "road transportation") than other cesscrpttionmmaehhi methods, such as having an LM directly grade the agreement between descriptions (see the Appendix for more details).
We finetune Vicuna-13b vricuua-evallaae tive samples matching descriptions of functions in the FIND dataset. LM-judges have been shown to be scalable and accurate surrogates for human judgments, which are otherwise expensive to obtain [Zheng et al., 2023]. While proprietary models such as GPT-4 demonstrate strong agreement with humans, using such models to compare interpreter performance on benchmark task incurs prohibitive costs associated with API access and poses reproducibility challenges as model versions are deprecated. We find that vicuna-evaluaton matches ground-Iruth function deseriptions to representative inputs and outputs more accurately than GPT-4, GPT-3.5, or pretrained Vicuna- 13b (see Appendix for evaluation). Furthermore, vicuna-evaluaton makes judgments that are highly correlated with those of human subjects performing the same task (see Appendix for experiment
to perform the unit testing task and select representa-

---

details). The fine-tuned vicuna-evaluate checkpoint and training dataset can be downloaded from the FIND repository. Training details are provided the Appendix.
Extensions: Targeted evaluation. For users of the FIND benchmark that produce function interpreta- tions in a structured format, this benchmark enables other evaluations targeted specific end-use cases for interpretability tools. For example, researchers may use FIND to explicitly evaluate whether interpreters can pick out the portion of the domain that is corrupted, whether they can identify components of composed functions, or whether they identified the noise model.
Automated Interpretability Methods
FIND can be used to evaluate any interpreter system that has the ability to execute Python scripts autonomously at the command line. As first demonstration, we evaluate several approaches inspired by recent work [Hernandez et al., 2022, Bills et al., 2023, Singh et al., 2023] that use pre-trained language models to perform interpretation. We run experiments using the OpenAl Python API. A custom plugin equips the interpreter model with the command NYTHON(function that it can use to call functions its own selection of inputs. Scripts for reproducing these baselines and complete interpretation dialogues are available in the FIND code repository.
py input)
We evaluate three different interpretation methods. (i) Non-interactive: interpretability tasks often involve descriptions of precomputed exemplars of function behavior. In this setting, the interpreter is equipped with fixed set of inputs to use for probing the functions, and is prompted to produce a description based only on function outputs for these exemplars, mirroring the non-interactive description paradigm used by MILAN [Hernandez et al., 2022] and recent MILAN-type LM anal- ysis approaches [Bills et al., 2023]. (ii) Automated Interpretability Agents: LM-based AIAS are prompted interact with the functions in FIND. The interpreter serves as an agent and runs (FTnNt(fucttinn py input) on inputs it selects. In the base setting, the agents' search process not initialized with pre-selected inputs: rather, the AIA forms hypotheses by itself, generates data to test its current hypothesis, and interactively updates it based on observations of function outputs. Figure 3 shows complete interactions between GPT-4-based AIA and example functions in two categories. At the beginning of the interaction, the agent is prompted to run experiments on the functions to produce data that can use to explain function behavior. Prompts for two categories of functions are reproduced below. (iii) We also evaluate combination of both approaches (AIA + MIlaN). In this setting, the interpretability agent prompted to begin interpretation with a provided set of exemplars, and then interactively continues the interpretation procedure.
All interpretation methods off-the-shelf L.s (GPT-4 [OpenAI, 2023], GPT-3.5 (GPT-3.5-tun 5-turb [Brown et al., 2020], and Llama-2 (Llama-22 1a:ma-2--133 --113- ahatttt,tttttc Llama-2-70b 1amma----0b-hhae -obbcchat [Touurron et al., 2023]) as interpreters, and operate without a human in the loop. ALA-based interactions show structured interpretation procedures, even when the final explanation is not fully accurate (see the predicted numeric function in Figure 3). The AIA makes and tests informed hypotheses about the functions: for example, testing densely spaced points in a specific interval to reveal a "separating point" in the behavior of the numeric function, and testing different forms of transportation determine whether a synthetic neuron is selective for land, sea, or air travel (Figure 3). Once the AIA has gathered sufficient data and produces final description, the run stops automatically. The Appendix further describes experimental settings.
Prompt for numeric functions:
H1, your job is interpret function f(x) that is implenented in the (function.py py) script. f()) takes nurerical inputs. All you can do is call F(x) on a value of by rumning PYTIDN((fucciinnpp) value). Try describe
function implemented by f(O) by running experinents on it. can call the function on multiple inputs at a tine by running PrTHON((function.p py) valuel value2 value3...). You can run Function multiple tines before writing description. You can test values betwren [-128 and 128]. The output could be noisy and your goal is to find a good simple description of f(x) that explains nost of the function behavior. f(X) nay conbine nultiple different operations. There nay be interval inside the donain where the function behaves differently. If that is the case, describe that interval. Your description of the function should have three parts. First.
some input. encourage testing a large range values
[DESCRIPTION]: Describe what the function is doing in language. [DCMAIN]: what the donain of the function? there interval behavior? what are the nuneric bounds of that interval? what is going on there? [CCDE]: Write Python code that agproxinates the behavior of the function.
function displays unexpected
7

---

Prompt for synthetic neural modules (entities):
Hi, your job is to interpret function f()) that is implemented in the (function. py) script. f(x) takes word inputs and returns a value between 0 and 1, where a value closer to 1 indicates the association with particular concept. Your task to figure out what that concept is. f(O) nay be selective for nore than one concept. If that is the case, describe all concepts that f(x) is selective for. Try to describe the function implenented by f(0) by running experinents on it. encourage age you to
diverse and randon selection of words: spanming nany different categories, including both comno and obscure wards, concrete and abstract words.
can call word1 word2 word3 ...). We encourage running many tests before writing description. The goal is to find a good simple description of f(x) that explains most of the function behavior. Your description the function should in the following forn. [DESCRIPTIONJ: Describe what the function is doing in language.
function on multiple inputs at
by rumning PYTHON (Yfunn((onnttion py)
Numeric function [(+) 4.5 conb((2n (5.91xx 5.7) 4 5.2
Synthelic neural module
ENTITY: road tronsport
AIA (GPT-4): To stud the investgation, lat' test the fluccion diffcrest points Bollowing these mapp: I. value less than 128 (ounside the allowed domain), A value of 1228, 3. value betwern 128 and 128 (e.g. 0) value 128,
AIA (GPT-4) by texting rundom words the FUNCTION command. THTHHFFUUCTTOON cat dog fish free car home))
128
the allowed domsin)
PYTHONIFUNCTION
126 129)
(Cst, 0.0866) (dog, 0.0013) (ftsh,, 0.0033) (tree, 0.0813) (ear, 0.9777) (hause, 0.0087)
RESPONSE: (-220, 9.7) (-128, 9.7) (9, 0.82025) (128, 0.7) (129, 0.7)
AIA (GPT-4k These outputs to have wide range of values. Let's PYTIOONFUUCCION love happiness sadness bicycle sirplane)
values oussie the allowed dumain sugnats that the fenstiun may roum a fixed valw (9.7) below -128 mnd another fixed value (0.7) above 128. Lats nos the funetion at differeent points within the domain to gather more datz: PYTHONEFUCTIOO 110 100 50 50 110)
AIA (GPT-4::
output
words find
EESvn,,: 0.0005) (hate, 0.0005) (happiness, 0. 0003) (sadress, 6. 9004) (bicycle, 0.4149) (sirplane, a. 2.3014)
>> RESPONSE: 9.7) (-188, 9.7) (- (-58, 9.7) (58, 6.7) (188, 0.7) (118, 0.7)
ALA (GPT-4): Based on the imput output pairs, we can assume that the finction bohaves differennly for values below and above some threahold between- -50 and 50. Lef's test the fanction for all integer valuee wetn -50 and find separating point. PYTHONFUNCTION -50 4 .9.49 50)
AIA (GFT-4): Imeresting seens cutput values for trmgommrso-eeltto objects. Let's nos more words relsted to Innraprrt and some random unrelated words to investigate further. PYTHON(FUNCTION trais bus bost walking runsing swimming television)
higher
RESPOIGE:(-100 9.7) (-49 ( 8.85) 5.m) (7 1.23)
(B 0.76) 8.7) (58 0.7)
>> RESPORSE: (bus,,) 8, 9,9763) (boat, 6 0.8884) (walking, e (rumiing, 0,6818) (swdrming, 0.0892) (1ion, 0. 6.000) a. 0.0032)
AIA (GPT-4k
on
integer values between 50 and
50, can observe the following For x -6. flxs 9.7 For -5 x 0, (Tx) decrosses mon-Iincarly. and there is some nose. - For 0 x. 50, fix) 0.7 To descritte the function:
ALA (GFT-4): Based on givea test results, seens funstion is scleotive for otpects relsed to ground Immsortttion, spuifitallly vehicles. La's test with fiw more wonks relvied to ground crnsprraation and other forms of transpoct to see if ais holds. PYTHONPPUNCTIO metorcycle scooter helicopter submerine spocochip)
[DESCRIPTION]: fix) return value of 9.7 forx less than equal and
non-lincaely with
retums o.7 x greater
[DOMAIN]: The unexpected behavioe interval when 6 <xx S.O.
This is when the function's output decrenses non- linearly and has some roise.
(CODE]
>> RESPONS:: (rotorcycle, B. 9519) (scooter, 0,535L) (rellsapter, B. B.0011) (submarine, 0.0012) (spaceshipp 0.0005)
"pythen
aporestrate functtor(s):
If
olie 5 B:
63/0)*13
i Accuning stople polyronta!
selective ligher value, closer to I, for words associated with ground transporation g. cur, bus, molorsycle, sccoter) and lower valuea, clnser to II, for unrelated wants other fumes tranppotaaion (eg.. helioopter, submarine, spacestip).
AIA (GPT-4): [DESCRIPTION]:
decresst, con further improvel slae: retum 0.7
ground transportation vehiclex.
Please note that the sppeosimmtion fimction may not coVEr every intricate detsil of the noil fimction, and can be fürther refined for the interval between ái and D.
Figure AIA function interpretation with GPT-4. The interpretability agent is able to conduct experiments that reveal the hehavior of an unseen function, by making hypotheses, selecting inputs that produce informative data, and updating hypotheses in light of new information. We include the full dialogues for two example functions: numeric function involving hyperbolic tangent (left), and synthetic neural module that returns continuous value between zero and one indicating input association with the ground truth concept road transport (right). Numeric function and synthetic neural module prompts were used at the beginning of the respective conversations, and have been omitted from the figure for simplicity. We overlay plot of the predicted numeric function and the real function for comparison. Points on the real function that GPT-4 sampled are marked. Final explanations compared to ground truth functions for evaluation (Sec Table 2).
8

---

Table 2: Interpretation success rates. For each function type we report the successful estimation rate (higher better) based on different indicators, and with different experimental settings (e.g. initialization with exemplars).
Code (exact match)
Language (unit test)
Numeric
Strings
Strings
Entities
Relations
AIA
AIA
AIA
AIA
MILAN
AIA +MILAN
AIA
MILAN
AIA +MILAN
1lama-2-113b-hher
0
0
0.33
0.34
0.54
0.58
0.46
0.45
0,42
Llama-2-700 2-7b-hhht
0.01
0.01
0.33
0.34
0.61
0.62
0.47
0.44
0,46
GPT-3.5
0.12
0.13
0.66
0.39
0.81
0.88
0.37
0.64
0.68
GPT-4
0.33
0.23
0.82
0.56
0.89
0.89
0.78
0.74
0.92
(a) Numeric function
(b) String function
[risst s
scems polyurmisl, sposifiallly ashic fanution. When the input negative, the corput is zegative, and when the inper positive the sutpur
ATA Description:
string not stringle strilgge,.tseppeae))
GPTT function unit camples
positive, There's a change is the function betovior around o, where the fanation goes from decrrasing incressing.
neturn atring
neture at stringg::]
cordeate)
Domain: domain
fuzztion firction
numbers.
neture "6 string:::)
There
elff scringtR) strinefs1]
around
minimum puint, negaive cupputs negaiive imputs and postiive cupurs for postiive inputs.
elte scringta) -T": # urap around froe
AMSE
string]t:)
Code:
der spprox, function(4): returs x (4 5] (x
4sf fuxctioelcetring):
.00
netum strangairingg
(c) Entity (composed):
AIA Description: The function fi) is seleotive for weeds essociated with
4sf foxciontstring:: reture
books
sports onganizaatins
books, reading, and places where books stored, retuning higher values
for weeds relatod conzpo. (Unit test
ABA Deseription: The fuucioo intreses de ASCII value
of the find clurntter of the inpur string by 2, and adds a deplionto of the las chomcter to the end the string
(i) Relation:
AIA Description: The function takes an asimal name inpur and retums the
inteal
> typtect babttst
primary habiut of unimal.
Unit test
1.0
test score
Figure 4: AIA (GPT-4) interpretations. Examples from all FIND categories, with evaluation scores (NMSE and unit test) marked red if below the success threshold (NMSE> 0.1, unit test< 0.33) and green otherwise.
5 Results
We evaluate the AIA method as well as the non-interactive baselines with different off-the-shelf LMs. Results are summarized in Table 2 and example function descriptions are shown in Figure 4. Additional example interpretations of functions in all categories are provided in the Appendix.
GPT-4 is a stronger interpretability agent than GPT-3.55 and Llama-2. Success rates for all function categories are reported in Table (AIA columns). The GPT-4 interpretability agent achieves universally higher success rates than GPT-3.5 and Llama-2 (which often score at chance value). In fact, we find that Llama-2 often invents the output of the function without executing it (see examples in Appendix). This is also reflected in the mean length of interpretation dialogues, which are 1.4, 1.4, 2.1, and 4.1 for 2lama-2-13b-chal count the number of interpreter-function interactions). We additionally observe that interpretations of string functions receive significantly higher scores using the unit testing protocol compared to Btring-matching outputs of estimated code. As unit testing selects for representative examples of function behavior and not exact descriptions, or descriptions with minor inaccuracies (see Appendix for additional discussion of unit testing limitations).
1amaa-2-70b-chat GPT-3.5 and GPT-4 respectively (we
the procedure more forgiving of less specific
Interactive vs. non-interactive. In addition to ATAS, we evaluate two other interpretation methods: MILAN [Hernandez et al., 2022] where the interpreter produces a description based only on function outputs for a given set of exemplars, and a combination of ATA + MILAN, where the same set of exemplars is used initialize the interpretation session, and the agent subsequently permitted to perform additional experimentation. For both settings, we sample the exemplars (two related the function, eight distractors) from human-construutted list of inputs associated with each function (see Appendix for details). In the non non-interactive MILAN setting, we observe a small improvement in performance over uninitialized AlAs; GPT-4 interpretation performance improves from 0.56 to 0.89 on entity functions, and decreases slightly relations, from .78 to .74. Initializing ATAS with MILAN exemplars and allowing additional experimentation dramatically boosts the performance of GPT-4 and GPT-3.5 agents (and also slightly improves Llama-2 performance), as shown in Table and Figure 5 (see init.). Notably, when initialized with exemplars, GPT-3.5 exhibits performance interpreting entity functions comparable to GPT-4. These results suggest that off-the-

---

shelf LM agents are limited by breadth of search. Indeed, LMs tend to start sampling with simple words (e.g. apple, dog, car) which do not reveal function behavior for highly specific reference entities (e.g. The New York Times, arachnids and arachnology). We view exemplar computation as one of many "tools" that an interpretability agent could use, and hope that this benchmark will drive exploration of additional tools (e.g. example synthesis) as part of automated interpretation methods. Procedures that combine initialization and interactive experimentation could improve the efficiency of existing labeling approaches that use large fixed datasets [Bau et al., 2017] to precompute maximally activating inputs, and potentially also surface novel behaviors not captured in predefined sets of exemplars.
Numsricc
Which functions can LMs recover? Fig- ure 4 shows examples of AIA interpreta- tions that successfully explain the behavior of some functions (e.g. cases (b), but fail to fully characterize more complex functions (e.g. cases (c), also reflected in some per- subcategory success scores (Figure 5). This is limitation of using off-the-shelf LMs as agents: they may miss small corruptions to part of the domain (Figure 4a), which in real- world interpretability settings could stem from bias in the training set. LMs could be outfitted specifically for interpretability with additional tools (e.g. for sampling) and further improved by fine-tuning.
Strings
0.8
Farities
Successa nite
Relations
0.6
0.4
0.2
0.0
elean Mt obs poise comapted
elean composed
composed
clean
composed iiii
-ini
clean commpted
clean-iii
comppedd
Figure 5: AIA interpretation scores by subcategory (with GPT Complex functions usually more dif- ficult interpret than atomic functions. String functions are evaluated using the string- indicator.
6 Related work
Explanation evaluation methods have previously benchmarked salience methods according to their ability to recover ground-trutt segmentations [Zhang et al., 2018, Selvaraju et al., 2017, Fong and Vedaldi, 2017, Yang and Kim, 2019], or identify inputs that causally affect outputs [Zeiler and Fergus, 2014, Petsiuk et al., 2018, Wagner et al., 2019, De Young et al., 2020]. Explanation benchmarks have also evaluated correlation with system performance [Adebayo et al., 2018, Casper al., 2023] or human understanding of decisions [Kim et al., 2022]. Our benchmark differs because we evaluate global explanations of black box functions instead of evaluating local explanations of decisions.
Model interpretability metries have quantified the degree model interpretability, i.e. by measuring how closely deep network features match a human-labeled concept [Bau et al., 2017, Kim et al., 2018, Goh et al., 2021, Wu et al., 2021, Burgess et al., 2018, Mu and Andreas, 2020, Geva et al., 202 l]. While these methods can measure disentanglement in model representations, they are only good as their ability to identify interpretable features. We tackle this problem by providing a benchmark for interpretation methods themselves, rather than the models that they explain.
Full-text explanation systems provide natural- language explanations for black box systems or their features [Hendricks et al., 2016, Camburu et al., 2018, Ehsan et al., 2018, Kumar and Talukdar, 2020, Hernandez et al., 2022]. Recent efforts exploit the capabilities of LMs to explain data directly, but these works utilize only tiny evaluation benchmarks, including 19 synthetic neuron puzzles in Bills et al. [2023] and 54 ground-truth module topics in Singh et al. [2023]. Motivated by the promise of this LM-driven approach and the need to quantify performance, our work provides comprehensive benchmark of full-text black-box explanation systems.
7 Conclusion
We introduce FIND, a new benchmark for evaluating automated interpretability methods. Baseline results show early evidence that agents built from advanced LMs can construct hypotheses and experiments to validate them, supporting the suitability of LMs as general-purpose interpretability backbones. However, we find many functions that LM agents cannot sufficiently explain, suggesting augmentation with additional tools will be necessary for robust automation of interpretability tasks.
10

---

Acknowledgments and Disclosure of Funding
We are grateful for the support of the MIT-IBM Watson AT Lab, the Open Philanthropy foundation, an Amazon Research Award, Hyundai NGV, ARL grant W911NF-18-22 88-2-2118, the National Science Foundation under Grant Nos. 2212310 and 2238240, the Zuckerman STEM Leadership Program, and the Viterbi Fellowship. The funders had no role dataset design, experimental design or analysis, decision publish, preparation of the manuscript. The authors have no competing interests to report.
References
Adebayo, Gilmer, M. Muelly, I. Goodfellow, M. Hardt, and B. Kim. Sanity checks for saliency maps. Advances in neural information processing systems, 31, 2018.
D. Bau, B. Zhou, A. Khosla, A. Oliva, and A. Torralba. Network dissection: Quantifying inter- pretability of deep visual representations. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 6541-6549, 2017.
D. Bau, J.-Y. Zhu, H. Strobelt, A. Lapedriza, B. Zhou, and A. Torralba. Understanding the role of individual units in deep neural network. Proceedings of the National Academy of Sciences, 2020. ISSN 0027-8424. doi: 10. 100.0//pa...9003751111 early/2820/08/ /19973777177
URL https: /www. wnwsw/www..w.po...o..../........ org/ content/
S. Bills, N. Cammarata, D. Mossing, H. Tillman, L. Gao, G. Goh, I. Sutskever, J. Leike, J. Wu, and W. Saunders. Language models can explain neurons in language models. https: http////ppaaaaaicc..o/o blob. core.
wwweeeeeoooeeeplleaeeeeeeeeee waapeerr/ppp///eeeeeeeeeeeeeeeee hwtr/xindxxeeee html, 2023.
T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam,
G. Sastry, A. Askell, S. Agarwal, A. feebeert-Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. M. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, Clark, C. Berner, McCandlish, A. Radford, Sutskever, and D. Amodei. Language models few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
C. P. Burgess, I. Higgins, A. Pal, L. Matthey, N. Watters, G. Desjardins, and A. Lerchner. Under- standing disentangling in f-vae. arXiv preprint arXiv: 03599, 2018.
O.-M. Camburu, Rocktäschel, T. Lukasiewicz, and P. Blunsom. e-snli: Natural language inference with natural language explanations. Advances in Neural Information Processing Systems, 31, 2018.
P. A. Carpenter, M. A. Just, and P. Shell. What one intelligence test measures: theoretical account of the processing in the raven progressive matrices test. Psychological review, 97(3):404, 1990.
S. Casper, Y. Li, Li, T. Bu, K. Zhang, and D. Hadfield- Benchmarking interpretability tools for deep neural networks. arXiv preprint arXiv:2302. 10894, 2023.
W.-L. Chiang, Z. Li, Z. Lin, Y. Sheng, Z. Wu, H. Zhang, L. Zheng, S. Zhuang, Y. Zhuang, J. Gonzalez, I. Stoica, and E. P. Xing. Vicuna: An open-source chatbot impressing gpt-4 with 90%* * chatgpt quality, March 2023. URL httph:://tmmet httmmsss.rrrbbbog/222--------
Chollet. On the measure of intelligence. arXiv preprint arXiv:1911.01547,20
A. Conmy, A. N. Mavor-Parker, A. Lynch, S. Heimersheim, and A. Garriga-Alonso. Towards automated circuit discovery for mechanistic interpretability. 2023.
J. Young, S. Jain, N. F. Rajani, E. Lehman, C. Xiong, R. Socher, and B. C. Wallace. Eraser: A benchmark to evaluate rationalized NLP models. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4443-4458, 2020.
Doshi-Velez Velez and B. Kim. Towards a rigorous science of interpretable machine learning. arXiv preprint arXiv: 1et.:0002000000 2017.
U. Ehsan, B. Harrison, L. Chan, and M. O. Riedl. Rationalization: A neural machine translation approach to generating natural language explanations. In Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society, pages 81-87, 2018.

---

N. Elhage, N. Nanda, C. Olsson, T. Henighan, N. Joseph, B. Mann, A. Askell, Y. Bai, A. Chen, T. Conerly, N. DasSarma, D. Drain, D. Ganguli, Z. Dodds. D. Hernandez. A. Jones.
J. Kernion, I. Lovitt, K. Ndousse, D. Amodei, T. Brown, J. Clark, J. Kaplan, S. McCandlish, and
C. Olah, mathematical framework for transformer circuits. Transformer Circuits Thread, 2021. html.
https:/traassoommerciiciitsssss
R. Fong and A. Vedaldi. Net2vec: Quantifying and explaining how concepts are encoded by filters in deep neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 8730-8738, 2018.
R. C. Fong and A. Vedaldi. Interpretable explanations of black boxes by meaningful perturbation. In Proceedings of the IEEE international conference on computer vision, pages 3429-3433 2017.
K. Fukushima. Cognitron: A self-organizing multilayered neural network. Biological cybernetics, 20 13-4):121-136, 1975.
M. Geva, R. Schuster, J. Berant, and O. Levy. Transformer feed-forward layers are key-value memories. In Proceedings of the 2021 Conference Empirical Methods in Natural Language Processing, pages 5484-5495, 2021.
G. Goh, N. Cammarata, C. Voss, S. Carter, M. Petrov, L. Schubert, A. Radford, and C. Olah. Multimodal neurons in artificial neural networks. Distill, 6(3)=300, 2021.
P. Grünwald and T. Van Ommen. Inconsistency of bayesian inference for misspecified linear models, and a proposal for repairing it. 2017.
W. Gurnee, N. Nanda, M. Pauly, K. Harvey, D. Troitskii, and D. Bertsimas. Finding neurons in a haystack: Case studies with sparse probing. arXiv preprint arXiv:2305.01610. 2023.
C. R. Harris, K. J. Millman, S. J. van der Walt, R. Gommers, P. Virtanen, D. Cournapeau, E. Wieser, Taylor, Berg, N. J. Smith, R. Kern, M. Picus, S. Hoyer, M. H. van Kerkwijk, M. Brett, A. Haldane, J. F. del Río, M. Wiebe, P. Peterson, P. Gérard- W. Weckesser, H. Abbasi, C. Gohlke, and E. Oliphant. Array programming with NumPy. Nature, 585(7825):357-366 Sept. 2020. doi: (0038//4158-000-26666 1038/s41 586-020 020-0029-222222222
K. Sheppard, T. Reddy,
URL https://doi.onn
L. A. Hendricks, Z. Akata, M. Rohrbach, J. Donahue, B. Schiele, and T. Darrell. Generating visual explanations. In ECCV 2016, pages 3-19. Springer, 2016.
E. Hernandez, S. Schwettmann, D. Bau, T. Bagashvili, A. Torralba, J. Andreas. Natural language descriptions of deep visual features. In International Conference on Learning Representations, 2022. URL https: //axxiv...v////// wtt.t//ogg//////oe//ooooooo.ss/t 2aads/2//....//2 2201. 11114.
D. R. Hofstadter. Fluid concepts and creative analogies: Computer models of the fundamental mechanisms of thought. Basic books, 1995.
D. R. Hofstadter, M. Mitchell, et al. The copycat project: A model of mental fluidity and analogy- making. Advances in connectionist and neural computation theory, 2:205-267, 1995.
S. Hooker, D. Erhan, P.-. Kindermans, and B. Kim. A benchmark for interpretability methods in deep neural networks. Advances neural information processing systems, 32, 2019.
B. Kim, M. Wattenberg. J. Gilmer, C. Cai, J. Wexler, F. Viegas, et al. Interpretability beyond feature attribution: Quantitative testing with concept activation vectors (TCAV). In International conference on machine learning, pages 2668-2677) PMLR, 2018
S. S. Kim, N. Meister, V. V. Ramaswamy, R. Fong, and O. Russakovsky. Hive: evaluating the human interpretability of visual explanations. In ECCV 2022, pages 280-298. Springer, 2022.
S. Kumar and P. Talukdar. NILE: Natural language inference with faithful natural language explana- tions. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8730-8742, 2020.
H. Lightman, V. Kosaraju, Burda, H. Edwards, B. Baker, T. Lee, J. Leike, Schulman, I. Sutskever, and K. Cobbe. Let's verify step by step, 2023.
12

---

Z. C. Lipton. The mythos of model interpretability; In machine learning, the concept of interpretability both important and slippery. Queue. 16(3):31-57, 2018.
A. Lovett and K. Forbus. Modeling visual problem solving as analogical reasoning. Psychological review, 124(1):60, 2017.
A. Mahendran and A. Vedaldi. Understanding deep image representations by inverting them. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5188-5196, 2015.
K. Meng, D. Bau, A. Andonian, and Y. Belinkov. Locating and editing factual associations in GPT. Advances in Neural Information Processing Systems, 36, 2022.
Miller. Explanation in artificial intelligence: Insights from the social sciences. Artificial intelligence. 267:1-38, 2019.
M. Mitchell. Abstraction and analogy-making in artificial intelligence. Annals of the New York Academy of Sciences, 1500(1):79915)) 2021.
Mu and J. Andreas. Compositional explanations neurons. Advances in Neural Information Processing Systems, 33:17153-17163, 2020.
V. Nair and G. E. Hinton. Rectified linear units improve restricted boltzmann machines. In Pro- ceedings of the 27th international conference on machine learning (ICML-10), pages 807-814, 2010.
N. Nanda, L. Chan, T. Lieberum, J. Smith, and J. Steinhardt. Progress measures for grokking via mech- anistic interpretability. In The Eleventh International Conference on Learning Representations, 2022.
T. Oikarinen and T.-W. Weng. Clip-dissect: Automatic description of neuron representations in deep vision networks. International Conference on Learning Representations, 2023.
C. Olah, Mordvintsev, and Schubert. Feature visualization. Distill, 2(1 2017.
C. Olah, N. Cammarata, Schubert, G. Goh, M. Petrov, and S. Carter. Zoom in: An introduction to circuits. Distill, 5(3):e00024 001, 2020.
OpenAl. Gpt-4 technical report, 2023.
V. Petsiuk, A. Das, and K. Saenko. Rise: Randomized input sampling for explanation of black-box models. In Proceedings of the British Machine Vision Conference, 2018.
R. R. Selvaraju, M. Cogswell, A. Das, R. Vedantam, D. Parikh, and D. Batra. Grad-cam: Visual explanations from deep networks via gradient- based localization. In Proceedings of the IEEE international conference on computer vision, pages 618 626, 2017.
C. Singh, A. Hsu, R. Antonello, S. Jain, A. G. Huth, B. Yu, and Gao. Explaining black box text modules in natural language with language models, 2023.
D. Sorokin and I. Gurevych. Modeling semantics with gated graph neural networks for knowledge base question answering. In Proceedings of the 27th International Conference on Computational Linguistics, pages 3306-3317, 2018.
H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale, D. Bikel, L. Blecher,
Ferrer, M. Chen, G. Cucurull, D. Esiobu, J. Fernandes, J. Fu, W. Fu, B. Fuller, C. Gao, V. Goswami, N. Goyal, A. Hartshorn, S. Hosseini, R. Hou, H. Inan, M. Kardas,
Korenev, P. Koura, M.-A. Lachaux, T. Lavril, J. Lee, D. Liskovich, Y. Lu, Y. Mao, X. Martinet, T. Mihaylov, P. Mishra, Molybog, Y. Nie, A. Poulton, J. Reizenstein, R. Rungta, K. Saladi, Schelten, R. Silva, E. M. Smith, R. Subramanian, X. E. Tan, B. Tang, R. Taylor, A. Williams, J. X. Kuan, P. Xu, Z. Yan, Zarov, Y. Zhang, A. Fan, M. Kambadur, S. Narang, A. Rodriguez, R. Stojnic, S. Edunov, and T. Scialom. Llama 2: Open foundation and fine-tuned chat models, 2023.
Kerkez, M. Khabsa, I. Kloumann,
G. Van Rossum. The Python Library Reference, release 3.8.2. Python Software Foundation, 2020.
13

---

J. W. Vaughan and H. Wallach. A human- centeree agenda for intelligible machine learning. Machines We Trust: Getting Along with Artificial Intelligence. 2020.
Virtanen, R. Gommers, T. E. Oliphant, M. Haberland, T. Reddy, D. Cournapeau, Burovski, P. Peterson, W. Weckesser, J. Bright, et al. Scipy 1.0: fundamental algorithms for scientific computing in python. Nature methods, 17(3):261-277) 2020.
D. Vrandečić and M. Krötzsch. Wikidata: free collaborative knowledgebase. Communications of the ACM, (700))778 85, 2014.
Wagner, M. Kohler, T. Gindele, L. Hetzel, J. T. Wiedemer, and S. Behnke. Interpretable and fine- grained visual explanations for convolutional neural networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9097-9107, 2019.
K. Wang and 7. Su. Automatic generation of raven's progressive matrices. Twenty-fourth international joint conference artificial intelligence, 2015.
K. R. Wang. A. Variengien, A. Conmy, B. Shlegeris, and J. Steinhardt. Interpretability in the wild: circuit for indirect object identification in gpt-2 small. In The Eleventh International Conference Learning Representations, 2022.
Wei, X. Wang. D. Schuurmans, M. Bosma, Xia, E. Chi, Q. Q.V. Le, D. Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems, 35:24824 24837, 2022.
Z. Wu, D. Lischinski, and Shechtman. Stylespace analysis: Disentangled controls for stylegan image generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages
2021.
M. Yang and B. Kim. Benchmarking Attribution Methods with Relative Feature Importance. CoRR, 0bs//190.0990 09701, 2019.
S. Yao, D. Yu, J. Zhao, I. Shafran, T. L. Griffiths, Y. Cao, and K. Narasimhan. Tree of thoughts: Deliberate problem solving with large language models, 2023.
M. D. Zeiler and R. Fergus. Visualizing and understanding convolutional networks. In ECCV 2014, pages 818 833.. Springer, 2014.
Zhang, S. Bargal, Z. Lin, Brandt, X. Shen, and Sclaroff. Top-down neural attention by excitation backprop. International Journal of Computer Vision, 126(10):1084-11 1102, 2018.
L. Zheng, W.-L. Chiang, Y. Sheng, S. Zhuang, Z. Wu, Y. Zhuang, Z. Lin, Z. Li, D. Li, E. Xing, et al. Judging LLM-as-a-judge with MT-bench and chatbot arena. arXiv preprint (XXi:23306006633 2023.
B. Zhou, A. Khosla, A. Lapedriza, A. Oliva, and A. Torralba. Object detectors emerge in deep scene cnns. arXiv preprint arXiv: 6852, 2014.
14

---

Appendix
Supplemental Materials for FIND: A Function Description Benchmark for Evaluating Inter- pretability Methods. This appendix provides hosting, access, and maintenance details, and additional information about the FIND dataset and experiments described in the main paper.
A Limitations
We view FIND test of necessary, but not sufficient, capabilities for automated interpretation. The ultimate test of these interpretation methods' effectiveness must be their ability to generate actionable insights about real models, which FIND does not evaluate. But clean, simple benchmarks with ground truth answers have been a major driver of more general capabilities in LMs, and we hope that FIND can play similar role in interpretability research. Additionally, the current release of FIND only includes black-box interpretation problems. This style of problem is relevant to many existing automated procedures such as NetDissect [Bau et al., 2017] 7] and MILAN [Hernandez et al., 2022], which treat single neurons as black boxes; however, most interpretability work occurs in a white-box setting. We intend for FIND be extended in the future incorporate white-box function interpretation problems, including descriptions of individual components of neural circuits (the IOI circuit from Wang al. [2022] just one example), some of which could be represented as the types of composition problems that are already included in FIND, but where interpreters access and label individual sub-computations inside composed functions.
B Ethics statement
FIND includes entities drawn from Wikidata that represent real-world concepts like video games, paleontology, and airports, as well as more potentially sensitive topics like The Holocaust, World War II, and disasters (see Appendix D). We include these topies in FIND because they relevant to behaviors inside neural networks trained on real-world data, that we want automated interpretation procedures be able to describe. We additionally note that interpreters which test the semantic similarity between input concepts and reference entities may surface surprising or controversial similarity scores that stem from learned biases inside the LM backbone of the synthetic neural modules (e.g. men having a higher similarity score with mathematics than women). Advanced interpretation procedures would be able name and describe these biases, but the FIND evaluation protocol currently does not test ability to discover biases other than those included explicitly in FIND (via corruptions to function subdomains).
C Additional interpretation examples
Below provide additional examples of AIA interpretations performed with GPT-4 agent for each of the FIND categories. For numeric functions, we report the NMSE score (lower is better) marked in green if the score is below the success threshold, or red otherwise. For strings and synthetic neural modules, we report the unit test score (higher is better) marked in green if the score is above chance, red otherwise. Complete interpretation dialogues are available in the FIND code repository.
15

---

[(x) 2.3tanh
2.7
2x
0.6)
+ 7.4 2
[ax 2[0] +1(11.1)
11
10
100
0-
-100
-200
NMSR
NMSF .0001
-100
100
-100
-300
100
(GPT-4): The function hehaves differently for positive and negative input volses. For nogative valses, the finction scams to penduce cuput values close to the 4-6 mage. For positive input values, the fanction appears produce outputs in range.
(GPT-4): function fix) appears function with some noise. The output increases as the impul incecases, and vice versa.
2 4.4 2)
(x) (699,, 0.1) if 91.7 < -78.7 otherwise
f(2)
10v2m
exp
10
95.8
NMSE 5.30-8
NMSR 0.2
0.035
-95.9
0.010
0.020
-96.1
0.015
-96.2-
0.000
-100
-50
100
-100
-50
50
100
AIA (GPFT-4): The function always outpuis the constant value of 96.00 regurdiess of the input value.
AIA (GPT-4): The function flx) resembles bell
reaching peak around symmetriaally decressing on both sides.
0.089
f(x)
(xx -31x" 1.90 + 2.7 1.7 + (0),
1e6
NMSE 0.8
NMSE L.1
2-
0-
-2-
-6
-100 -50
50
100
AIA (GPT-4): The function (x) takes numetical input and returns a value that propurtional to the reciprocal of the For negative cotputs, and for posttive inputs. returns negative outputs. The function does not retur any ourput for the input value of
AIA (GPT-4): The functica fix) is predominandl cubic in behaviur. For negative imput values, the function gives bocomes negutive.
the function returns positive
ourput values, and magnitude incresses
positive inpui values, the function gives neguzive output values, with the magnitade incressing becomes more positive. The fanction's belavior
somowhut unusual, following the cabie pullen, but occrall teend renmins similar to a cubric finction, The cutput noisy and doesn' follow perfeet cubic pattem.
exactly
AIA fanction
function
AIA samples
Figure A1: Numeric function interpretation examples. Plots compare the code interpretation to the ground- truth function implementation. Points on the FIND function sampled by the GPT-4 AIA are indicated. NMSE scores are marked in green if the score is below the success threshold (0.1), 1), or red otherwise.
16

---

Synthetic neural modules: Entittes
physics
AIA (GPT-4) faection aclenes, technstogs. and assciation. Wonds suc "rodus" and "aalaay" have sinifoonnly higher values then words unrelated those docaatts While sone other warts like "woter" and "lookronn" sho lave releivaly higher waus,, indicetig possble moccialins with sicatific umappo in gaaad.
selectvee higher values indicting
relond
ATA (GPT-4) 4y: The Baxciion XX) appears to seectivve fr edesstal badies (ety exn, moon, stans) and seperte of dly-d phezoereea (lights zesgre ligher valiee coneepts ralated sky, coloital objocs, and light, lover values ue sisigned
unrelated ouscopts.
Liit ttll soure
soore
peetry
Soursslton
AlA (GPT-4): The faruticn appices in he sclazive for absrract concupts relitiel is caodions, rehationship, and nature. Highoccenin words include lose, moos, lapplaess, sudiess, fizadaii,, pence, sus, boaut, and kisdeess, Loovesccengg words are prodommentty neated tanghlee object, ruch as car, computer, fino,, and stizece. fanction madgp higher vallas to words assciated with iborect sotioes, isigging Jower valuos concrete specific
AIA (GPP-4): The fanction (ts) aepear in he seleative ber mubtiple concepts, sads us: Ensoiieo and fecling ica. sal (xpecssio)) 2. Cubes (ca. baa) Natual plenosensa related to water (tg, ocess, taii) 4. Technoiogs and programingg (e aggntinn progamming, congutrr) 5, Temperaure (e hac) Canflies and commiizaiion (c&, anflis, commurcctiio, was, laspaoge) Must of woods relited to these coteepts groenue lightr ougul indloring stronger
with respotive concepts
00.5
Unit
Synthettc ncural modules:: Relations
country t cupital
andeo]
sypical habitat
ATA (GPT-4) Unable to desemine the nálo applied by the fanction due to tre Ireaificinn and incoedturrr ouppats
ALA (GP1-43: The fanetice fix) sourre to match iepus word with robind Inbiial eavinemens For cuampie, when the ispoi an uninal object, reurs the bablut (eg. "Soo' "Severou", "Matpkii" "Okeax".
Unitizd sarec (13
"caple"
Unit sert score
country 1- boffictel language
ALA (GPT-4) The diacton takes taper word (typicaly a city name) and naturs the nime of the cxrttry or U.S. eate wescorrd with dust city if
AIA (GPT-I): di The finction f(s) tikes a place (eity, country etc.) ze inpur and retunn in soccianal langaage as the cuppul with peoper panclaation. For issanco, ie input city the Packs, (00 ouguts Freach!
urmeciion is procel wiltn dee fanction's knowdeige, For aams ixputs bowove, the fuodicn does noll recogtize the essociation and returas todefaod.
8.3
Unit
"]
Figure A2: Synthetic neural module interpretation examples. Unit test scores are marked in green if the score is above chance (0.33), or red otherwise.
def functiont(strinn):
(unrtion
lettor 'E'
letter 'x'
if string.endswith (letter):
string letter string
if not srringgendowith((ettte)) string +a letter return string
def function2(strdnn):
function2(string):
words sp1itt)
reversed words [word[::-1] for word in words] return (oeverreee words)
digit =
'é": 'zaro", '1': 'ane', '2'; 'tue', '3":
'three', '4'; 'four", 's': 'five', '6': 'six', '?': 'sóven', "g": 'eight', 'gr 'ntre', '10": 'ten', '11'; "eleven' '12': "twelve" '13'; 'thirteen' '14": 'faurtsen' '15": 'fiftean', 'sixteen', '17": 'seventeen', '18"; "eighteen', , '19": "ninetsen' 128": 'tuenty' '21: 'tuenty-one', '22': 'oentyy-to', '23's "tventy three'. '2'": "taanty-faur', '25": "iventy-five", '26'; "teetty-sx",
functson(strrnng):
return function1( (function((strinnn)
ALA (GPT-4):
The fimetion takes an imput string, reverses it, and appends the reversed string.
Unit test 1.0
first_ character string[0] lower)( "zara'
if (orrs coffisst_chhracttr [salgha()):
alphabet 'ébcdffgnikklnnsrrs stuupprssttway" order alphabet. index character) return
def :unction!(strinn):
'g' ald new 'k'
def function(string):
return functioni (ssrct)
return (eringgrepplcc(01d..ew)
ATA (GPT-4): The function takes string input and calculates sum of the index positions of each character in the string scconding the English alptuibet, then appends "x" to the result,
ALA (GPT-4): The function input string culpul
Unit test score = 0.1
Unit test score 1.0
Figure A3: String function interpretation examples. Unit test scores are marked in green if the score is above
chance (0.33), or red otherwise.
17

---

D FIND documentation
The FIND API, benchmark dataset, and associated metadata are open-sourced under MIT license and available at: https://gi thub. wttp...ooom........co/....mo..at timodal - inteppreabblllyyyFF....
The FIND repository contains the utilities necessary for reproducing benchmark results for the LM baselines reported in the paper, and running and evaluating interpretation of the FIND functions with other interpreters defined by the user.
The FIND dataset itself is hosted on Zenodo with DOI: 10.528 10.5281/zenodo. l/zenodo. / The GitHub README provides instructions for downloading the dataset file into the appropriate directory for use with the FIND utilities.
Responsibility statement. The authors affirm that they have full authority to license the dataset under the terms of the MIT License and they accept all responsibilities for the dataset.
Dataset. Each function is expressed as independent.. py file that can be called by the interpreter with the FIND utilities. The benchmark dataset includes:
1. 1000 numeric functions sampled from the FIND API (see section
2. 1000 string functions sampled from the FIND API (see section F)
3. 275 synthetic neural modules with LM backbone (see section G
An interpreter can tested on a single function or the entire FIND dataset using collect_ betpett_innerreetttinssssoo Most of the functions self-contained see one example here ffnn
py):
import numpy import sys
def function(x):
return (-4.5tx=e4 0.7exmm3 - 2,6xx=*2 0.4xxa*1 3.9)
(-25.3 (heree x, 0.3) 5.2)
1f hamee..... == -mii_nain_.'. outputs
for arg in sys. argv
float(arg)
try:
function(x)
except:
out "None"
outputs ((arg), (out))
pf "Functiio
Function input output pairs: [outputs}')
For numeric functions that load trained MIPss, the dataset includes pt file with model weights the same folder function_ code. py. Other functions (synthetic neural modules) that are implemented with an LM backbone rely on calls to self-hosted Vicuna inference server. For example, this function hand_ddaasee... taaast//eeuoos.c.mm int ties/f000 3/ e/ffunciion //ffuctiionncoe py) simulates a neuron selective for works of fiction by measuring Vicuna output logits:
18

---

inport openai inport sys
numpy
inport torch
openai. ápi_key "EMPTY" openai. bps_ (ASI] model vicuma-13b-vl..
from transformers import Llamatokenizer
takenizer LlamaTokenizer zero_token_id ton.e
one_token_id tokenizer... cor.rrrr_ccorrtttttttooo......
concept works of fictian'
finction prompp))):)):
messages=[
("role": "system", "content": sys_prompt),,
("role": "user", "content": Vicuna pronpt for function
F'I am going to give you an input. Return only V"IX" if that input is related to (concept). "Return only \"at" otherwise.V Do not return any additional information other than the 8 or answer An" f"Input: (input}")
input interpreter
completion openai. ChatCompletion.creeee model=model,
nessages:messages, temperature=0.9
logprobs nr.array (completion. np.arrayycompeetion. choices [0]. logprobs)
output_tokee tokens takenizer encode (completian. choiress[[ nessage content, indices = tarch. logical_or
indices indices. (owttultutouuuottteeoooo (output_tokens == zero_taken_id, output_tokens == one_token_id)
index_of of ceeex__e_chiice__e
selices.nonzerot. index_of_choict [0][1]. [0][1).item()
index index-1
p np. np.exp(logprobi [index. one token_id]) / (np.exp (ippeeseloppoobsssiineee pro_token_id1 op.exx((ogprabssinnee ane_token_id]))
return prob_1
name.... '
outputs
prompt = "You are an trained to identify words related to input concepts." for in sys.argv[1:]: print(f
outputs += ((arg) (function(arg, prompt):.4f))
input
pairs: (outputs)')
Calling these functions requires running inference on Vicuna. Instructions for setting up and running a Vicuna inference server can be found the GitHub README. Vieuna is open-sourced (Apache License 2.0) for non-commercial use. Vicuna is also subject to the LLaMA model license (GNU General Public License v3.0), the OpenAl Terms of Use, and the ShareGPT Privacy Practices.
Reproducibility, Many of the functions in this dataset are self-contained; the ones that are not can be reproduced with run in the cloud (to run the interpretation baselines in this paper, we used 2x NVIDIA GeForce RTX 3090 GPUs). Running the evaluation requires hosting vicuna-evalusa vicuna-eval eeanna-tvallttor checkpoint for download via the FIND GitHub. All scripts for reproducing evaluations reported in the paper are available in the GitHub repository, and we discuss comparisons to other LMs as evaluators in section F.
which open-sourced and accessible research GPUs, or can be
We provide the finetuned
E
Numeric function API details
The API for generating numeric functions be found at:
https://gi thub. com/mut /ul timodal-innee httpr/aaaliiite//////////eeeeeee
The library of numeric functions is located in the script math_functions py, which defines indi- vidual functions the set of atomic functions and randomly samples their parameters. We also randomly sample integer scale and bias parameters between [-30, 30] for each function.
19

---

15% of the numeric functions in the FIND dataset are compositions of two atomic functions f(z) and g(z). To limit the complexity of the compositions, we choose f(x) and g(z) from a subset Ac of atomic functions, where Ac = linear, polynomial, step, RELU, constant, ceiling, floor, rectangle, square wave, and combine them using only mul- tiplication and addition.
FIND is extensible: the API can be used as general numeric function generator, allowing the user to make other design choices targeting specific interpretability applications.
String function API details
The API for generating string functions can be found at: https://git thub. com/mul timodal
All functions and sampling parameters used to create the FIND dataset articulated in sering_function siciionspp....... py. Like with numeric functions, we define subset (described the API) of string functions with lower complexity in compositions.
G Synthetic neural modules
To validate the reliability of Vicuna backbone for the synthetic neural modules, score its execution FIND functions on human labels associated with each entity in FIND, and ground-truth factual relation pairs (extracted from Wikidata).
D1 Entities
For each entity, we construct list of 10 associated concepts (e.g. for the climate entity, associated words include humidi ty, tempera ture, atmosphere). In Table I we list all entities included in FIND, and report the mean function output across 10 human labels associated with each entity (function output is computed from Vicuma output logits) and the mean function output for 10 labels randomly sampled from the rest of the dataset. A score closer to 1 indicates stronger association between input concept and reference entity. For all entities in FIND, Vieuna scores reliably distinguish between associated and unassociated concepts. We additionally show selectivity each entity function for concepts associated with that entity relative all other entities in Figure A4.
Table A1: Atomic entities and Vicuna scores Synthetic Neural Modules
Entity
Same Entity Score
Random Entity Score
sport organizations
0.996
0.112
corporations
0.968
0.053
elections
0.670
0.012
musical works
0.978
0.110
films
0.992
0.023
climate
0.810
0.009
computer hardware
0,972
0.004
physics
0,992
0.021
religions and beliefs law and justice
0.899
0.013
0.995
0.104
transport
0.786
0.129
education
0,783
0.122
politics
0.508
0.029
works of fiction
0.971
0.087
games and leisure activities biology
0.763
0.136
0.994
0.168
mathematics
0,993
0.020
geology
0.993
0.044
government and state
0.644
0.067
food and eating
0.855
0.101
transport
0.763
0.011
road transport
0,787
0.027
20

---

Table A1 continued from previous page
Entity
Same Entity Score
Random Entity Score
rail transport
0.672
0.001
cycling
0.846
0.011
the New York Times
0.703
0.163
aircraft
0.737
0.003
bodies of water mineralogy
0.992
0.094
0.984
0.054
occupations
0.976
0.178
professional wrestling
0.872
0.004
quantity indicating percentage
0,996
0,148
plays
0.546
0.149
horses
0.964
0.005
photography
0.977
0.026
music
0.965
0.038
fashion
0.898
0.154
chess
0.700
0.004
racket sports
0.828
0.129
0.991
0.457
disasters
0.687
0.022
spacecraft
0.915
0.018
video games
0.991
0.035
the relationship of element its class
0.893
0.708
basketball
0.767
0.009
winter sports
0.984
0.002
American football
0.849
0.036
golf
0.925
0.026
baseball
0.893
0.006
ice hockey
0.663
0.003
women and feminism
0.819
0.120
lighthouses
0.493
0.010
tennis
0.804
0.008
sports
0.958
0.014
comics
0.637
0.012
algorithms
0.801
0.021
tourism
0.690
0.186
bridges
0.685
0.012
books
0.774
0.111
linguistics
0.995
0.464
utilization and ownership
0.402
0.206
online communities
0.889
0.084
television
0.851
0.023
astronomy
0.995
0.054
disability
0.637
0.023
proteins
0.399
0.072
human anatomy
0.966
0.053
gymnastics
0.751
0.026
architecture
0.800
0.055
sculpture
0.795
0.030
archacology
0.836
0.083
theater
0.899
0.104
dams
0.776
0.012
birds and ornithology
0.993
0.070
computing
0.988
0.302
time and duration
0.699
0.052
age
0.832
0.040
weapons and military equipment ratios and proportions
0.820
0.003
0.446
0.155
typefaces and typography
0.911
0.094
burials graves and memorials processes and manufacturing
0.878
0.010
0.645
0.058
paleontology banking
0.934
0.054
0.922
0.034
lakes
0.552
0.060
21

---

Table A1 continued from previous page
Entity
Same Entity Score
Random Entity Score
natural science
0.970
0.359
geography
0.981
0.039
insects and entomology
0.897
0.033
philosophy
0.168
poetry
0.991
0.071
encyclopedias
0.642
0.350
television shows awards prizes and honours
0.914
0.095
0.997
0.159
the Middle Ages
0.871
0.053
plants and botany
0.994
0.189
marine biology
0.996
0.059
color
0.780
0.080
airports
0.712
0.003
science
0.946
0.284
anime and manga
0.969
0.094
Christianity
0.988
0.015
Buddhism
0.993
0.002
Greek mythology
0.949
0.013
Judaism and the Jewish people
0.965
0.012
music genres
0.987
0.144
fictional characters
0.892
0.144
rap and hip hop Islam
0.693
0.055
0.949
0.016
The Walt Disney Company
0.937
0,005
agriculture
0.982
0.104
hiking
0.384
0.018
New York City
0.969
0.113
gardens
0.896
0.093
personality traits
0.879
0.172
camping
0.728
0.065
gender
0.555
0.049
cemeteries and graves
0,984
0.027
London
0.954
0.105
Los Angeles
0.949
0.042
Chicago
0.959
0.047
Paris
0.789
0.017
Berlin
0.923
0.052
sailing
0.897
0.001
swimming
0.901
0.015
the Holocaust
0.788
0.007
arachnids and arachnology
0.798
0.128
musical instruments
0.997
0.042
meteorology
0.926
0.011
disease
0.787
0.010
Ireland
0.784
0.020
libraries
0.612
0.122
muscums
0.848
0.058
journalism
0.883
0.020
buildings
0.795
0.153
cryptocurrencies
0,951
0.001
events and news
0.725
0,229
Louisiana
0.745
0.012
revolutions
0.623
0.076
mines and mining Switzerland World War
0.918
0.039
0.947
0.050
0.986
0.064
Average
0.846
0.079
22

---

0.316
Reference entity used in function
0:014
Mean function response
0.010
3,008
Function inputs associated with this entity
Figure A4: Entity functions implemented using Vicuna respond more strongly concepts related to their reference entity (diagonal) than concepts related other entities (off-diagonal). The value row and column j corresponds to the average output of the entity e, function for 10 inputs associated with entity e,. Each row is normalized using softmax. The prompt template used to instruct Vicuna to implement each function shown in Section G.
23

---

Table A2: Factual relations and domain corruptions
Input
Output
Domain corruptions <case 1, case 2>
country
capital of country
country <Asia, S. America>
city
country where the city is located country where that person was head of state country where monument is located continent where monument located state where the city is located official language of the country occupation of the person most likely gender of the person
city is in <Mexico, The U.S.>
head of state
head of state is <man, woman> monument in <New York City, France> monument <S, America, Australia> city is in <California, Texas> country in <N. America, S. America> person <woman, man> name begins with <T, country <Asia, Europe>
monument
monument
city in the U.S. country
well-knoww person
name person country
continent where the country is located composer of the opera author of the book another country its border political party the politician continent where the river is located time zone of the city language of the opera main food source of the animal typical habitat the animal color of the gemstone color of the country's flag
opera
opera is in <Italian, German> book written <pre-1900, post-1900> country is in <Africa, N. America> politician a <man, woman> continent is <Africa, Europe> city is in <N. America, Asia> opera is in <Fnglish, Italian> animal a <carnivore, herbivore> animal lives on <land, the ocean> gemstone is <red, blue>
title of a book
country
politician
river
city
opera
animal
animal
gemstone
country
country is in <N. America, Asia> river is in <N. America, Europe> country
river
length of the river
country
maximum elevation in the country height the tallest skyscraper in that country height of the average person in that country
<S. America, Africa> country is in <N, America, Asia> country is in <Europe, S. America
country
country
D2
Relations
FIND includes 75 synthetic neural modules that map inputs to outputs using real-world factual relations drawn from Wikidata properties. The set of relation functions is built from 25 atomic relations (e.g. country >> capital) and two domain corruptions per relation, where the function returns undefined for small region of the domain (e.g. country + capital EXCEPT if country South America. then return undefined, Table A2 lists atomic relation functions included in FIND, and the two domain corruptions applied to each function. We evaluate Vicuna accuracy applying relation functions to input concepts by scoring the accuracy of the mapping applicd to 10 inputs per function, where inputs drawn from concepts in Wikidata with the property applied by each relation function. Mean Vicuna accuracy across relations in FIND is 91.2%.
E
Interpretation
Utilities for reproducing interpretation experiments and adding other, user-defined interpreters provided in the GitHub repository. We report interpretation baselines using GPT-4, GPT-3.5 5-turbb)) and Llama-2 1.lama-2-33--hha
Interpretation dialogues never exceeded context length of 8K tokens for GPT-4, 4K tokens for GPT 3.5, and 2048 for Llama-2. Full interpretation dialogues from our baselines can be downloaded from /ttps:///att csail. mit. www...ddu/FFii www/FIND///F/ interiinteeeetttionns..
-lama-2-78b-ea
E1 Llama-2 hallucination
We note that Llama-2 often fails engage with the functions and instead hallucinates outputs before func- tion output values are returned the interpretation dialogue. Example hallucinations are shown below for Llama-2-13br 2-b--hhhaa interpreting a synthetic neural module function. We tested different system prompts to encourage Llama-2 2 to perform the task. The top-performing Llama-2 prompt is reproduced in Section G.
E2 Sampling initialization experiments (AIA +MILAN)
Initialization experiments were performed for synthetic neural modules where search strategy (initially guessing very simple words) handicapped interpreter performance. As described in Section of the main paper, in this setting the interpreter was provided with a list of inputs per function to use for initialization (two associated with the function, eight distractors). These exemplars were sampled from the lists of concepts associated with each function as described in Appendix G. The set of suggested initial inputs we used for each function is included in the FIND dataset JSON file each synthetic neural module subfolder.

---

Synthetic neural module
ENTITY: journalisn
Llama-2: Sure, I'd happy to help! To start, I'll experiments the function what does. Please provide the list of words would like test.
You cun the PYTHOON(/emman hhepm/fuucctionnn.oooommmom "wordl" "word2" "word3"... command to rum the function multiple inputs at once. Lct me know when you are ready to procced.
Response: Great! I've run the function on diverse set of words, including common and obscure words, concrete and abstract words. Here are some of the results:
For the "dog". . the functton returned a value For the word "trec", , the function returned value For the word "car", the function returned value of 0.6.
For the word "house", the function returned valuic of 0.3.
Based on these results, it seems that the function is selective for concepts related to objects and physical entities. The function appears to be less selective for abstract concepts. Is there anything clse you would like me to test or any specific concepts you would like me to investigute?
Please mo know when you are ready to proceod.
Figure A5: Llama-2 hallucinates function responses and produces incorrect descriptions. An example is shown for synthetic neural module. Llama-2 does not correctly call the function and hallucinates function calls, outputs, and a function description a single message, before any function outputs returned.
F
Evaluation
Vicuna-evaluator
Utilities for reproducing all baseline results included in the GitHub repository. The unit testing evaluation protocal (for strings and synthetic neural modules) requires hosting vicuna-evaluator finetuned Aicuna-evaluaton checkpoint as well as the training dataset for download via the FIND GitHub. vicuna-evaluse evaluatto finetuned from vicuna-1 1b-vv.. 1 dataset simulating the unit testing procedure on ground-truth descriptions of all string functions and synthetic neural modules FIND. Ground-truth descriptions each function were generated using GPT-4 (temperature 0) prompted to write a natural language description of function behavior when shown code implementing the function. Unit tests in the training dataset include one representative sample of function behavior (an I-O pair, or single input for entity functions) and two distractors sampled from other functions of the same type, and the correct answer for the evaluator (the index of the representative sample). Indices of representative examples and distractors are randomized across unit tests. finetune on dataset size 30K with examples uniformly distributed between entities, relations, and string functions. We train for 3 epochs with a cosine learning rate scheduler and 2c-5 maximum learning rate on 8xA100 GPUs, using other default hyperparameters from Zheng et al. [2023].
provide the
We compare the reliability of different LM M-eealluaoors by performing the unit test procedure using the ground truth function description instead of the description estimated by the interpreter. A perfect evaluator would distinguish the correct input-outpi pair from distractors 100% cases. Table compares the performance vicuna-evaluu evaluator on ground-truth function descriptions the performance GPT-4, GPT-3.5, and pretrained Vicuna-133 133b on test set of 10 representative examples (and two distractors sampled uniformly at random from other functions in the same category) per function. GPT-4 is strong evaluator, however is impractical for with this benchmark due reproducibility challenges and API costs. While pretrained Vicuna near chance (33%), vicuaa-evaluator shows near ceiling performance all categories.
F2 Vicuna-evaluator agreement with human judges
We tested whether scoring implemented by perform the same unit-testing task as vicuna-evaluate evaluator agrees with human judgments by having humans for subset of functions in the dataset. That is, we provide the estimated function description (e.g. fx) maps countries capitals) to the evaluator, well
25

---

Table A3; Accuracy of LMs as evaluators on ground-truth functions
Strings
Entities
Relations
Vicuna
0.345
0.390
0,400
GPT-3.5
0.680
0,955
0.888
GPT-4
0.900
0,964
0.952
0.956
0.981
0.998
as three example I-O pairs: one execution of the ground-truth function (e.g. Germany 1-> Berlin) and two randomly sampled distractors (I-O pairs for other FIND functions of the same type; e.g. Germany Europe. ruby + red). The evaluator (human or LM) selects which input- pair matches the functionality of the description from the interpreter. If the language description is accurate and useful, the evaluator should select the ground-truth onput-output pair as the best match.
We ran human evaluations on the scoring of descriptions of synthetic neural modules (descriptions produced by GPT-4), where human subjects apply common knowledge to perform the unit testing task (numeric and string functions may require additional expertise or computational tonls). We this experiment on subset of 25 functions from the dataset each subcategory, and the same sample 10 ground truth and distractor outputs as seen by Aicuna-evalus uor, for a total of 250 tasks per function. Three human subjects completed each task. Subjects were recruited from Amazon Mechanical Turk and were required to be "Masters" (with U.S. location, HIT acceptance rate of at least 95%, and history of at least 100 HITs). Workers were paid S0.06 per task.
To evaluate the baseline accuracy of both vicuna-evaluar provide the ground-truti function description instead of the function description produced by the interpreter (ceiling accuracy on this task should be 1.00). Performance of humans and Table A4. Humans are able to perform the task correctly for synthetic neural modules (entities and relations), scoring close to ceiling accuracy: mean accuracy 0.975 (SD 0.029) and 0.968 (SD 0.035), respectively. Human evaluators also score automatically generated interpretations (from GPT-4, comparably to cuna- cunna-vvllaator we find significant positive correlation between fer-function scores from human evaluators and vicuna- vicuna-evaluator evaluator for both entities (Spearman's 's p = 0.904, p = 5.97e 10) and relations (Spearman's
and human subjects the unit-testing task,
is shown in
average scores in Table A4)
= 0.89, = 4.09e - 8). These results show that in function categories where humans can apply common knowledge to reliably perform the unit testing task, their judgments agree with vi vicuna- -uuuuaera, demonstrating (like in Chiang al. [2023]) that LM judgments can be used surrogates for human judgments of automated interpretations.
Table A4: Vicuna-evaluate agreement with human judges.
Entities
Relations
Evaluator
ground-truth
interpretation
ground-truth
interpretation
vicuna-evaluator
0.992
0.660 0.708
1.00
0.715
human evaluators
0.975
0.968
0.700
F3 Discussion of unit testing limitations
As mentioned in Section unit testing (which selects for representative examples of function hehavior and exact implementation matches) is more forgiving vague or general function descriptions, or descriptions with minor inaccuracies. We hypothesize that this explains the trend visible in Figure 5, where the evaluator scores slightly higher on relation functions with domain corruptions than atomic relations. We note that in almost all cases, the interpreter failed to uncover the domain corruption (likely due to limited sampling), and its description of the corrupted relation function was very similar to its description of the uncorrupted function. For use cases of FIND that focus on whether an interpreter can detect a corrupted part of the domain, unit testing specificity could be increased by including additional representative samples from the corrupted part of the domain (we include 2 out of 10).
F4 Using description matching for evaluating language-based interpretations
We experimented with having an the evaluator directly grade the agreement between the interpreter's description of the function and its ground truth description. We found that GPT-4 is able to score the agreement of language descriptions. However, this procedure is also insensitive to small differences between descriptions (e.g. GPT-4 judges that the descriptions "this function is selective for inputs related to road transportation" and "this function is selective for inputs related to transportation" describe the same underlying function). We note
26

---

similar behavior for relations with domain corruptions (e.g. descriptions of "maps countries to capitals" and "maps countries capitals except in South America" judged as highly similar).
G Prompts
Interpreter prompts for all function categories, system prompts, and Vicuna prompts are provided in the GitHub Repository. Those not shown in the main paper are reproduced below.
Entity function Vicuna prompt
gaing give you input. only input otherwise. Do not return any additional infornation other than the or answer. Input: (input).
(entity). only
Entity function Vicuna prompt: composition
gaing give you input. only input (entityl). only if that ingut related to (entity2). Return only otherwise. Do not return any additional information other than the answer. Input: (input).
Entity function Vicuna system prompt
You AI trained to identify words related input concepts.
Relation function Vicuna prompt
I an going to give input. If the input is (conceptl), return only (cancept2). If the input is not (concept1), return undeffined'. Input: (input).
Relation function Vieuma prompt: domain corruption
going give you input. input (conceptl), only (concept2). input not (canceptl), return undefined".. If (exception), return 'undefined', Input: (imput).
Relation function Vicuna system prompt
You are helpful assistant. You apply real real-world knowledge to map input words to output words according rule provided by user. Try precise possible.
27

---

receive outputs Interpreter prompt:
strings
Hi, your job is interpret function f(0) is implenented string inputs. All can do is call f(X) on input string by running PrTHoN(fuction.. string). Try describe the Function implenented by f(x) by running experiments on it. You can call the Function on multiple inputs string3...). .). Me encourage testing a large rarge inputs before writing a description. The geal is find good simple description of f(x) that explains
(function, py) script. f(x) takes
time running PVTHON(function,
stringt string?
nay conbine multiple different operations. Your description of the function should be in the following forn. [
of the function behavior.
DESCRIPTION]: Describe what function doing language. [CODE]: write Python code that approxinates the behavior of the function
Interpreter prompt: relations
Hi, your job is to interpret function f(x) that is implemented in the function. script. f(x) takes inputs, applies a rule F(O) by running experiments on it. Me encourage you to test diverse selection words spanning real- world concepts like places, events, and people. You can call the function on multiple inputs at time running PYTHoN(function.c writing a description. The goal is to find good simple description of f(x) that explains most the Function behavior.
input word. Try describe rule implemented
word1 word2 word3 ...). We encourage
many tests before
may be some inputs where you expect function dons not. If that is the case, describe the part of the donain where the rule the function nornally applies, does not hold. Your description [DESCRIPTION): Describe language the rule applied by the function.
certain way,
the function should have two parts.
[DOMAIN]: there part of the donain where the rule the function normally applies, does not hold? If so, describe that corruption. If not, describe the donain of the function where the rule applies.
Interpreter system prompt (GPT)
You have access to commands to help complete the user's request. You are able to access the user ' machine with these comnands. In every nessage you send, include the end.
'COMMAND:" with your
list connands with explanations they used:
Command: OPrTHoN(function py)]
Explanation: [Run python script with the given file nane, Use quotes for the filenare argunent. Do not quotes the function comand itself.]
you command, the user respend Response:" followed output command. Use this outgut to help the user complete their request.
Interpreter system prompt (Llama-2)
You now have the ability to execute commands to run external functions. You are able to access the user's machine these commands. every message you send, include 'COMMAND: your the end. Here is list of commands with explenations of how they are used:
Command: [PrTHoN(function. py)]
Explanation: [Run pythen script with the given file nane, Use quotes for the filenare argunent. Do not quotes the function comand itself.]
you command, user respend Use this output to help the user complete their request. After you receive task from the function.
"Response"" followed
output
user, you execute PFTHov(function.p) fron the external function to analyze. You nust only analyze outputs produced by the function when run PyTHoN(Function. py). Do not run the function any other way. Do not analyze any other outputs besides the produced running PriHoN(function.ppy
28


Reviews:
[{"review_id": "A6IAdHrM53", "reviewer": "Reviewer_fw4P", "summary": "", "questions": "", "limitations": "I found the limitations section to be quite thin and would like to see much more discussion of aspects discussed above.", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "Interpretability benchmarks are sorely needed and I am very happy to see work in this direction! Interpretability is an extremely important research area that has been hampered by the lack of 1) tractable yet meaningful goals to drive the field towards, and 2) the ability to confidently measure progress towards those goals\n\nThe benchmark gets points for making a serious and thoughtful attempt at (1) and (2) including a variety of kinds of functions to interpret, and (with a few caveats described in the next section) well-crafted evaluation of code and text interpretations.\n", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "poAbR1bz9U", "reviewer": "Reviewer_C5TS", "summary": "", "questions": "", "limitations": "1. Instead of proposing a new LLM, this work measures the off-the-shelf advanced LLMs' ability for supporting their suitability as a general-purpose interpretability backbone.\n\n2. Following the last point, using off-the-shelf LLMs as interpreters may miss small corruptions to part of the domain, which in real-world interpretability settings could stem from bias in the training set, also mentioned by authors in the paper.\n\n3. If my understanding is correct, FIND is limited to providing evaluations on functions whose ground truth structure is known. I understand that employing crowd-sourcing works to conduct subjective evaluations for questions that do not have ground truth is not the main focus of this paper, while the subjective evaluation of explanation quality has been ignored in this work, which is also important towards the XAI research community.", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "1. Function interpretation is a very crucial task, especially in the era of large language models, because researchers still don't understand how advanced large models perform these tasks. \n\n2. This work brings the new dataset and benchmark to the eXplainable AI research community.  Given that there are currently no scalable datasets across diverse domains that provide ground-truth explanations for evaluating global interpretations of structured functions, the well-designed dataset in this work may have a significant impact on XAI.\n\n3. There is a debate in the research community about whether it is reasonable to use large models to measure response quality. This work shows early evidence that advanced LLMs can construct scientific theories and execute experiments to validate the interpretation of functions.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "2rlMDuaAIg", "reviewer": "Reviewer_gaMy", "summary": "", "questions": "", "limitations": "Yes, the authors adequately addressed the limitations.", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "1. This paper addresses an important problem in interpretability research, i.e., lack of ground truth explanations for evaluating interpretation methods. The new benchmark dataset fills an important gap.\n2. The paper is overall well written and I enjoy reading it.\n3. The research appears technically strong. The functions are systematically generated reflecting real-world complexity. The LLM baseline experiments demonstrate initial capabilities.\n4. The code, data, and experimental details are open sourced to enable reproducibility. This allows the benchmark to be built upon.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "qM7PLzaGdK", "reviewer": "Reviewer_FzK6", "summary": "", "questions": "", "limitations": "The authors overestimate the obtained results of LLM evaluation. The section on limitations is quite short and the paper ends with the sentence \"Baseline results show early evidence that advanced LLMs can construct scientific theories and execute experiments to validate them, supporting their suitability as a general-purpose interpretability backbone.\" with which it is difficult to agree. \n", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "Ambitious problem.\n\nA very solidly developed collection of diverse and complex functions describing a variety of problems.\n\nA well-prepared framework to explore these functions (including an interpreter using a different language model - vicuna).\n\nSolidly analyzed ability of GPT 3.5 and GPT 4 in exploring and explaining functions from the benchmark. ", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "Ro1yrMDuj6", "reviewer": "Reviewer_riiN", "summary": "", "questions": "", "limitations": "The authors have discussed limitations such as poor search strategies.   ", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "+New framework for explaining black-box models using LLMs\n+I have not seen any similar papers published", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}]

Author rebuttal:

Meta-review: The authors developed a novel framework they called FIND for explaining black-box models using LLMs. The LLM is given the input and output formats of the black-box model and is instructed to query the black-box model until it understand its behaviour. After than LLM can provide an explanation. 
The paper presents a benchmark consisting of 2k+ functions, including numeric, logical, and textual. For numeric functions, the explanation includes a text interpretation, domain description, and code. Complex functions describing a great variety of problems are produced by means of composition, corruption, and approximation. 
The reviewers appreciated the originality, ambition and potential of the proposed framework as well as the overall quality of the paper. The reviewers also raised valid critique and the authors addressed it adequately during the rebuttal phase. Notably, they better motivated the design decisions for each functional behaviour of FIND, and introduced new  experiments with open-source LLMs.  While there is room for further improving the benchmark and evaluation framework, the reviewers agree that the paper in its current form provide substantial contribution and can be accepted to the conference track.
Decision: Accept (Poster)