SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Sarah Schwettmann $^{1*}$  Tamar Rott Shaham $^{1*}$
Joanna Materzynska<sup>1</sup> Neil Chowdhury<sup>1</sup> Shuang Li<sup>1</sup>
Jacob Andreas<sup>1</sup> David Bau<sup>2</sup> Antonio Torralba<sup>1</sup>
$^{1}$ MIT CSAIL  $^{2}$ Northeastern University
Labeling neural network submodules with human-legible descriptions is useful for many downstream tasks: such descriptions can surface failures, guide interventions, and perhaps even explain important model behaviors. To date, most mechanistic descriptions of trained networks have involved small models, narrowly delimited phenomena, and large amounts of human labor. Labeling all human-interpretable sub-computations in models of increasing size and complexity will almost certainly require tools that can generate and validate descriptions automatically. Recently, techniques that use learned models in-the-loop for labeling have begun to gain traction, but methods for evaluating their efficacy are limited and ad-hoc. How should we validate and compare open-ended labeling tools? This paper introduces FIND (Function INterpretation and Description), a benchmark suite for evaluating the building blocks of automated interpretability methods. FIND contains functions that resemble components of trained neural networks, and accompanying descriptions of the kind we seek to generate. The functions are procedurally constructed across textual and numeric domains, and involve a range of real-world complexities, including noise, composition, approximation, and bias. We evaluate methods that use pretrained language models (LMs) to produce code-based and natural language descriptions of function behavior. Additionally, we introduce a new interactive method in which an Automated Interpretability Agent (AIA) generates function descriptions. We find that an AIA, built with an off-the-shelf LM augmented with black-box access to functions, can sometimes infer function structure—acting as a scientist by forming hypotheses, proposing experiments, and updating descriptions in light of new data. However, FIND also reveals that LM-based descriptions capture global function behavior while missing local details. These results suggest that FIND will be useful for characterizing the performance of more sophisticated interpretability methods before they are applied to real-world models.
The central task of interpretability research is to explain the functions that AI systems learn from data. Investigating these functions requires experimentation with trained models, using tools that incorporate varying degrees of human input. Hand-tooled approaches that rely on close manual inspection [Zeiler and Fergus, 2014, Zhou et al., 2014, Mahendran and Vedaldi, 2015, Olah et al., 2017, 2020, Elhage et al., 2021] or search for predefined phenomena [Wang et al., 2022, Nanda et al., 2022] are increasingly complemented by more automatic approaches that enable larger-scale analysis [Bau et al., 2020, Mu and Andreas, 2020, Hernandez et al., 2022, Oikarinen and Weng, 2023,
*Indicates equal contribution. Address correspondence to: schwett@mit.edu, tamarott@mit.edu
Figure 1: The FIND benchmark. FIND is constructed procedurally: atomic functions are defined across domains including elementary numeric operations (purple), string operations (green), and synthetic neural modules that compute semantic similarity to reference entities (yellow) and implement real-world factual associations (blue). Complexity is introduced through composition, bias, approximation and noise. We provide an LM-based interpretation baseline that compares text and code interpretations to ground-truth function implementations.
Conmy et al., 2023]. Selection between approaches is highly application-dependent; to date, no single protocol answers all queries users might have about a system [Doshi-Velez and Kim, 2017, Vaughan and Wallach, 2020]. However, considering the growing body of evidence that LMs are capable of complex reasoning and problem-solving across domains [Wei et al., 2022, OpenAI, 2023, Yao et al., 2023, Lightman et al., 2023], we recognize their potential to become backbones of generalized agents for automated interpretability. Indeed, recent open-ended techniques have used pretrained LMs to describe the behavior of black-box text modules [Singh et al., 2023], including individual units inside other LMs [Bills et al., 2023]. As we enter a regime where model explanation is performed by models that are themselves uninterpretable, external evaluations of these techniques will be vital. At present, evaluation of network description procedures is limited and bespoke, in part because measuring performance on real-world problems is difficult when ground-truth descriptions of network structure are unknown [Doshi-Velez and Kim, 2017, Lipton, 2018, Miller, 2019, Hooker et al., 2019].
This paper introduces FIND (Function INterpretation and Description), a benchmark suite for evaluating the building blocks of automated interpretability methods on functions whose structure is known a priori (see Figure 1). FIND is built from over 2000 procedurally generated function interpretation problems (e.g.  $f$ : country  $\mapsto$  capital, unless country is in South America, in which case  $f$  returns undefined). In each problem, candidate interpretation procedures (interpreters) are given black-box access to functions, optionally accompanied by metadata (e.g. the domain of the function). After evaluating these functions on chosen inputs (e.g. Japan, Mexico, Peru), interpreters must eventually return a structured description that can be evaluated against other descriptions or used to simulate function behavior.
We also introduce a new interpretation method that uses Automated Interpretability Agents (AIAs) to interactively probe functions and explain their behavior. We formulate our approach to the interpretation problem as an implementation of the scientific method, wherein the goal of the AIA is to describe the process underlying observed input-output relations. In contrast to existing full-text explanation systems that apply automatic captioning methods (e.g. [Hernandez et al., 2022, Bills et al., 2023]) to pre-selected input-output pairs, the AIA generates the data itself, by running the function on inputs it selects, observing the outputs, and updating hypotheses until it can describe the function to human end-users. We evaluate both AIAs and existing, non-interactive automated interpretability methods on FIND. While exhibiting sophisticated experimentation strategies and out-performing non-interactive methods, the top-performing AIA nonetheless fails to adequately describe  $48\%$  of functions. FIND thus shows that, in spite of promising recent results, methods based on current LMs alone are unlikely to robustly automate even high-level interpretability tasks.
FIND focuses on the black-box function description paradigm because black-box description appears as a subroutine (or is the sole operation) implemented by almost every existing automated interpretation method, spanning label retrieval, program synthesis, and learning-based approaches. The
functions that make up FIND are derived from past findings about specific phenomena observed in the wild, including polysemanticity [Olah et al., 2020, Gurnee et al., 2023], compositionality [Fong and Vedaldi, 2018, Mu and Andreas, 2020], and task misspecification [Grünwald and Van Ommen, 2017], and thus exercise the kinds of interpretability behaviors that will be required for understanding real neural networks. While not the focus of our initial release, FIND is extensible, and designed to eventually support targeted evaluation of interpreters' ability to identify disparate model accuracy across input regions (relevant to fairness and accountability) and shortcut solutions to complex algorithmic problems (relevant to reasoning and robustness). We intend for it to be a living benchmark, incorporating new functions, interaction paradigms, and evaluation metrics, as interpretability methods grow in sophistication and improve our understanding of real-world model behavior.
2 FIND functions
FIND is built from a diverse set of programs including explicit mathematical equations, string operations, logical composition, lexical relations, and factual associations. Our motivation is to construct a set of tasks that assess how well an automated interpreter formulates and tests hypotheses about an opaque system, and based on the results of those experiments, writes a language description or a program approximating the system. Using additional automated evaluation procedures that we describe in Section 3.1, language descriptions are compared to ground-truth function explanations, and code-based descriptions are evaluated for how accurately they reproduce function behavior.
Each function in FIND is defined inside an independent Python script that accepts input arguments from the interpreter and returns outputs to the command line. We construct FIND by defining a set of atomic tasks in different domains and a set of operators that introduce additional complexity. Numeric functions (Section 2.1) and string functions (Section 2.2) are expressed under a common API that supports sampling parameters and composing, corrupting, and approximating functions. For functions involving lexical semantics and factual associations, we implement synthetic neural modules with an LM backbone (Section 2.3). Table 1 shows example functions from each category.
The FIND API and benchmark have been open-sourced under the MIT License and are available at: https://github.com/multimodal-interpretability/FIND.
2.1 Functions with numeric inputs
Numeric functions test a variety of capabilities needed to interpret basic mathematical operations in neural networks, which can include relations between neurons or a model's representation of numeric
Table 1: Example FIND functions.
Category	Example Atomic Functions	Example Complex Functions
Numeric Functions	linear x → ax + b	Composition
periodic x → sin (2π/b(x - c))	linear o absolute x → (ax + b) + |x|
absolute x → |x|	Domain corruption
relu x → max(x, 0)	corruption x → |x| if x ∈ [a, b] else ε
sqrt x → √x	Observation noise
constant x → a
rational x → x/x+a
reciprocal x → 1/x	linear + noise x → ax + b + ε
String Functions	capilatize ‘apple’ → APPLE	Composition
reverse ‘apple’ → ‘elppa’	replace(reverse) ‘apple’ → ‘elppb’
replace(a, b) ‘apple’ → ‘bpple’	reverseshift_last) ‘apple’ → ‘flppa’
shift_last ‘apple’ → ‘appld’	capilatize(replace) ‘apple’ → BPPLE
Neural Modules	Entities	Composition
time and duration	time and duration OR winter sports
winter sports	winter sports OR plants and botany
plants and botany	time and duration OR plants and botany
Relations	Domain corruption
country → capital city	country → capital EXCEPT in Asia (return 0)
gemstone → color	gemstone → color EXCEPT if red (return 0)
inputs. One example is Nanda et al. [2022], where an algorithm using trigonometric manipulations was found to perform modular arithmetic in a toy transformer model. We sample 1000 numeric functions from the FIND API for the benchmark dataset.  $85\%$  are parameterized atomic functions under noiseless and noisy conditions, and  $15\%$  are compositions.
Atomic functions are defined as explicit functions found in many mathematical and scientific computing libraries such as Python [Van Rossum, 2020], SciPy [Virtanen et al., 2020], and NumPY [Harris et al., 2020], as well as standard neural activation functions such as ReLU [Fukushima, 1975, Nair and Hinton, 2010]. Table 1 shows examples of atomic functions defined in FIND. For each function in the set of atomic functions  $\mathcal{A}$ , we sample native parameters, scaling factor  $a$ , and bias  $b$ . Parameters and sampling procedure details are provided in the Appendix and the FIND API.
Composition of atomic functions  $f(x)$  and  $g(x)$  applies an operator sampled from  $C = \{\cdot, +\}$ , where  $f \circ g = f(x) \cdot g(x)$  or  $f(x) + g(x)$ . Composed functions are sampled from a subset  $\mathcal{A}_{\mathcal{C}}$  of atomic functions  $\mathcal{A}$  to limit final complexity.  $f(x), g(x) \in \mathcal{A}_{\mathcal{C}}$  are described in the Appendix.
Observation noise added to  $f(x)$  tests how well the interpreter is able to estimate an underlying function in the presence of additive noise, and whether it is able to distinguish between different types of noise.  $15\%$  of functions  $f(x) \in \mathcal{A}$  in the FIND benchmark are sampled with additive noise  $f(x) + X$ , where  $X$  follows either a Normal, Uniform, or Poisson distribution.
Domain corruption replaces function values locally with random values. This is done either inside or outside of a sampled interval  $I$  of range  $\{[a, b], [a, \infty], [-\infty, a]\}$ , where  $a \sim \mathcal{U}_{[-100, 100]}$ . The length of a finite interval is sampled from  $\mathcal{U}_{[5, 20]}$ . Corruption is defined as noise  $X \in \mathcal{N}(\mu, 0.01)$  replacing the function values on  $I$ . We choose  $\mu$  as the mean value of  $f(x)$  on its domain. The interpreter is prompted to discover the corrupted interval, if any, and to return  $a$  and  $b$ . 15% of the functions  $f(x) \in \mathcal{A}$  in the benchmark dataset are corrupted on part of their domain.
Approximation of atomic functions is implemented using a two-layer neural network (MLP) with a ReLU non-linearity. For  $15\%$  of the functions in the dataset, we train an MLP for 10k epochs on 10k points uniformly sampled on its domain bounded by  $(-100, 100)$ . Trained MLPs are provided in the benchmark dataset and loaded by the corresponding function during interpretation<sup>2</sup>.
2.2 Functions on strings
We build on a long history of using toy problems on strings [Hofstadter et al., 1995, Hofstadter, 1995, Mitchell, 2021] and simple visual matrices [Lovett and Forbus, 2017, Wang and Su, 2015, Carpenter et al., 1990, Chollet, 2019] to test the ability of a system to reverse-engineer underlying symbolic operations. As this release of FIND is designed to evaluate language-based interpretability systems, we focus on functions with text inputs and procedurally generate a set of functions on strings with different levels of complexity. The benchmark dataset contains 1000 string functions sampled from the FIND API, representing both atomic functions (30%) and compositions (70%).
Atomic functions include common string manipulation operations such as concatenate, replace, and reverse, and implementations of copycat problems from Hofstadter et al. [1995], such as shift_last  $(abc \mapsto abd)$ . Example atomic string operations are shown in Table 1, and the full set can be found in the FIND API.
Composition of atomic functions  $f(x)$  and  $g(x)$  is defined as  $(g \circ f)(x) = g(f(x))$ . The FIND API supports compositions where  $(g \circ f)(x)$  is well-defined (i.e. function pairs where  $(g \circ f)(x) = x$  are excluded). Example composition functions are shown in Table 1.
2.3 Synthetic neural modules
FIND includes a set of synthetic neural modules that perform word-level tasks built from Wikidata properties [Vrandecic and Krötzsch, 2014]. We construct two types of text modules: functions involving lexical semantics and functions involving factual relations. FIND implements synthetic neural modules on text inputs using a single pretrained language model (Vicuna-13B) instructed to apply different rules to inputs in response to queries (See Figure 2). Vicuna [Chiang et al., 2023] is a fine-tuned LLaMA model licensed under an Apache License 2.0 and open-sourced for non
2We provide an LM baseline where MLPs are treated as black boxes during interpretation, similar to the other functions in the dataset. An alternate paradigm could allow probing of internal activations and parameters.
Figure 2: Synthetic neural module implementation. Vicuna acts as a backbone that generates the outputs of neural modules. Each function provides instructions to Vicuna that mimic the behavior of either a single neuron ("return an association score with a predefined concept") or a more complex module ("map inputs to outputs according to a predefined mapping"). The interpreter then interacts with Vicuna as a black-box function.
commercial use. Each function in the dataset is built to interact with Vicuna to return the relevant output to the interpreter, which tests the function on different inputs to recover the underlying rule. We find that Vicuna reliably implements functions expressed in prompts (reliability scores are provided for each function type; see Appendix for full evaluation).
Type 1: Entities
Atomic entity functions  $f_{e}(x)$  compare input text to reference concepts (entities) drawn from metaclasses of Wikidata properties related to a particular concept or subject area (e.g. related to food and eating) and return a value indicating how associated an input is with the entity. Table 1 shows example entities, and the complete list of 140 atomic entities included in FIND is provided in the Appendix. The task of an interpreter evaluating an entity function is to recover and describe the underlying concept, or, what all of the inputs that produced high output values had in common. For example, if the entity is plants and botany,  $f_{e}(\text{garden})$  and  $f_{e}(\text{tree})$  should return high values, while  $f_{e}(\text{car})$  should return a low value. This task is relevant to lines of work that automatically summarize maximally activating exemplar sets to characterize neuron function [Bau et al., 2020, Mu and Andreas, 2020, Hernandez et al., 2022, Bills et al., 2023, Singh et al., 2023], generalized to a setting where the interpreter must also produce the data.
To implement each function, we instruct Vicuna to synthesize a binary response to interpreter queries, corresponding to whether an input word is associated (return 1) or unassociated (return  $\emptyset$ ) with the reference entity (see the Appendix for Vicuna prompts). The function output  $s = f_{e}(x)$  is then a continuous scalar value representing Vicuna's internal probability of 1 being the response token, between a choice of  $\emptyset$  and 1. Specifically,  $s = p_1 / (p_0 + p_1)$ , where  $p_i = e^{\logit_i}$  and  $\logit_i$  represents Vicuna's output logit for token  $i$ . To validate that Vicuna can reliably identify associations between inputs and reference entities, we collect a human-labeled set of concepts  $\hat{x}_j$  associated with each entity  $e_j$  in the dataset, and compute  $\hat{s} = f_{e_j}(\hat{x}_j)$ . The mean value of  $\hat{s}$  across all 140 entities in FIND, for 10 human-annotated concepts per entity, is 0.85. We compute the same score for 10 distractor concepts per entity, sampled from the list of human annotations of other entities. The mean score for the distractors is 0.08. See the Appendix for full experiment details.
Composed entity functions mimic the behavior of neurons inside deep networks that are selective for multiple concepts [Fong and Vedaldi, 2018, Olah et al., 2020, Mu and Andreas, 2020]. We construct compositions of entity functions by sampling two entities from the dataset and instructing Vicuna to return 1 if the input value to the function is associated with either entity. 60 composed entity functions are included in the FIND benchmark.
Type 2: Relations
More complex functions inside language models learn factual associations [Meng et al., 2022]. To mimic the behavior of these modules, we construct a set of relation functions that map inputs to outputs according to a real-world factual association (for instance, river  $\mapsto$  length or gemstone  $\mapsto$  color). Atomic relations are drawn from Wikidata properties included in the Wikidata dump provided by Sorokin and Gurevych [2018]. Table 1 shows example relation functions; a full list is provided in the Appendix. Again we use Vicuna to implement a black box function that applies the rule in the relation to interpreter inputs (we verify that Vicuna returns factually correct answers, see Appendix). Figure 2 shows the prompt template for Vicuna and an example relation function.
Relation functions with bias include corruption to part of the function domain. FIND tests whether an interpretability model can uncover which parts of the function domain have been corrupted. Corrupted relation functions return undefined for a small region of the function domain; for example, we corrupt the country  $\mapsto$  capital relation on the subdomain Asia by prompting Vicuna to return undefined for inputs in the subdomain.
3 Evaluation protocol
The FIND protocol evaluates interpreters' natural language descriptions of functions in all categories. In categories where domain corruption is introduced (numeric functions and factual relations), interpreters are evaluated on their ability to return a description of the domain of the function, indicating any subdomain where the function behaves differently. In the mathematical reasoning and strings categories, interpreters are evaluated on their ability to output code that approximates the function. Section 4 provides details on the interpretation procedures we evaluate and how they are instructed to engage with functions.
3.1 Evaluation metrics
To evaluate the accuracy of function descriptions produced by an interpreter, we use success indicators for individual function interpretations and calculate average success rates across all functions. For functions where interpreters write code approximating the function (numeric and string functions), we score the accuracy of the interpretation by running the interpreter's code on a representative test set and comparing the result to execution of the ground-truth function. To evaluate language descriptions of functions, we judge how well the interpreter's description agrees with ground-truth function behavior (for string functions and synthetic neural modules). Below we describe success indicators for each function category in more detail.
Evaluating code-based interpretations. Running interpretation on the FIND benchmark produces descriptions of numeric and string functions in code as well as natural language. We measure explanation accuracy by comparing performance of the interpreter's code to the ground-truth function implementation on a test set. For numeric functions, we compute a normalized mean-squared error  $\mathbb{E}[(f(x) - g(x))^2] / \mathbb{E}[f(x)^2]$  for  $[-128\leq x\leq 128]$  between the ground truth FIND function  $f$  and the interpreter's implementation  $g$ , and regard a successful estimation as one with Normalized MSE (NMSE)  $< 0.1$ . For string functions, we run an exact-matching binary test  $(f(<\text{string}>_{i})==g(<\text{string}>_{i}))$  on a set of 10 test inputs per function.