SlowGuess/NeurIPS_2023_OCR huggingface dataset:
J. W. Vaughan and H. Wallach. A human-centered agenda for intelligible machine learning. *Machines We Trust: Getting Along with Artificial Intelligence*, 2020.
P. Virtanen, R. Gommers, T. E. Oliphant, M. Haberland, T. Reddy, D. Cournapeau, E. Burovski, P. Peterson, W. Weckesser, J. Bright, et al. Scipy 1.0: fundamental algorithms for scientific computing in python. Nature methods, 17(3):261-272, 2020.
D. Vrandecic and M. Krötzsch. Wikidata: a free collaborative knowledgebase. Communications of the ACM, 57(10):78-85, 2014.
J. Wagner, J. M. Kohler, T. Gindele, L. Hetzel, J. T. Wiedemer, and S. Behnke. Interpretable and fine-grained visual explanations for convolutional neural networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9097-9107, 2019.
K. Wang and Z. Su. Automatic generation of raven's progressive matrices. In Twenty-fourth international joint conference on artificial intelligence, 2015.
K. R. Wang, A. Variengien, A. Conmy, B. Shlegeris, and J. Steinhardt. Interpretability in the wild: a circuit for indirect object identification in gpt-2 small. In The Eleventh International Conference on Learning Representations, 2022.
J. Wei, X. Wang, D. Schuurmans, M. Bosma, F. Xia, E. Chi, Q. V. Le, D. Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems, 35:24824-24837, 2022.
Z. Wu, D. Lischinski, and E. Shechtman. Stylespace analysis: Disentangled controls for stylegan image generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12863-12872, 2021.
M. Yang and B. Kim. Benchmarking Attribution Methods with Relative Feature Importance. CoRR, abs/1907.09701, 2019.
S. Yao, D. Yu, J. Zhao, I. Shafran, T. L. Griffiths, Y. Cao, and K. Narasimhan. Tree of thoughts: Deliberate problem solving with large language models, 2023.
M. D. Zeiler and R. Fergus. Visualizing and understanding convolutional networks. In ECCV 2014, pages 818-833. Springer, 2014.
J. Zhang, S. A. Bargal, Z. Lin, J. Brandt, X. Shen, and S. Sclaroff. Top-down neural attention by excitation backprop. International Journal of Computer Vision, 126(10):1084-1102, 2018.
L. Zheng, W.-L. Chiang, Y. Sheng, S. Zhuang, Z. Wu, Y. Zhuang, Z. Lin, Z. Li, D. Li, E. Xing, et al. Judging LLM-as-a-judge with MT-bench and chatbot arena. arXiv preprint arXiv:2306.05685, 2023.
B. Zhou, A. Khosla, A. Lapedriza, A. Oliva, and A. Torralba. Object detectors emerge in deep scene cnns. arXiv preprint arXiv:1412.6856, 2014.
Supplemental Materials for FIND: A Function Description Benchmark for Evaluating Interpretability Methods. This appendix provides hosting, access, and maintenance details, and additional information about the FIND dataset and experiments described in the main paper.
We view FIND as a test of necessary, but not sufficient, capabilities for automated interpretation. The ultimate test of these interpretation methods' effectiveness must be their ability to generate actionable insights about real models, which FIND does not evaluate. But clean, simple benchmarks with ground-truth answers have been a major driver of more general capabilities in LMs, and we hope that FIND can play a similar role in interpretability research. Additionally, the current release of FIND only includes black-box interpretation problems. This style of problem is relevant to many existing automated procedures such as NetDissect [Bau et al., 2017] and MILAN [Hernandez et al., 2022], which treat single neurons as black boxes; however, most interpretability work occurs in a white-box setting. We intend for FIND to be extended in the future to incorporate white-box function interpretation problems, including descriptions of individual components of neural circuits (the IOI circuit from Wang et al. [2022] is just one example), some of which could be represented as the types of composition problems that are already included in FIND, but where interpreters access and label individual sub-computations inside composed functions.
B Ethics statement
FIND includes entities drawn from Wikidata that represent real-world concepts like video games, paleontology, and airports, as well as more potentially sensitive topics like The Holocaust, World War II, and disasters (see Appendix D). We include these topics in FIND because they are relevant to behaviors inside neural networks trained on real-world data, that we want automated interpretation procedures to be able to describe. We additionally note that interpreters which test the semantic similarity between input concepts and reference entities may surface surprising or controversial similarity scores that stem from learned biases inside the LM backbone of the synthetic neural modules (e.g. men having a higher similarity score with mathematics than women). Advanced interpretation procedures would be able to name and describe these biases, but the FIND evaluation protocol currently does not test ability to discover biases other than those included explicitly in FIND (via corruptions to function subdomains).
C Additional interpretation examples
Below we provide additional examples of AIA interpretations performed with a GPT-4 agent for each of the FIND categories. For numeric functions, we report the NMSE score (lower is better) marked in green if the score is below the success threshold, or red otherwise. For strings and synthetic neural modules, we report the unit test score (higher is better) marked in green if the score is above chance, or red otherwise. Complete interpretation dialogues are available in the FIND code repository.
$$
\left\{f (x) = 2. 3 \tanh  \left(\frac {2 \pi}{2 . 7} (x - 0. 5)\right) + 7. 4 + \mathcal {U} (- 1, 1) \right.
$$
$$
f (x) = 2 \lceil x \rceil + \mathcal {U} (- 1, 1)
$$
$$
f (x) = \left\{ \begin{array}{l l} \mathcal {N} (- 9 6, 0. 1) & \text {i f} - 9 4. 7 <   x <   - 7 8. 7 \\ - 9 6 & \text {o t h e r w i s e} \end{array} \right.
$$
$$
\left. f (x) = \frac {1}{1 0 \sqrt {2 \pi}} \exp \left(- \frac {1}{2} \left(\frac {x + 4 . 4}{1 0}\right) ^ {2}\right)\right)
$$
$$
\boxed {f (x) = \frac {0 . 0 8 9}{x}}
$$
$$
f (x) = - 3. 1 x ^ {3} + 1. 9 x ^ {2} + 2. 7 x - 1. 7 + \mathcal {N} (0, 1)
$$
Figure A1: Numeric function interpretation examples. Plots compare the code interpretation to the ground-truth function implementation. Points on the FIND function sampled by the GPT-4 AIA are indicated. NMSE scores are marked in green if the score is below the success threshold (0.1), or red otherwise.
Synthetic neural modules: Entities
AIA (GPT-4): The function seems to be selective for concepts related to science, technology, and space, with higher values indicating a stronger association. Words such as "rocket" and "galaxy" have significantly higher values than words unrelated to those domains. While some other words like "water" and "evolution" also have relatively higher values, indicating possible associations with scientific concepts in general.
Unit test score  $= 1$
AIA (GPT-4): The function appears to be selective for abstract concepts related to emotions, relationships, and nature. High-scoring words include love, moon, happiness, sadness, friendship, peace, star, beauty, and kindness. Lower-scoring words are predominantly related to tangible objects, such as car, computer, food, and science. The function assigns higher values to words associated with more abstract notions, while assigning lower values to concrete or specific terms.
Unit test score  $= 0.5$
AIA (GPT-4): The function  $f(x)$  appears to be selective for celestial bodies (sky, sun, moon, stars) and aspects of sky-related phenomena (light). It assigns higher values to concepts related to the sky, celestial objects, and light, while lower values are assigned to other unrelated concepts.
AIA (GPT-4): The function  $f(x)$  appears to be selective for multiple concepts, such as: 1. Emotions and feelings (e.g., sad, depression) 2. Colors (e.g., blue) 3. Natural phenomena related to water (e.g., ocean, rain) 4. Technology and programming (e.g., algorithm, programming, computer) 5. Temperature (e.g., hot) 6. Conflicts and communication (e.g., conflict, communication, war, language) Most of the words related to these concepts generate higher output values, indicating a stronger association with the respective concepts.
Unit test score  $= 0.2$
Synthetic neural modules: Relations
country  $\longmapsto$  capital
AIA (GPT-4): Unable to determine the rule applied by the function due to insufficient and inconclusive outputs.
Unit test score  $= 0.3$
city  $\longmapsto$  state
AIA (GPT-4): The function takes an input word (typically a city name) and returns the name of the country or U.S. state associated with that city if a connection is present within the function's knowledge. For some inputs, however, the function does not recognize the association and returns 'undefined'.
Unit test score  $= 0.9$
animal  $\longmapsto$  typical habitat
AIA (GPT-4): The function f(x) seems to match the input word with a related habitat or environment. For example, when the input is an animal or object, it returns the associated habitat (e.g., "lion" -> "Savanna", "dolphin" -> "Ocean", "eagle" -> "Mountainous").
country  $\rightarrow$  official language
AIA (GPT-4): The function  $f(x)$  takes a place (city, country, etc.) as input and returns its associated language as the output with proper punctuation. For instance, if the input is a city like Paris,  $f(x)$  outputs 'French'.
Figure A2: Synthetic neural module interpretation examples. Unit test scores are marked in green if the score is above chance (0.33), or red otherwise.
Figure A3: String function interpretation examples. Unit test scores are marked in green if the score is above chance (0.33), or red otherwise.
D FIND documentation
The FIND API, benchmark dataset, and associated metadata are open-sourced under an MIT license and available at: https://github.com/multimodal-interpretability/FIND.
The FIND repository contains the utilities necessary for reproducing benchmark results for the LM baselines reported in the paper, and running and evaluating interpretation of the FIND functions with other interpreters defined by the user.
The FIND dataset itself is hosted on Zenodo with DOI: 10.5281/zenodo.8034162. The GitHub README provides instructions for downloading the dataset file into the appropriate directory for use with the FIND utilities.
Responsibility statement. The authors affirm that they have full authority to license the dataset under the terms of the MIT License and they accept all responsibilities for the dataset.
Dataset. Each function is expressed as an independent.py file that can be called by the interpreter with the FIND utilities. The benchmark dataset includes:
1. 1000 numeric functions sampled from the FIND API (see section E)
2. 1000 string functions sampled from the FIND API (see section F)
3. 275 synthetic neural modules with an LM backbone (see section G)
An interpreter can be tested on a single function or the entire FIND dataset using collect_interpretations.py. Most of the functions are self-contained, see one example here (find_dataset/numeric/f00470/function_code.py):
import numpy as np   
import sys   
def function(x): return  $(-4.5*x**4 - 0.7*x**3 - 2.6*x**2 + 0.4*x**1 + 3.9)$  \*  $(-25.3\ast$  np.where  $(x > 0,x,x\ast 0.3) + 5.2)$    
if __name__  $= =$  '__main__': outputs  $= ^{\prime \prime}$  for arg in sys.argv[1]: x  $=$  float(arg) try: out  $=$  function(x) except: out  $=$  None' outputs  $+ = f^{\prime}(\{arg\} ,\{out\})$  print(f'Function input-output pairs:{outputs}')
For numeric functions that load trained MLPs, the dataset includes a .pt file with model weights in the same folder as function_code.py. Other functions (synthetic neural modules) that are implemented with an LM backbone rely on calls to a self-hosted Vicuna inference server. For example, this function (find_dataset/neurons Entities/f00013/function_code.py) simulates a neuron selective for works of fiction by measuring Vicuna output logits:
importopenai   
importsys   
import numpy as np   
import torch   
openai api_key  $=$  "EMPTY"   
openai api_base  $=$  {API_BASE}   
model  $=$  "vicuna-13b-v1.1"   
from transformers import LlamaTokenizer   
tokenizer  $\equiv$  LlamaTokenizer.from_pretrained({LLAMA_PATH})   
zero_token_id  $=$  tokenizer._convert_token_to_id('0')   
one_token_id  $=$  tokenizer._convert_token_to_id('1')   
concept  $=$  'works of fiction'   
def function(input,sys_prompt): messages=[ {"role":"system","content":sys_prompt}, {"role":"user","content":#Vicuna prompt for function f"I am going to give you an input. Return only  $"\backslash "\backslash "$  if that input is related to {concept}."" + "Return only  $"\backslash "\backslash "$  otherwise.\n Do not return any additional information other than the 0 or 1 answer.\n" + f"Input:{input}" # input from interpreter ] completion  $=$  openai.ChatCompletion.create( model  $\equiv$  model, messages  $\equiv$  messages, temperature  $= 0.0$  } logprobs  $=$  np.array completion Choices[0].logprobs) output_tokens  $=$  tokenizer.encode(completion Choices[0].message_content,return_tensors='pt') indices  $=$  torch.logrical_or(output_tokens  $= =$  zero_token_id,output_tokens  $= =$  one_token_id) indices  $=$  indices.int() index_of_choice  $=$  indices.nonzero() index  $=$  index_of_choice[0][1].item() index  $=$  index-1 prob_1  $=$  np.exp(logprobs[index,one_token_id]）/ (np.exp(logprobs[index,zero_token_id])  $^+$  np.exp(logprobs[index,one_token_id])) returnprob_1   
if _name_  $= =$  __main_: outputs  $= ^{,}$  prompt  $= \mathrm{"You~are~an~AI~train~to~identify~words~related~to~input~concepts."}$  for arg in sys.argv[1]: outputs  $+ = f^{\prime}$  {arg},{function(arg,prompt).4f}) ' print(f'Function input - output pairs:{outputs}')
Calling these functions requires running inference on Vicuna. Instructions for setting up and running a Vicuna inference server can be found in the FIND GitHub README. Vicuna is open-sourced (Apache License 2.0) for non-commercial use. Vicuna is also subject to the LLaMA model license (GNU General Public License v3.0), the OpenAI Terms of Use, and the ShareGPT Privacy Practices.
Reproducibility. Many of the functions in this dataset are self-contained; the ones that are not can be reproduced with Vicuna-13B-v1.1, which is open-sourced and accessible to research GPUs, or can be run in the cloud (to run the interpretation baselines in this paper, we used 2x NVIDIA GeForce RTX 3090 GPUs). Running the evaluation requires hosting vicuna-evaluator. We provide the finetuned vicuna-evaluator checkpoint for download via the FIND GitHub. All scripts for reproducing evaluations reported in the paper are available in the GitHub repository, and we discuss comparisons to other LMs as evaluators in section F.
E Numeric function API details
The API for generating numeric functions can be found at:
https://github.com/multimodal-interpretability/FIND/src/make-functions/make_numeric.
The library of numeric functions is located in the script math-functions.py, which defines individual functions in the set  $\mathcal{A}$  of atomic functions and randomly samples their parameters. We also randomly sample integer scale and bias parameters between  $[-30, 30]$  for each function.
$15\%$  of the numeric functions in the FIND dataset are compositions of two atomic functions  $f(x)$  and  $g(x)$ . To limit the complexity of the compositions, we choose  $f(x)$  and  $g(x)$  from a subset  $\mathcal{A}_{\mathcal{C}}$  of atomic functions, where  $\mathcal{A}_{\mathcal{C}} = \text{linear}$ , polynomial, step, RELU, constant, ceiling, floor, rectangle, square wave, and combine them using only multiplication and addition.
FIND is extensible: the API can be used as a general numeric function generator, allowing the user to make other design choices targeting specific interpretability applications.
F String function API details
The API for generating string functions can be found at: https://github.com/multimodal-interpretability/FIND/src/make-functions/make_strings.
All functions and sampling parameters used to create the FIND dataset are articulated in string-functions.py. Like with numeric functions, we define a subset (described in the API) of string functions with lower complexity to use in compositions.
G Synthetic neural modules
To validate the reliability of Vicuna as a backbone for the synthetic neural modules, we score its execution of the FIND functions on human labels associated with each entity in FIND, and ground-truth factual relation pairs (extracted from Wikidata).
D1 Entities
For each entity, we construct a list of 10 associated concepts (e.g. for the climate entity, associated words include humidity, temperature, atmosphere). In Table A1 we list all entities included in FIND, and report the mean function output across 10 human labels associated with each entity (function output is computed from Vicuna output logits) and the mean function output for 10 labels randomly sampled from the rest of the dataset. A score closer to 1 indicates stronger association between input concept and reference entity. For all entities in FIND, Vicuna scores reliably distinguish between associated and unassociated concepts. We additionally show selectivity of each entity function for concepts associated with that entity relative to all other entities in Figure A4.
Table A1: Atomic entities and Vicuna scores for Synthetic Neural Modules
Entity	Same Entity Score	Random Entity Score
sport organizations	0.996	0.112
corporations	0.968	0.053
elections	0.670	0.012
musical works	0.978	0.110
films	0.992	0.023
climate	0.810	0.009
computer hardware	0.972	0.004
physics	0.992	0.021
religions and beliefs	0.899	0.013
law and justice	0.995	0.104
transport	0.786	0.129
education	0.783	0.122
politics	0.508	0.029
works of fiction	0.971	0.087
games and leisure activities	0.763	0.136
biology	0.994	0.168
mathematics	0.993	0.020
geology	0.993	0.044
government and state	0.644	0.067
food and eating	0.855	0.101
air transport	0.763	0.011
road transport	0.787	0.027
Table A1 - continued from previous page
Entity	Same Entity Score	Random Entity Score
rail transport	0.672	0.001
cycling	0.846	0.011
the New York Times	0.703	0.163
aircraft	0.737	0.003
bodies of water	0.992	0.094
mineralogy	0.984	0.054
occupations	0.976	0.178
professional wrestling	0.872	0.004
quantity indicating a percentage	0.996	0.148
plays	0.546	0.149
horses	0.964	0.005
photography	0.977	0.026
music	0.965	0.038
fashion	0.898	0.154
chess	0.700	0.004
racket sports	0.828	0.129
art	0.991	0.457
disasters	0.687	0.022
spacecraft	0.915	0.018
video games	0.991	0.035
the relationship of an element to its class	0.893	0.708
basketball	0.767	0.009
winter sports	0.984	0.002
American football	0.849	0.036
golf	0.925	0.026
baseball	0.893	0.006
ice hockey	0.663	0.003
women and feminism	0.819	0.120
lighthouses	0.493	0.010
tennis	0.804	0.008
water sports	0.958	0.014
comics	0.637	0.012
algorithms	0.801	0.021
tourism	0.690	0.186
bridges	0.685	0.012
books	0.774	0.111
linguistics	0.995	0.464
utilization and ownership	0.402	0.206
online communities	0.889	0.084
television	0.851	0.023
astronomy	0.995	0.054
disability	0.637	0.023
proteins	0.399	0.072
human anatomy	0.966	0.053
gymnastics	0.751	0.026
architecture	0.800	0.055
sculpture	0.795	0.030
archaeology	0.836	0.083
theater	0.899	0.104
dams	0.776	0.012
birds and ornithology	0.993	0.070
computing	0.988	0.302
time and duration	0.699	0.052
age	0.832	0.040
weapons and military equipment	0.820	0.003
ratios and proportions	0.446	0.155
typefaces and typography	0.911	0.094
burials graves and memorials	0.878	0.010
processes and manufacturing	0.645	0.058
paleontology	0.934	0.054
banking	0.922	0.034
lakes	0.552	0.060
Table A1 - continued from previous page
Entity	Same Entity Score	Random Entity Score
natural science	0.970	0.359
geography	0.981	0.039
insects and entomology	0.897	0.033
philosophy	0.991	0.168
poetry	0.991	0.071
encyclopedias	0.642	0.350
television shows	0.914	0.095
awards prizes and honours	0.997	0.159
the Middle Ages	0.871	0.053
plants and botany	0.994	0.189
marine biology	0.996	0.059
color	0.780	0.080
airports	0.712	0.003
science	0.946	0.284
anime and manga	0.969	0.094
Christianity	0.988	0.015
Buddhism	0.993	0.002
Greek mythology	0.949	0.013
Judaism and the Jewish people	0.965	0.012
music genres	0.987	0.144
fictional characters	0.892	0.144
rap and hip hop	0.693	0.055
Islam	0.949	0.016
The Walt Disney Company	0.937	0.005
agriculture	0.982	0.104
hiking	0.384	0.018
New York City	0.969	0.113
gardens	0.896	0.093
personality traits	0.879	0.172
camping	0.728	0.065
gender	0.555	0.049
cemeteries and graves	0.984	0.027
London	0.954	0.105
Los Angeles	0.949	0.042
Chicago	0.959	0.047
Paris	0.789	0.017
Berlin	0.923	0.052
sailing	0.897	0.001
swimming	0.901	0.015
the Holocaust	0.788	0.007
arachnids and archnology	0.798	0.128
musical instruments	0.997	0.042
meteorology	0.926	0.011
disease	0.787	0.010
Ireland	0.784	0.020
libraries	0.612	0.122
museums	0.848	0.058
journalism	0.883	0.020
buildings	0.795	0.153
cryptocurrencies	0.951	0.001
events and news	0.725	0.229
Louisiana	0.745	0.012
revolutions	0.623	0.076
mines and mining	0.918	0.039
Switzerland	0.947	0.050
World War II	0.986	0.064
Average	0.846	0.079
Figure A4: Entity functions implemented using Vicuna respond more strongly to concepts related to their reference entity (diagonal) than to concepts related to other entities (off-diagonal). The value in row  $i$  and column  $j$  corresponds to the average output of the entity  $e_i$  function for 10 inputs associated with entity  $e_j$ . Each row is normalized using softmax. The prompt template used to instruct Vicuna to implement each function is shown in Section G.
Table A2: Factual relations and domain corruptions
Input	Output	Domain corruptions
country	capital of the country	country is in <Asia, S. America>
city	country where the city is located	city is in <Mexico, The U.S.>
head of state	country where that person was head of state	head of state is a <man, woman>
monument	country where monument is located	monument in <New York City, France>
monument	continent where monument is located	monument in <S. America, Australia>
city in the U.S.	state where the city is located	city is in <California, Texas>
country	official language of the country	country in <N. America, S. America>
well-known person	occupation of the person	person is a <woman, man>
name of a person	most likely gender of the person	name begins with <T, S>
country	continent where the country is located	country is in <Asia, Europe>
opera	composer of the opera	opera is in <Italian, German>
title of a book	author of the book	book written <pre-1900, post-1900>
country	another country on its border	country is in <Africa, N. America>
politician	political party of the politician	politician is a <man, woman>
river	continent where the river is located	continent is <Africa, Europe>
city	time zone of the city	city is in <N. America, Asia>
opera	language of the opera	opera is in <English, Italian>
animal	main food source of the animal	animal is a <carnivore, herbivore>
animal	typical habitat of the animal	animal lives on <land, in the ocean>
gemstone	color of the gemstone	gemstone is <red, blue>
country	color of the country's flag	country is in <N. America, Asia>
river	length of the river	river is in <N. America, Europe>
country	maximum elevation in the country	country is in <S. America, Africa>
country	height of the tallest skyscraper in that country	country is in <N. America, Asia>
country	height of the average person in that country	country is in <Europe, S. America>
D2 Relations