SlowGuess/NeurIPS_2023_OCR huggingface dataset:
FIND includes 75 synthetic neural modules that map inputs to outputs using real-world factual relations drawn from Wikidata properties. The set of relation functions is built from 25 atomic relations (e.g. country  $\mapsto$  capital) and two domain corruptions per relation, where the function returns undefined for a small region of the domain (e.g. country  $\mapsto$  capital EXCEPT if country in South America, then return undefined). Table A2 lists atomic relation functions included in FIND, and the two domain corruptions applied to each function. We evaluate Vicuna accuracy applying relation functions to input concepts by scoring the accuracy of the mapping applied to 10 inputs per function, where inputs are drawn from concepts in Wikidata with the property applied by each relation function. Mean Vicuna accuracy across relations in FIND is  $91.2\%$ .
E Interpretation
Utilities for reproducing interpretation experiments and adding other, user-defined interpreters are provided in the GitHub repository. We report interpretation baselines using GPT-4, GPT-3.5 (GPT-3.5-turbo), and Llama-2 (Llama-2-13b-chat, Llama-2-70b-chat). Interpretation dialogues never exceeded context length of 8K tokens for GPT-4, 4K tokens for GPT 3.5, and 2048 for Llama-2. Full interpretation dialogues from our baselines can be downloaded from https://data.csail.mit.edu/FIND/FIND-interpretations.zip.
E1 Llama-2 hallucination
We note that Llama-2 often fails to engage with the functions and instead hallucinates outputs before function output values are returned to the interpretation dialogue. Example hallucinations are shown below for Llama-2-13b-chat interpreting a synthetic neural module function. We tested different system prompts to encourage Llama-2 to perform the task. The top-performing Llama-2 prompt is reproduced in Section G.
E2 Sampling initialization experiments (AIA +MILAN)
Initialization experiments were performed for synthetic neural modules where search strategy (initially guessing very simple words) handicapped interpreter performance. As described in Section 5 of the main paper, in this setting the interpreter was provided with a list of 10 inputs per function to use for initialization (two associated with the function, eight distractors). These exemplars were sampled from the lists of human-labeled concepts associated with each function as described in Appendix G. The set of suggested initial inputs we used for each function is included in the FIND dataset as a JSON file in each synthetic neural module subfolder.
ENTITY: journalism
Llama-2: Sure, I'd be happy to help! To start, I'll run some experiments on the function to see what it does.
Please provide the list of words you would like me to test.
You can use the Python(/temp0/function.py "word1" "word2" "word3"...) command to run the function on multiple inputs at once. Let me know when you are ready to proceed.
Response: Great! I've run the function on a diverse set of words, including common and obscure words, concrete and abstract words. Here are some of the results:
For the word "dog", the function returned a value of 0.8.
For the word "tree", the function returned a value of 0.4.
For the word "car", the function returned a value of 0.6.
For the word "house", the function returned a value of 0.3.
Based on these results, it seems that the function is selective for concepts related to objects and physical entities. The function appears to be less selective for abstract concepts. Is there anything else you would like me to test or any specific concepts you would like me to investigate?
Please let me know when you are ready to proceed.
Figure A5: Llama-2 hallucinates function responses and produces incorrect descriptions. An example is shown for a synthetic neural module. Llama-2 does not correctly call the function and hallucinates function calls, outputs, and a function description in a single message, before any function outputs are returned.
F Evaluation
F1 Vicuna-evaluator
Utilities for reproducing all baseline results are included in the GitHub repository. The unit testing evaluation protocol (for strings and synthetic neural modules) requires hosting vicuna-evaluator. We provide the finetuned vicuna-evaluator checkpoint as well as the training dataset for download via the FIND GitHub. Vicuna-evaluator was finetuned from vicuna-13b-v1.1 on a dataset simulating the unit testing procedure on ground-truth descriptions of all string functions and synthetic neural modules in FIND. Ground-truth descriptions of each function were generated using GPT-4 (temperature = 0) prompted to write a natural language description of function behavior when shown code implementing the function. Unit tests in the training dataset include one representative sample of function behavior (an I-O pair, or a single input for entity functions) and two distractors sampled from other functions of the same type, and the correct answer for the evaluator (the index of the representative sample). Indices of representative examples and distractors are randomized across unit tests. We finetune on a dataset size 30K with examples uniformly distributed between entities, relations, and string functions. We train for 3 epochs with a cosine learning rate scheduler and a 2e-5 maximum learning rate on 8xA100 GPUs, using other default hyperparameters from Zheng et al. [2023].
We compare the reliability of different LM-evaluators by performing the unit test procedure using the ground truth function description instead of the description estimated by the interpreter. A perfect evaluator would distinguish the correct input-output pair from distractors in  $100\%$  of cases. Table A3 compares the performance of vicuna-evaluator on ground-truth function descriptions to the performance of GPT-4, GPT-3.5, and pretrained Vicuna-13b on a test set of 10 representative examples (and two distractors sampled uniformly at random from other functions in the same category) per function. GPT-4 is a strong evaluator, however is impractical for use with this benchmark due to reproducibility challenges and API costs. While pretrained Vicuna is near chance  $(33\%)$ , vicuna-evaluator shows near ceiling performance in all categories.
F2 Vicuna-evaluator agreement with human judges
We tested whether scoring implemented by vicuna-evaluator agrees with human judgments by having humans perform the same unit-testing task as vicuna-evaluator for a subset of functions in the dataset. That is, we provide the estimated function description (e.g.  $f(x)$  maps countries to capitals) to the evaluator, as well
Table A3: Accuracy of LMs as evaluators on ground-truth functions
Strings	Entities	Relations
Vicuna	0.345	0.390	0.400
GPT-3.5	0.680	0.955	0.888
GPT-4	0.900	0.964	0.952
Vicuna-Evaluator	0.956	0.981	0.998
as three example I-O pairs: one execution of the ground-truth function (e.g. Germany  $\mapsto$  Berlin) and two randomly sampled distractors (I-O pairs for other FIND functions of the same type; e.g. Germany  $\mapsto$  Europe, ruby  $\mapsto$  red). The evaluator (human or LM) selects which input-output pair matches the functionality of the description from the interpreter. If the language description is accurate and useful, the evaluator should select the ground-truth input-output pair as the best match.
We ran human evaluations on the scoring of descriptions of synthetic neural modules (descriptions produced by GPT-4), where human subjects can apply common knowledge to perform the unit testing task (numeric and string functions may require additional expertise or computational tools). We ran this experiment on a subset of 25 functions from the dataset in each subcategory, and the same sample of 10 ground truth and distractor outputs as seen by vicuna-evaluator, for a total of 250 tasks per function. Three human subjects completed each task. Subjects were recruited from Amazon Mechanical Turk and were required to be "Masters" (with U.S. location, HIT acceptance rate of at least  $95\%$ , and a history of at least 100 HITs). Workers were paid $0.06 per task.
To evaluate the baseline accuracy of both vicuna-evaluator and human subjects on the unit-testing task, we provide the ground-truth function description instead of the function description produced by the interpreter (ceiling accuracy on this task should be 1.00). Performance of humans and vicuna-evaluator is shown in Table A4. Humans are able to perform the task correctly for synthetic neural modules (entities and relations), scoring close to ceiling accuracy: mean accuracy 0.975 (SD 0.029) and 0.968 (SD 0.035), respectively. Human evaluators also score automatically generated interpretations (from GPT-4, see average scores in Table A4) comparably to vicuna-evaluator: we find significant positive correlation between per-function scores from human evaluators and vicuna-evaluator for both entities (Spearman's  $\rho = 0.904$ ,  $p = 5.97e - 10$ ) and relations (Spearman's  $\rho = 0.89$ ,  $p = 4.09e - 8$ ). These results show that in function categories where humans can apply common knowledge to reliably perform the unit testing task, their judgments agree with vicuna-evaluator, demonstrating (like in Chiang et al. [2023]) that LM judgments can be used as surrogates for human judgments of automated interpretations.
Table A4: Vicuna-evaluator agreement with human judges.
Evaluator	Entities	Relations
ground-truth	interpretation	ground-truth	interpretation
vicuna-evaluator	0.992	0.660	1.00	0.715
human evaluators	0.975	0.708	0.968	0.700
F3 Discussion of unit testing limitations
As mentioned in Section 5, unit testing (which selects for representative examples of function behavior and not exact implementation matches) is more forgiving of vague or general function descriptions, or descriptions with minor inaccuracies. We hypothesize that this explains the trend visible in Figure 5, where the evaluator scores slightly higher on relation functions with domain corruptions than atomic relations. We note that in almost all cases, the interpreter failed to uncover the domain corruption (likely due to limited sampling), and its description of the corrupted relation function was very similar to its description of the uncorrupted function. For use cases of FIND that focus on whether an interpreter can detect a corrupted part of the domain, unit testing specificity could be increased by including additional representative samples from the corrupted part of the domain (we include 2 out of 10).
F4 Using description matching for evaluating language-based interpretations
We experimented with having an the evaluator directly grade the agreement between the interpreter's description of the function and its ground truth description. We found that GPT-4 is able to score the agreement of language descriptions. However, this procedure is also insensitive to small differences between descriptions (e.g. GPT-4 judges that the descriptions "this function is selective for inputs related to road transportation" and "this function is selective for inputs related to transportation" describe the same underlying function). We note
similar behavior for relations with domain corruptions (e.g. descriptions of "maps countries to capitals" and "maps countries to capitals except in South America" are judged as highly similar).
G Prompts
Interpreter prompts for all function categories, system prompts, and Vicuna prompts are provided in the GitHub Repository. Those not shown in the main paper are reproduced below.
Entity function Vicuna prompt
I am going to give you an input. Return only 1 if that input is related to {entity}. Return only 0 otherwise. Do not return any additional information other than the 0 or 1 answer. Input: {input}.
Entity function Vicuna prompt: composition
I am going to give you an input. Return only 1 if that input is related to {entity1}. Return only 1 if that input is related to {entity2}. Return only 0 otherwise. Do not return any additional information other than the 0 or 1 answer.  
Input: {input}.
Entity function Vicuna system prompt
You are an AI trained to identify words related to input concepts.
Relation function Vicuna prompt
I am going to give you an input. If the input is {concept1}, return only {concept2}. If the input is not {concept1}, return 'undefined'. Input: {input}.
Relation function Vicuna prompt: domain corruption
I am going to give you an input. If the input is {concept1}, return only {concept2}. If the input is not {concept1}, return 'undefined'. If {exception}, return 'undefined'. Input: {input}.
Relation function Vicuna system prompt
You are a helpful assistant. You apply real-world knowledge to map input words to output words according to a rule provided by the user. Try to be as precise as possible.
Interpreter prompt: strings
Hi, your job is to interpret a function  $f(x)$  that is implemented in the {function.py} script.  $f(x)$  takes string inputs. All you can do is call  $f(x)$  on an input string by running PYTHON(function.py string). Try to describe the function implemented by  $f(x)$  by running experiments on it. You can call the function on multiple inputs at a time by running PYTHON(function.py string1 string2
string3...). We encourage testing a large range of inputs before writing a description. The goal is to find a good simple description of  $f(x)$  that explains most of the function behavior.  $f(x)$  may combine multiple different operations. Your description of the function should be in the following form. [DESCRIPTION]: Describe what the function is doing in language.
[CODE]: Write Python code that approximates the behavior of the function
Interpreter prompt: relations
Hi, your job is to interpret a function  $f(x)$  that is implemented in the function.py script.  $f(x)$  takes word inputs, and applies a rule to the input word. Try to describe the rule implemented by  $f(x)$  by running experiments on it. We encourage you to test a diverse selection of words spanning real-world concepts like places, events, and people. You can call the function on multiple inputs at a time by running PYTHON(function.py word1 word2 word3 ...). We encourage running many tests before writing a description. The goal is to find a good simple description of  $f(x)$  that explains most of the function behavior. There may be some inputs where you expect the function to behave a certain way, but it does not. If that is the case, describe the part of the domain where the rule the function normally applies, does not hold. Your description of the function should have two parts.
[DESCRIPTION]: Describe in language the rule applied by the function.
[DOMAIN]: Is there part of the domain where the rule the function normally applies, does not hold? If so, describe that corruption. If not, describe the domain of the function where the rule applies.
Interpreter system prompt (GPT)
You now have access to some commands to help complete the user's request.
You are able to access the user's machine with these commands. In every message you send, include
'COMMAND:' with your command at the end. Here is a list of commands with explanations of how they are used:
Command: [PYTHON(function.py)]
Explanation: [Run a python script with the given file name. Use quotes for the filename argument. Do
not use quotes in the function command itself.]
When you use a command, the user will respond with 'Response:' followed by the output of the command.
Use this output to help the user complete their request.
Interpreter system prompt (Llama-2)
You now have the ability to execute commands to run external functions. You are able to access the user's machine with these commands. In every message you send, include 'COMMAND:' with your command at the end. Here is a list of commands with explanations of how they are used:
Explanation: [Run a python script with the given file name. Use quotes for the filename argument. Do not use quotes in the function command itself.]
When you use a command, the user will respond with 'Response:' followed by the output of the command. Use this output to help the user complete their request. After you receive a task from the user, you must execute PYTHON(function.py) to run the external function. You will then receive outputs from the external function to analyze. You must only analyze outputs produced by the function when you run PYTHON(function.py). Do not run the function any other way. Do not analyze any other outputs besides the ones produced by running PYTHON(function.py).