Title: FIND: A Function Description Benchmark for Evaluating Interpretability Methods
Benchmark: FIND
Summary: Evaluating the ability of LLMs as interpretability agents as a proxy for how well they might perform in automated interpretability pipelines. i.e. can LLMs recover the functions from input/output data alone.
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: Automated interpretability capabilities
Phenomenon defined by authors: No
Definition scope: Comprehensive
Definition integrity: Single cohesive phenomenon
Definition integrity detail: Not applicable
Phenomenon contested: Contested
Purpose notes: Very broad phenomena "evaluating automated interpretability methods"
Task definition: Recover the details about a program (e.g. a maths program/string program...) from examples of just the input and outputs. 
Task item definition: A ground truth function and an operator that might add some noise to the data. Operationalised in an agentic way where the model can call the function many times and examine its behaviour.
Task definition detail: Very simplified version of the real-life phenomena.
Task source: Procedurally-generated task examples (e.g. Creating instances from a template)
Task ecology: Constructed task (e.g. predicting medical diagnoses from clinicians' notes)
Task ecology detail: 2,275
Dataset metadata available: Yes
Dataset metadata detail: Type of function (numeric, string, and synthetic neural modules)
Sampling method: Random sample (creators defined a task space and sampled from it), Targeted items (creators defined a task space and chose tasks within it strategically)
Split reviewed: Test
Response format: Free response (e.g. summary paragraph, executable code)
Metric definition: Exact Match (accuracy, F1, precision, recall), LLM-as-a-Judge (text quality, preferences, NOT extracting answers for other metrics), Execution-based (unit tests)
Metric aggregation: Simple Mean
Metric statistics: Mean
Metric subscores: Yes
Metric subscores detail: Function type (numeric, string, neural modules)
Metric access: Outputs alone
Few-shot evaluation: No
Authorship: Academia
Benchmark availability: Yes
Task face validity: Highly simplified version of the phenomena
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: No
Results comparison explanation: No comparisons made
Results realism: No
Human baseline: No
Authors discuss validity: Yes
Author validity detail: Discuss that the benchmark is a test of "necessary, but not sufficient, capabilities for automated interpretation." "The ultimate test of these interpretation methods’ effectiveness must be their ability to generate actionable insights about real models, which FIND does not evaluate."
Taxonomy: Reasoning / 
New taxonomy: General Capability / Logical Reasoning