AmadeusGPT: a natural language interface for interactive animal behavioral analysis
Authors: Shaokai Ye
Jessy Lauer
Mu Zhou
Alexander Mathis
Mackenzie W Mathis
Venue: NeurIPS 2023 poster (2023)
Keywords: ChatGPT
GPT3.5
GPT4
behavioral analysis
LLMs
human-AI interaction
behavioral neuroscience
TL;DR: AmadeusGPT is a natural language interface that turns language descriptions of animal behaviors into machine-executable code, utilizing a novel dual-memory mechanism augmented GPT3.5/4 to overcome its context window limitation

Abstract: The process of quantifying and analyzing animal behavior involves translating the naturally occurring descriptive language of their actions into machine-readable code. Yet, codifying behavior analysis is often challenging without deep understanding of animal behavior and technical machine learning knowledge. To limit this gap, we introduce AmadeusGPT: a natural language interface that turns natural language descriptions of behaviors into machine-executable code. Large-language models (LLMs) such as GPT3.5 and GPT4 allow for interactive language-based queries that are potentially well suited for making interactive behavior analysis. However, the comprehension capability of these LLMs is limited by the context window size, which prevents it from remembering distant conversations. To overcome the context window limitation, we implement a novel dual-memory mechanism to allow communication between short-term and long-term memory using symbols as context pointers for retrieval and saving. Concretely, users directly use language-based definitions of behavior and our augmented GPT develops code based on the core AmadeusGPT API, which contains machine learning, computer vision, spatio-temporal reasoning, and visualization modules. Users then can interactively refine results, and seamlessly add new behavioral modules as needed. We used the MABe 2022 behavior challenge tasks to benchmark AmadeusGPT and show excellent performance. Note, an end-user would not need to write any code to achieve this. Thus, collectively AmadeusGPT presents a novel way to merge deep biological knowledge, large-language models, and core computer vision modules into a more naturally intelligent system. Code and demos can be found at: https://github.com/AdaptiveMotorControlLab/AmadeusGPT

AmadeusGPT: a natural language interface for interactive animal behavioral analysis
Shaokai Ye EPFL Geneva, CH
Jessy Lauer EPFL Geneva, CH
Mu Zhou EPFL Geneva, CH
Alexander Mathis EPFL Geneva, CH
Mackenzie W. Mathis EPFL
Geneva, CH
mackenzie.mathi@@epl...
Abstract
The process of quantifying and analyzing animal behavior involves translating the naturally occurring descriptive language of their actions into machine-readable code. Yet, codifying behavior analysis is often challenging without deep understanding animal behavior and technical machine learning knowledge. To limit this gap, we introduce AmadeusGPT: a natural language interface that turns natural language descriptions of behaviors into machine-executable code. Large-language models (LLMs) such as GPT3.5 and GPT4 allow for interactive language- based queries that are potentially well suited for making interactive behavior analysis. However, the comprehension capability of these LLMs is limited by the context window size, which prevents it from remembering distant conversations. To overcome the context window limitation, we implement novel dual-memory mechanism to allow communication between short-term and long-term memory using symbols as context pointers for retrieval and saving. Concretely, users directly use language- based definitions of behavior and our augmented GPT develops code based on the core AmadeusGPT API, which contains machine learning, computer vision, spatio-temporal reasoning, and visualization modules. Users then can interactively refine results, and seamlessly add new behavioral modules needed. We used the MABe 2022 behavior challenge tasks to benchmark AmadeusGPT and show excellent performance. Note, end-user would not need to write any code achieve this. Thus, collectively AmadeusGPT presents a novel way to merge deep biological knowledge, large-language models, and core computer vision modules into a more naturally intelligent system. Code and demos can be found at: https://gi thub. ctmo http:///ddapttieeeooooooooooooaa
Introduction
Efficiently describing and analyzing animal behavior offers valuable insights into their motivations and underlying neural circuits, making it critical aspect of modern ethology, neuroscience, medicine, Yet behavior is complex. often multi-faceted, and context-dependent, making it challenging to quantify and analyze [5) The process of translating animal behavior into machine-readable code often involves handcrafted features, unsupervised gre-processing, or neural network training, which may not be intuitive to develop for life scientists.
and technology I
To understand animal behavior needs to complete series of sub-tasks, such obtaining animal poses and identities, object locations and segmentation masks, and then specifying events or actions the animal performs. Significant progress has been made in automating sub-tasks of behavioral analysis such as animal tracking 7
g 9. object segmentation [10 11], and behavior classification 3
4
yet behavioral phenotyping requires additional analysis and reasoning
16
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

17 8].
This is typically done with feature computations such as measuring time spent in regions of
interest or general statistics of locomotion 13
g 18
The challenge of making behav-
ior analysis accessible and inter-
Input Video
Query
pretable is hindered by the diffi- culty of combining task-specific models and the lack of an intu-
"When is the mouse the treadmil1?"
itive natural language interface to produce machine code. In an ideal scenario, behavior anal- ysis practitioner would be able to explore behavioral data, de- fine their desired actions using natural language, and visualize captured behaviors without need- ing to learn how to train mod- els, write scripts, or integrate code bases. Our framework, AmadeusGPT, takes the first step towards achieving this goal. AmadeusGPT provides natural language interface that bridges users' prompts and behavioral modules designed for sub-tasks of behavior analysis. Our results show that AmadeusGPT outper- forms machine learning-based behavior analysis classification
AmadeusGPT
GP73.5 BLK
OpenAI
core behavior API
Genecate Code baxed Integrations
clang wntorkeallee
def taka, pww.reagnann
gm axtert AANALL Laenune)
nomes (13813
suitala SEAAA erNCASTALLE ctase
return did, an
('421
Python Interpreter
powe watiination
ab ject secaentation Lon
00 SAM
protrained DEC model
use pretrained SAM mode:
XOupput
"Result for query : 12,836 frames the mouse is on the treadmill (total video length is 107,800 frames)
tasks in the MABe [9] bench- mark by using prompts highly similar their official defini- tions of each behavior, namely with small modifications and only three tunable parameters.
Figure 1: AmadeusGPT: LLM, computer vision, and task reasoning platform for animal behavior. Users input video and prompts and AmadeusGPT queries API does and reasons about which models and analysis to deploy.
AmadeusGPT offers novel human-computer interaction ap- proach behavior analysis, providing a unique user experience for those interested exploring their behavioral data. Through natural language prompts. users can ask questions. define behaviors on-the-fly, and visualize the resulting analyses plus the language output (Fi We show that with our novel dual-memory mechanism, defined behaviors are not lost (when exceeding the token limit), wording can be automatically rephrased for robustness, and the state the application can be restored when relaunched, providing seamless and intuitive use (Figures
To capture animal-environment states, AmadeusGPT leverages Mate-of-the-an pretrained models, such as SuperAnimale (17] for animal pose estimation and Segment- Degment-Anything (SAM) for object segmentation (TT]. The platform enables spatio-temporal reasoning parse the outputs of computer vision models into quantitative behavior analysis. Additionally, AmadeusGPT simplifies the integra- tion of arbitrary behavioral modules, making it casier to combine tools for task-specific models and interface with machine code.
2 Related Work
Large Language Models. In recent years there has been surge in the development of large language models (LLMs) 20 21 22 23 that show exceptional abilities in natural language processing tasks such as language generation, interaction, and reasoning. One notable example ChatGPT, which is built on GPT-3+ and trained on massive corpus of text data 22 ChatGPT has demonstrated impressive performance in a wide range of natural language processing tasks, including text classifi- cation, language modeling, and guestion-answering. In addition to language processing, LLMs have

---

also been applied to other domains such as image and video processing 24), audio processing, and natural language generation for multi-modal inputs 24 25.
LLM Integrations. Recent works leverage LLMs to generate code for computers to execute, including visual programming with VisProg [26] and ViperGPT [27]. and HuggingGPT [8 that exploits pre-trained AI models and smartly handle API-GPT interactions [26] 27 However, animal behavior analysis lacks a natural language-computer vision pipeline and requires diverse open-source code-base to complete the sub-tasks. Concurrent work by Park et al. [29] on Generative Agents uses a dual-memory system and iterative chatting, but has a more expensive execution flow compared to our approach, which uses memory mechanism with symbols as pointers.
Behavioral Analysis. Machine learning and computer vision techniques have increasingly been employed in animal behavior analysis in recent years 7 9 12 30
32). These techniques offer advantages such as automation, high throughput, and objectivity, compared to traditional methods that rely on manual observation and annotation [T. Deep learning models, such as convolutional neural networks (CNNs) and transformers, have been utilized for feature extraction and classification of animal behaviors in various domains such as social behavior, locomotion, and posture analysis (reviewed in (2). Yet, universal interfaces to the plethora of pose estimation tools and downstream analysis methods are lacking.
We used rule-based approach that leveraged the pose estimation outputs to compute behaviors, similar to the approach used in LiveMouse Tracker (LMT) [33]. However, unlike LMT, which requires users interact with code and requires specific hardware collect data, AmadeusGPT is agnostic to data collection and provides a natural language interface that is easy to use and customize. Additionally, unlike LMT, AmadeusGPT includes object segmentation, enabling it to capture animal-object interactions.
Task Programming for Behavior. Task programming for behavior analysis 30 aims leverage the domain knowledge from experts to help extract the most relevant features that are useful for the downstream behavior classification task. However, task programs were only considered as Python functions that help extract better features (i.c., features built from poses). In this work, we generalize the scope of task programming: any sub-task that can be achieved as machine- eeccuteeexecutable Python code considered task programs in AmadeusGPT, and can be queried and developed with natural language. Thus, we name the generated function in AmadeusGPTT (target) task programs.
AmadeusGPT
Lang term memory
loader
Short term memory
Memory
Visualization
SAM
Error handling
Embeddlings
SuperAnimal
Robustness testing
Reflection
AmadcusGPT.
Integration
Animal & objects
Matplottib polygons
Rephreser
modules
Data flleering
Self correction
Kinematics
Explainer
Behavior LLM
Bodypart kiremati
Dynemic module matching
Sperio-lempcrel reasoning
Rule-oesed benavior capturing
Embadding LLM
Figure 2: Schematic of AmadeusGPT design and features.
AmadeusGPT is an interactive human-computer platform that allows users to describe in natural language the behavioral analysis they want to be performed. It utilizes ChatGPT as the user-guided controller and a range of machine learning and computer vision models as collaborative executors to analyze animal behavior, starting from raw video and leveraging new pretrained pose estimation models that can run inference across species and settings [7]. and powerful objection segmentation models (SAM (TT) (Figure 2 Here we focus primarily mice, but also show it works for horses to demonstrate that nothing precludes AmadeusGPT to be used on a range of animals.
AmadeusGPT uses LLMs to generate Python executable code that fulfills user-specified queries in the prompt. Building such a system requires LLMs to learn to manipulate core process resources in constrained way. If the user's prompt is unclear or beyond the system's 's capacity, the generated code might result emrors that require programming expertise. Intuitive error messages are therefore essential for ensuring consistent natural language experience, while maintaining the ability for users
3

---

(a) Short questions leading to active refinement and decomposition into to multiple task programs
Queries
AmadeusGPT
Short-tern menory
long-term momory
corputar vision modela extract pase and objecta
Query 1
"When in the monee on the treadnilly"
specicies object "Ereadrill"
Genorated code
Query
4 Chat Output:
"Plot the correspending
for quezy
object ethogran"
Query 3
"Plot the trajectary of the of sninel"
API DOCk
Pythos Interpreter
API Implementation
Seroe
video length 107,
Visual Outputs: quazy
Code Outputs:
ded teiloprogran)
abjent
Bszavior www.yaaaaaa... www.sie, wbw.ect.bttt.....oo..... wwwcta("11""
'oveciap" beoypastenn "=1l"")
Fvent, event.eventend
www.eeeteeeeeettoooooooooo _8 18_evesse)
Fraens, on jaje mu..aajfeeee _atteeee_______ wonk)
Pue quaryy 2: ded task park_progran))
antariie_ammiyyut bainelleeaaccccnnnllssssss
['1a')
query
[ig, MM behavior_ wwl.yllllllo.o. wwwpppooteoccooooooooooo obfeet (objee (objeel___esss]) (MRES) return fig, ax
Ithogran for Object
Animel overlup
7or query 3: 1. The etrer zeccage suggette that the bodypart "center" 15 not detined in the data. Check if "center" is valid andypart for the animal in the data. 2. It "center" is not valid bodypart, try using different bodypart that supported the data. 5, Osca you have idantified valid bodypart. updase 3odypart instead of "center".
2350 oec (693 E36 120010001100
tzak_ progran() function to that
(b) Long description decomposed to multiple task programs Query
Feature ustraction
Behavior definttonn
Visuslization
Casput pase
/heed dipal> behastor animal mouse, center and neck are in the and redtpoint
Plct Irajectory"
*des poss from the video define <I hoad dipal> behavior visre amimal nouse centaz and neck are coid its head sidpoint
Preprocessing
Behavfor
roil. than peint the count Erajuctory the happens, at und,
zany dips
K Code Outputs:
animal's head nidpoint when haad, dipa happans."
(aok_proggaac)
betarior enwlynie Antms Bot.w//aaasyrooeaaaitte get ecenis head dipe Lappene
behavlor, www.aaalllle.....o anteals object, coenter "ROZD" negater/alan)
bedyporssa "mouse_ "mook")
Iocal, wwol_owbaddcee. ru. cveeee
"overlap bolypprrd-i "nooo__idlpentt" nepatextror)
hoad.
Breat, add add siaatttoooo wwt.://aaaaeecccp..ooo.a.
count coeecttcccccseee
tineu head dips huppezs
Intrratti::
Run pose escimation SuperAr modill
Piat trajectory of head cips
coont bead
&orwwng ROIs
tatp
e ploc tan trajectocy of haad dipa
zig.
heed dip bout and bout
hhad acdpnlen evenssatiod silp eveenss) contt_send_dppe.
Figure 3: AmadeusGPT: natural language model enhanced behavioral analysis system. (a) Users can start by uploading a video and asking a question to AmadeusGPT about what they want to do. AmadeusGPT will run computer vision models if the target task depends pose estimation (e.g.. the posture or location of the animal) and object segmentation (e.g., when does the animal overlaps with an object) and/or ask clarifying questions. Once correct, the user also save these queries/code outputs. Three example queries and outputs are shown. (b) Alternatively, users can provide a detailed recipe for how they want the data analyzed. The colored text highlights action-items for AmadeusGPT (matching the colored boxes).
to iteratively refine generated code with language. Therefore, build AmadeusGPT we leveraged GPT3.5, augmented it, developed dual mual-memory system and core behavioral utilities that together make AmadeusGPT an end-to-end, human language-Al AI system (Figure
3.1 LLMs as natural interfaces for code development and execution
ChatGPT (i.e., with GPT3.5 or 4) [23 able to generate code that corresponds to users' prompts due to instruction tuning paired with access to large amount of code in its training data. However, the native ChatGPT/GPT3.5 or API cannot serve as the language interface for behavior analysis for the following reasons: (1) it cannot work with private APIs that not in its training data; (2) it

---

hallucinates functions if too many implementation details are exposed or asked to be provided; (3) the context window does not have sufficient capacity size for complex tasks that require reading large source code files; (4) it has no Python interpreter to execute the code it suggests. Therefore, we
built an augmented version of GPT to overcome these limitations (see Sections 3.2 and 3.3
3.2 GuT and API design
AmadeusGPT sends its API documentation to ChatGPT in order to smartly constrain the output code and immediate Python execution and thereby always augments the data available to GPT3.5 or GPT4. In the API documentation, implementation details are encapsulated and only the function documentation is exposed (see Appendix for example API docs). Importantly, there is an "explanation prompt" with each function example that serves as a hint for GPT3.5 to understand what the API is for. This separation of API documentation and implementation has two advantages: it reduces token use, and prevents hallucinating resources that do not exist in the core functions or modules linked to AmadeusGPT. Concretely, we prompt GPT3.5 to strictly follow our API (see Appendix). This design improves the reliability of code generation while imposing quality control.
We followed three core principles when designing the API. Firstly, we developed code with an atomic API that consists of simple functions designed to perform specific operations (Figure 2). These functions API functions to create more complex functionality. The use of atomic API improves the readability of the code for users and developers and reduces the chance of LLMs confusing ambiguous API calls. Secondly, we leverage polymorphism in code design to generalize our API to variants of a sub-task routed by parameter values and input types since it is not desirable realistic to provide an example code for every possible sub-task. Thirdly, to make AmadeusGPT cover a range of behavioral analysis tasks, identify core behavioral modules that cover common behavior analysis sub-tasks and additionally use integration behavioral modules for task-specific sub-tasks.
modular, reusable, and composable, making them easy to combine with other
3.3 Dual Memory Mechanism
As GPT3.5 is implemented with a transformer architecture that has context window size limitation of 4,096 tokens for hosting the history of conversations (22). This would limit how far back AmadeusGPT can use the context, such as a user-defined behavior or its own generated code. We demonstrate in Figure f that without tackling the context window limitation, forgetting will happen, which results in wrong code that ballucinates functions or variables that do not exist.
To tackle this limitation, we implemented dual-memory mechanism that consists of both short-term memory and long-term memory banks. In this work, short-term memory is implemented as a dynamic deque that holds the chat history, pulling in new tokens and removing older ones when full (i.e., beyond the 4,096 tokens). At inference time, all text in the deque is sent to the ChatGPT API call.
Long-term memory is implemented as dictionary in RAM. The keys of such a dictionary are called symbols in this work. We define read and write symbols to instruct the communication between short-term memory and long-term memory. With the help of regular expression, symbol that is enclosed by < || >> triggers a memory writing operation that writes the context of the corresponding chat into the dictionary using the symbol name as the key. As in Figure 3 keyword triggers memory writing that stores the whole sentence under the key name "head dips". Similarly, <> triggers reading operation that pushes the stored sentence into the front of short-term memory with an additional prompt to remind GPT3.5 that the retrieved memory is used for context instead of formal instruction. This allows read operation retrieval from the long-term memory to be used as a context for a new instruction. Additionally, with the long-term memory, AmadeusGPT can store its states into disk and the user can restore the states after re-launching the application (See Appendix). Collectively, we call this augmented-GPT3.5
head dips >>
3.4 Core Behavioral Modules
The language query feature of ChatGPT inspired us to provide more natural way for humans to perform behavior analysis. Namely, provides users with a platform to perform behavior analysis by asking questions or instructing tasks in an interactive manner. We imagine that users will either ask question or provide longer instructions to complete a task (Figure In addition, users ask follow- up questions to a previous answer from AmadeusGPT, it attempts answer with executable code
5

---

(a) Memory operations
Copylig from short- t-erm memory long- larm memory
Retrioval from long term memory
Active forgatting short- term memory
Dynamicslly loading integration modules
Deline sccontecil as where two aninals
econtaat
Compute SU embroding using cebra
inBalization
Text embediing
Short- tarm memory
Long- memery
Ga bagg collection
<||>
Syymboll Read symbol
errbeddingg
impiemented process momery clikk
Long tomm memory Short toomttorm momory implemanted by dequeue context GPTS.
(b) Short term memory retrieval
(d) Long term memory retrieval
norber boste for head diga*
Ocent: number bouts for <head dipa- progeatt))
beter for ane lysis Asime www.eeeocteeaaaeeeoo (ynin()
behevior, eww.yyissll...................s. faina www.beeeevvichaaapoo lysin head dips rot avanta Eabavise scalynis aninala object emensa))
bead dips roi areass batacioe andlyaia an cbfect eventst
bodyparcaml "sorae__enterr"
bodypartsn| "acaas_eentterr
regateural )
megtideralnn)
baad, dips sead avents bahavior, www.yass, antnatnn sneneet
head, dips 2ead events behavior www.ynsa animaii aventa)
bodypuren nospoiin""""""""
bosypartse ("teet_iieeei
sepate-trus)
segasertirre)
depp_ _rent_ Ivent, stmestaneoo aventat
head_dips, _event Event traittennns aventat
head, hoad,
head lead evenevttte)
count. basd, dage behavior soslsyla wallayyy..notttt. conntt South Soutntbredd avpentt
count bead, dipa behavior, anslyssa count_ (tata (twdd_dips_.ddoneee.e mectt))
return court_heed_uu
return ceurt_hrer_ddpp
(c) Forgetting
Many Herations cycles later, contect window hul and mode sutjui is not-llnnciiosl
of hoste head dipe"
evar lave tre Vekorier you ont Stamer sedde teoeell (leastine))
coaticate Aiee Besiicie www.enneeeaaaaa.......
Figure 4: Dual Memory Mechanism for augmenting GPT3.5 or 4. (a) We introduce long- term memory module that overcomes running out of tokens, and a dynamic loading system for code integrations such as for advanced uses like dimensionality reduction with UMAP [4] or CEBRA (35) (see Appendix). (b) An example query and output from short-term memory only (early in interactive session), (c) if long-term memory is ablated after running out of tokens the output is non-functional. (d) Yet, with our new memory system, it can easily retrieve the identical, correct output within long session upon restarting.
called "task programs" that are executed by the backend Python interpreter. In the API documentation mostly specify code examples for uni uni-purpose task programs (such as counting events). However, multi-purpose purpose task programs (such as computing events and interactions with objects over time to produce plots).
we show in Figure that our augmented-Gri -PT33555 API is able to compose multi
Figure 33 shows how the user can give complex instruction that covers multiple sub-tasks, including pose extraction, behavioral definitions, interactively drawing regions of interest (ROIs), then visual- izing and performing tasks, such as behavior event counting. In this case, AmadeusGPT is able to decompose the description into multiple task programs and assemble the final program. Alternatively, the user can also ask a question "When the mouse on the treadmill?" or "Count the number of head dips per ROI" as follow-up queries to AmadeusGPT's answer, change the color map of the previous plot, etc. This all relies on the core behavioral modules and their ability to be combinatorically used (Figure 2).
Our core behavioral modules try to cover the most common behavioral sub-tasks. Generally speaking, typical behavior analysis task asks question about what animals do through a spatio-temporal space. This can simply be the amount of time that an animal spends moving in a particular ROI, to more advanced analysis like measuring amimal-animal varying levels of key-point tracking, object segmentation, and video (temporal) reasoning. Thus, we implemented the following core modules and always send them with the queries during augmented- GPT API calls (Figure 2.
animal-objec interactions. This requires
Kinematic feature analysis. As many behaviors are concerned with the kinematics of bodyparts, we provide a module for kinematics analysis. This includes filtering the pose data, calculating speed, velocity, and acceleration. It also includes movement-ban based statistical analysis, such as queries related to computing the amount of time spent locomoting (i.e., velocity over some minimal threshold computed across the video, or time spent in an ROI).

---

Animal-object enviroument states. To capture animals' and environment states, we deploy pretrained computer vision models such as pose estimation with SuperAnimal models [7] and objects with
SAM [1]] Note they can be easily substituted by tailored supervised models. To support end-to-end behavior analysis with raw video only, we use them as the default models for their wide coverage of animals and objects. We also implemented code infrastructure that abstracts "static objects" and "moving objects" to represent objects in the environment as well as customized ROI objects and animals respectively. The animal-object relation and animal-animal relation are modeled and saved in lookup tables. These relations mostly cover binary spatial relations such as "to the left", "to the right", "overlap", "orientation", and numerical spatial relations such as "distance", "angle" (i.e., the angle between animals, based on computed neck-to- to-tailbase body axes) and "gazing angle" (i.e., the angle between animals' head coordinate systems, determined from the nose-neck line) (see Appendix).
Spatio-temporal reasoning. After computer vision models track animals and segment objects, their outputs are used to build internal spatio-temporal relation tables among animals-animals and animals- objects. We provide code infrastructures to support queries of events (i.e., sequential spatio-temporal relations and simultaneous spatio-temporal relations). Users can query events where animals move from one state to the other (see also Figure 5p).
3.5 Integrations Beyond Core Behavioral Modules
Integration modules aim to cover task- specific behavior analysis sub-tasks such as dataset loading, embedding calculations, and visualization tools. Because they are task-specific, we do not send the API documentation of these modules. Instead, we rely on "dynamic loading" to load only few of the selected API documents for integration modules at inference time. To allow for dynamic loading, we use the embedding API from OpenAl to turn API documents of integration modules into text embedding vectors and cache them in RAM or disk. The user's prompt then acts as a query to retrieve & (a hyperparameter) most relevant integration modules by cosine similarity (Figure We also allow users to manually load integration modules by prompting "loading module 'module-pathy if users want save the cost of using the embedding API and/or they are creating their own modules.
Error handling. We include error handling to help users understand when instructions are beyond the system's 's capabilities or ambiguous. Here, AmadeusGPT forwards prompts, eπтоr messages, and API does to the ChatGPT API for natural language explanations (Figure 3h, Query 3 example).
Rephraser. Users can ask questions with language that might be very different from those in our API docs, which can cause performance degradation due to -uito-ddistrbuttin prompts (See Section To overcome this we leverage a native GPT3.5 (i.e., without our augmentation) that we call "Rephraser" that is tasked to turn users' expressions into a style that is similar to that found in our API docs. We wrote in the system prompt of Rephraser a few examples of such rephrasing, hoping the GP7 can learn it via few-shot learning. Therefore, Rephraser tasked to act as test-time domain adaptation component for system.
Self-correction. There are incidences where CharGPT API makes obvious mistakes that cause runtime errors, therefore we implemented a self-correction mechanism to help ameliorate this. When there an error executing the generated python code, we forward the error message and the output of our error handler to an independent ChatGPT API connection with a system prompt that encourages it to revise the function. The number of times for such retrying is flexible, but setting this to three in practice generally improves the success rates (See Appendix for an example). We keep the history of retries the context window to help it to learn from failures it takes more than one trial to correct the code. Note that ChatGPT API does not have read/write access to the implementation of our APIs thus self-correction cannot correctly revise the code if the errors stem from our code.
Explainer module. AmadeusGPT takes users' queries and generates Python code to address the queries. The executed function returns results of multiple data types, including plots, strings, numpy arrays, etc. However, it might not always be straightforward for users to link those returned results to the queries. Moreover, the users do not know how much they can trust the returned results. In many cases, checking the generated code can help. However, inspecting the code requires python programming knowledge. Therefore we add and explainer module, which is another L.M whose job is to explain to the users how to link the results to the queries. The explainer is implemented as an independent ChatGPT API connection with its independent system prompt. the system prompt, we ask the explainer to take the thought process parsed from the code generator, the return values
7

---

(a) EPM
Code Outputs
(a.1) EPM: GT vs. A
Query 1:
bebavior analysis Aniim./aebaaiirrkkaalyyaagg
taak_ prograoo))
"How auca time does the spend
hebavior asalyaia. antmala obfect events(
'overlap" bodyparis- "eil"),
negole-Falsc)
j0__00i__ERR_ovett
A Chat Output
Results for query 12376 Erenes
(total video length is 15076 franes)
natert
2000
4000
8000
10000
14000
Query 2:
Code Outputs
"Bow much time does the mouse spend outside
velavior__asalyen Buttside_ROI__cee
-nttoolBehhrrrorhssasyssss) behavlor, waw.rrrrannllllssseeee..e
pasimals_objeet.. pveatee 'ROzO", "overlap", ('allatsss"""a""a)"aaaa"""""a'a' negute-True)
K Chat Output 2:
sodyparts-
outside_ _OIC_ evente
Results for query 2700 franes
(total video length trames)
(b) MausHaus
nurber
& Query:
c: Query:
&: Query:
"When the animal on the tresdmill, which is object 52"
"Plos the trajectory the sninal."
"When the animal close the object 35, if define close ss Less than 50 pixels?"
L130
def task prograp( )
betavior acalysis Antmeeeehhrroochaatyyss
pperlap_ohjeere evente behavior, analysiee animale, object eventss 's", 'overlap', bodyparte=["ahe ('a11'3)
tig, ax behavior_analyee www.ysie..llllll.oo..o.. plot sxnjectory(bodrpatee all" eventeroverieg wttperrrrpppppoopee_eeeee B_ {_eeent) retarn fig, ax
(c) MABE Challenge dataset
Official definition
Our prompt
Chase
Watching
Chase: ane moving above crrese. spart, mica less than 30 degr begroes, for leazt 80% fremee within leir one secuard, Merge utut lees than seconds aport.
sontal betaelor whace closeet discance between chis animal and other anizain in than other anirala have
"pefine < <]chasee]>
than
30 and this aziasl has to teavel tastes than piaclo.
clases happen?"
Watching: viice are more than cm apart less tirm 20 spant, and suece ofliee, mouse thar deoness from duastian secaantse Merge bouts lesas than 0.5 seconds spart.
"uedine i </satchiing
noctal
betactor whene distance batwees azimale in ieen than pizmis inrger
Luau than The smouth windov
15. Wher. does watetingg heppent*
Figure 5: Results on classical behavioral tasks with AmadeusGPT. (a) Result on EPM showing limited time in the open arms by the mouse. The raster plot is ethogram where there is tick for every related event in time, i.e., the mouse is in the closed arm or in the open arm. (a.1) shows AmadeusGPT counts vs. three human raters [3] across five videos (colored dots). (b) Animal-object interactions can be computed in natural home cage settings. (c) Behavioral images and original description given in MABe, vs. our prompts that produce the quantitative results shown in Table For Chase, we visualize the chasing animal's trajectory that overlaps with the predicted mask and the chasing animal's sss trajectory for the ground-trunt truth mask. For Watching, we visualize the visual cone for each animal from each animal's 's head coordinate (axis by neck-nose vector and its normal vector).

---

T4
T5
T6
T7
T8
T9
T10
T11
T12
Tasks
approach
chase
close
contact
huddles
watch
PCA baseline
0
0.13
0.0008
0
0
0
0
Top-entry 1
4
0,020
0.010
0.708
0,558
0.310
0.0056
0.015
0.013
0.182
Top-entry 2
4
0.026
0.050
0.7072
0.561
0.214
0.0084
0.029
0.023
0.159
Top-entry
4
0.022
0.029
0.655
0.515
0.249
0.0062
0.015
0.014
0.162
BAMS [37
0.02
0.023
0.664
0.533
0.302
0.0045
0.0165
0.014
0.191
AmadcusGPT 0,014
0.274
0.700
0.572
0.380
0.05
0.05
0.024
0.600
Table 1: F1 scores by AmadeusGPT on several tasks from the MABe Behavior Challenge 2022 We do not use representation learning (the aim of the benchmark), only rule-based task programming. *T9: oral-ear-contact; T10: oral-genital-conacte T11: oral-oral-contat..
4
from python as well as the user queries to explain whether the code and its return values meet the queries (See Appendix for an example).
4 Experiments
To evaluate AmadeusGPT ran series of qualitative and quantitative experiments. We used three datasets that highlight standard behavioral neuroscience settings. The first is an open-access Elevated Plus Maze (EPM) dataset from Sturman al. [13]. The second is called MausHaus and is video of a mouse for one hour (108K frames) in an enriched home-cage setting [17]. The third is video data of three mice interacting from the MABe 2022 Challenge
We also include fourth in-the-wild
horse dataset [36].
EPM. The EPM is classical task in neuroscience research test an animal's anxiety about being in exposed "open arms" vs. "closed arms" [13. We show that with minimal prompts and ROI interactive plotting user can measure these events (Figure 5b, also see Figure 3 Here, we query AmadeusGPT to report both open and closed arms and note that the resulting raster plots (ethograms) do not identify the same frames, as one expects it is correct (i.e., the mouse cannot be in both states at once). We also show that AmadeusGPT counts the number of events similar to those reported in ground truth annotations by three raters across five different videos (Figure
MausHaus: enriched mouse-cage monitoring. Studying more natural behavior is becoming important in research settings. Here, we demonstrate intuitive prompts that run pretrained models (SuperAnimal- Top TeeeTToooiieeeon ViewMMuse and SAM) then query spatio-temporal relationships between the model outputs (Figure Sp). As SAM only provides object labels, we use object number or interactive clicking on the object to ground the analysis (see also Figure 3).
Horse gait analysis. AmadeusGPT has no innate mouse preference. Therefore show another application we performed equine gait analysis [36). We leveraged horse videos with ground truth keypoint annotations every frame and used SeatrAnimal-uuadruue [17], and found comparable results within AmadeusGPT to human-level labeling and stride analysis (see Appendix Figure K).
MABe 2022 Benchmark. The benchmark had two rounds for the three mouse dataset. We used the more popular first round (evaluation split) and therefore provided the pre-computed keypoints as inputs to APT In Figure 5c. we show how AmadeusGPT captures two representative tasks from MABe benchmark, Chase and Watching. We use text that is close to the original definition to define the behaviors we want to capture. Note the units in our prompt are pixels for distance, radians for degree and pixel per frame for speed. We tested nine behaviors and report the Fl score as computed in the MABe evaluation (Table I Our approach is purely rule-based, thus machine learning is needed and only three parameters are needed to be given or tuned: a smoothing window size for merging neighboring bouts, a minimal window size for dropping short events, and the pixel per centimeter. Note that tasks that are hard for machine learning models are also hard for our rule-based approach (Table I Tasks 9-11).
We do not intend to formally claim rate-of-the-are on the benchmark, as the goal was to evaluate unsupervised representational learning models. Nevertheless, we show that in practice one can use text definition of behavior to capture behaviors that are on par with, or better than, the competitive representation learning approaches.

---

Robustness tests and stress-testing AmadeusGPT. Users might query AmadeusGPT with expres- sions that are very different from the explanation text we provided in the API documentation. A potential pitfall is that AmadeusGPT overfits to our (the developers) expressions and biases. To test how robust AmadeusGPT is, we crafted five out-of-distribution (OOD) base questions for an EPM video. Then we asked a native GPT3.5 (part of our reflection module, see Figure 2 to generate five variants of each OOD question. We manually checked whether AmadeusGPT generated consistently correct results on those 25 generated questions, and it did 88% of the time with our Rephraser module vs. 32% without the Rephraser. The total number of tokens consumed using the ChatGPT-API was 4,322 and 5,002 with and without using Rephraser, respectively. Note that in both cases, the consumed number of tokens is larger than the maximal context window size of 4,096. This also shows that AmadeusGPT passes the stress-test we set for it in two key ways: (1) The short-term memory deque correctly maintains the constrained size without getting an error from the OpenAl
API due maximal token size error; (2) The diverse questions in the short-term memory do not result in mode collapse or severe performance degradation.
In additional testing with naive users we found that better LLMs boost performance. In brief, we sampled 30 naive user prompts at random out of 362 submitted via an App we developed, and found an 18% error rate with GPT3.5 yet with GPT4
38 only 10%). See the Appendix for examples.
Discussion
AMadeusGPT is a novel system for interactive behavior analysis. By providing a user-friendly interface and leveraging the power of language models, AmadeusGPT enables researchers to more efficiently and effectively study animal behavior. We believe this system offers a significant contribu- tion to the field of behavioral analysis. The natural language interface of AmadeusGPT empowers non-experts conduct advanced behavioral analysis with cutting-edge computer vision (such as with SAM 111 and Animal [17). but its reliance on L.LMs raises potential ethical concerns such as bias amplification. The use of a custom-designed API module in AmadeusGPT helps limit potential biases in outputs, but future work should consider how integration modules can introduce biases (39] It also allows for domain-specifer human-Al interactions 440
Our proposed natural language interface with the augmented-GPT3.5 shows promise, but there are limitations that need to be addressed in future work. These include enhancing robustness by reducing bias in prompt writing, enabling more goal-driven code reasoning to make AmadeusGPT more collaborative, and improving generalization by extending the available APIs and preventing ChatGPT from writing incorrect code. We also only support the English language, but multi-lingual support believe AmadeusGPT promises to open new avenues
with be explored in the future. Collectively, for both leveraging the utility of, and developing on, cutting-edge computer vision and machine learning tools by using intuitive language.
also worth noting that AmadeusGPT leverages ChatGPT API calls, which means we do not have access to the weights of the underlying LLMs thus we cannot yet fine-tune or augment the LLMs. Therefore, our dual dual-memory leverages the process memory and disks to augment the language model, in contrast to works that augment LLMs by assuming the access to the LLMs 42 43). Similarly, without fine-tuning the underlying LLMs, we constrain the behaviors of the LLMs only via in-context learning for both the code generator and rephraser. While recent models such as GPT4 388. CLAUDE [44] continue to make context window bigger and in-context learning more powerful, deploying them can be slower and more expensive and a strategy of combining smaller model and a bigger model is worth exploring [45. In the future, it would be interesting to compare our in- process-memory with in-context memory in terms of reliability and cost. It is also interesting to compare fine-tuned LLMs vs. LLMs that are constrained via in-context learning for future works that use natural language interface for behavior analysis.
Of course, AmadeusGPT is not limited to using GPT3.5 or GPT-4, and new models such as CLAUDE 44] others could be used. However, the larger the model the slower the response time and the higher the computational cost. Thus, while these excellent works have started to over- come token limits, there still limits such that our computationally efficient dual-memory system will be of broad interest to those developing both domain-specific solutions (such as AmadeusGPT) and generalist models (the base GPT models). Collectively, we believe AmadeusGPT promises to open new avenues for both leveraging the utility of, and developing on, cutting-edge computer vision and machine learning tools with minimal no coding namely, by only using intuitive language.
10

---

6 Acknowledgements
This work was supported by The Vallee Foundation Scholar award MWM. We thank NeuroX at EPFL for travel support. We thank members of the M. M.t.hs Lab and A. A.Mathis Group at EPFL for feedback, A. Iqbal for computing animal-object interactions used in Appendix Figure 6 St. Schneider and Kinematik Al for assistance and hosting our demo, and the alpha testers. COI: MWM a co-founder and equity holder in Kinematik AL. Author Contributions: Conceptualization: SY,
MWM; Methodology & Software: SY, JL. MWM, MZ, AM; Writing: MWM, SY; -riting-Editing: JI. AM; Visualization: MWM, SY, MZ; Funding acquisition: MWM.
References
[1] David J Anderson and Pietro Perona. Toward a science of computational ethology. Neuron, 84(1):18-31, 2014.
[2] Mackenzie Weygandt Mathis and Alexander Mathis. Deep learning tools for the measurement animal behavior in neuroscience. Current opinion in neurobiology, 10:1-11, 2020.
[] Mayank Kabra, Alice Robie, Marta Rivera-Alba, Steve Branson, and Kristin Branson. Jaaba: interactive machine learning for automatic annotation of animal behavior. Nature Methods, 10:64-67, 2013.
Jennifer Sun, Andrew Ulmer, Dipam Chakraborty, Brian Geuther, Edward Hayes, Heng Jia, Vivek Kumar, Zachary Partridge, Alice Robie, Catherine Schretter, et al. The mabe22 henchmarks for representation learning of multi-agent behavior. arXiv preprint arXiv: 10555, 2022.
[5] Konrad Lorenz. The foundations of ethology. Springer verlag, 1981.
[6] Patrick C. Friman. Cooper, heron, and heward's applied behavior analysis (2nd ed.): Checkered flag for students and professors, yellow flag for the field. Journal of Applied Behavior Analysis, 43:161-174, 2010.
[7] Alexander Mathis, Pranav Mamidanna, Kevin M Cury, Taiga Abe, Venkatesh N Murthy, Mackenzie Wey- gandt Mathis, and Matthias Bethge. Deeplaheut: markerless pose estimation of user-defined body parts with deep learning. Nature neuroscience, 118111221-1288, 2018
[8] Fabrice de Chaumont, Elodie Ey, Nicolas Torquet, Thibault Lagache, Stéphane Dallongeville, Albane Imbert, Thierry Legou, Anne-Marie Le Sourd, Philippe Faure, Thomas Bourgeron, and -ean-Chrittophe Olivo-Marin. Live mouse tracker: real-time behavioral analysis of groups of mice. bioRxiv, 2018.
Talmo D Pereira, Diego Aldarondo, Lindsay Willmore, Mikhail Kislin, Samuel S-H Wang. Mala Murthy, and Joshua W Shaevitz. Fast animal pose estimation using deep neural networks. Nature methods, 16(1):117-225, 2019.
[10] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In Proceedings of the IEFF international conference on computer vision, pages 2961-2969, 2017.
[11] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson. Tete Xiao, Spencer Whitchead, Alexander C. Berg, Wan-Yee Yen Lo, Piotr Dollár, and Ross Girshick. Segment anything. arXiv:2304.0m 02643, 2023.
[12] James Bohnslav, Nivanthika K Wimalasena, Kelsey Clausing, Yu Y Dai, David A Yarmolinsky, Tomás Cruz, Adam D Kashlan, M Eugenia Chiappe, Lauren L Orefice, Clifford J Woolf, et al. Deepethogram, a machine learning pipeline for supervised behavior classification from raw pixels. Elife, 10:e63377, 2021.
[13] Oliver Sturman, Lukas von Ziegler, Christa Schläppi, Furkan Akyol, Mattia Privitera, Daria Slominski. Christina Grimm, Laetitia Thieren, Valerio Zerbi, Benjamin Grewe, et al. Deep saaring-basee behavioral analysis reaches human accuracy and is capable of outperforming commercial solutions. Neuropxychophur- macology, 45(1 11:1944-11152 2020.
[14] Simon RO Nilsson, Nastacia I. Goodwin, Jia Jie Choong, Sophia Hwang, Hayden R Wright, 7ane C Norville, Xiaoyu Tong, Dayu Lin, Brandon S Bentzley, Neir Fshel, et al. Simple hehavioral analysis (simba)-an open source toolkit for computer classification complex social behaviors in experimental animals. BioRxiv, pages 2020-04, 2020.
[15] Markus Marks, Qiuhan Jin, Oliver Sturman, Lukas M. von Ziegler, Sepp Kollmorgen, Wolfger von der Behrens, Valerio Mante, Johannes Bohacek, and Mehmet Fatih Yanik. Deep-learning based identifica- tion, tracking, pose estimation, and behavior classification of interacting primates and mice in complex environments. Nature machine intelligence, 4:331 340, 2020.

---

[16] Cristina Segalin, Jalani Williams. Tomomi Karigo, May Hui, Moriel Zelikowsky, Jennifer J Sun, Pietro Perona, David Anderson, and Ann Kennedy. The mouse action recognition system (mars) software pipeline for automated analysis of social behaviors mice. Elife. 10, 2021.
[17] Shaokai Ye, Anastasiia Filippova, Jessy Lauer, Maxime Vidal, Steffen Schneider, Tian Qiu, Alexander Mathis, and Mackenzie Weygandt Mathis. Superanimal pretrained pose estimation models for hehavioral analysis. arXiv preprint arXiv:2203.074300 2023.
[18] Zachary T. Pennington, Zhe Dong. Yu Feng, Lauren M. Vetere, Lucia Page-Harley, Tristan Shuman, and Denise J. Cai. eztrack: An open-source video analysis pipeline for the investigation of animal behavior. Scientific Reports, 9, 2019.
[19] Mehdi Azabou, Michael Mendelson, Maks Sorokin. Shantanu Thakoor, Nauman Abad, Carolina Urzay, and Eva L Dyer. Learning behavior representations through multi- multi-timescale bootstrapping. arXiv preprint arXiv:2206. 200:00046.0701, 2022.
[20] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirec- tional transformers for language understanding. arXiv preprint arXiv:1810.0480 2018.
[21] Colin Raffel, Noam Shazcer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. The Journal of Machine Learning Research, 21 21()):4855-551, 2020.
[22] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Ncelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 13:1877-1901, 2020.
[23] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35:27730- 27744, 2022.
[24] Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Micch, Iain Barr, Yana Hasson, Karel Lene, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances in Neural Information Processing Systems, 35:23716-2222 16-2333,, 2022.
[25] Yaru Hao, Haoyu Song, Li Dong, Shaohan Huang, Zewen Chi, Wenhui Wang, Shuming Ma, and Furu Wei. Language models are general-purpose interfaces. arXiv preprint arXiv:2206.063336, 2022.
[26] Tanmay Gupta and Aniruddha Kembhavi. Visual programming: Compositional visual reasoning without training, arXiv preprint arXiv:2211.11555 2022.
[27] Dídac Surís, Sachit Menon, and Carl Vondrick. Vipergpt: Visual inference via python execution for reasoning. arXiv preprint 2333:23303 2023.
[28] Yongliang Shen, Kaitao Song, Xu Tan, Dongsheng Li, Weiming Lu, and Yueting Zhuang. Hugginggpt: Solving tasks with chatgpt and its friends in huggingface. arXiv preprint (tt.::3333.75880 2023.
[29] Joon Sung Park, Joseph C O' Brien, Carrie Cai, Meredith Ringel Morris. Percy Liang, and Michael S Bernstein. Generative agents: Interactive simulacra of human behavior. arXiv preprint arXiv:2304.00 03442, 2023.
[30] Jennifer J Sun, Ann Kennedy, Eric Zhan, David J Anderson, Yisong Yue, and Pietro Perona. Task program- ming: Learning data efficient behavior representations. Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 2876-2885, 2021.
[31] Kevin Luxem, Petra Mocellin, Falko Fuhrmann, Johannes Kürsch, Stefan Remy, and Pavol Bauer. Identify- ing behavioral structure from deep variational embeddings of animal motion. Communications Biology, 5, 2020.
[32] Eleanor Batty, Matthew R Whiteway, Shreya Saxena, Dan Biderman, Taiga Abe. Simon Musall, Winthrop F. Gillis, Jeffrey E. Markowitz, Anne K. Churchland, John P. Cunningham. Sandeep Robert Datta, Scott W. Linderman, and Liam Paninski. Behavenet: nonlinear embedding and bayesian neural decoding of behavioral vidcos. In Neural Information Processing Systems, 2019.
[33] Fabrice De Chaumont, Elodie Ey, Nicolas Torquet, Thibault Lagache, Stéphane Dallongeville, Albane Imbert, Thierry Legou, Anne-Marie Le Sourd, Philippe Faure, Thomas Bourgeron, et al. Real- time-ttmeeaae analysis of the behaviour of groups of mice via a seppt--eening camera and machine learning. Nature biomedical engineering, 3(11):930-9)) 942, 2019.
12

---

[34] Leland McInnes and John Healy. Umap: Uniform manifold approximation and projection for dimension reduction. ArXiv, abs/1802.03426, 2018.
[35] Steffen Schneider, Jin Hwa Lee, and Mackenzie Weygandt Mathis, Learnable latent embeddings for joint behavioural and neural analysis. Nature, May 2023.
[36] Alexander Mathis, Thomas Biasi, Steffen Schneider. Mert Yuksekgonul, Byron Rogers, Matthias Bethge, and Mackenzie W Mathis. Pretraining boosts out-of-domain robustness for pose estimation. Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 1859 1169-168, 2021.
[37] Mehdi Azabou, Michael Mendelson, Maks Sorokin, Shantanu Thakoor, Nauman Ahad, Carolina Urzay, and L. Dyer. Learning behavior representations through multi-timescale bootstrapping. ArXiv, abs/2206.07041, 2022.
[38] OpenAL. Gpt-4 technical report, 2023.
[39] Hanzhou Li, John T. Moon, Saptarshi Purkayastha, Leo Anthony Celi, Hari Trivedi, and Judy Wawira Gichoya, Ethics large language models medicine and medical research. Lancet. Digital health, 2023.
[40] Tongshuang Sherry Wu, Michael Terry, and Carrie Cai. chains: Transparent and controllable human-at interaction by chaining large language model prompts. Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems, 2021.
[41] April Yi Wang, Dakuo Wang, Jaimie Drozdal, Michael Muller, Soya Park, Justin D Weisz, Xuye Liu, Lingfei Wu, and Casey Dugan. Documentation matters: Human- ai system to assist data science code documentation in computational notebooks. ACM Transactions Computer Computer-Human Interaction. 29(2):1-33, 2022.
[42] Zexuan Zhong, Tao Lei, and Danqi Chen. Training language models with memory augmentation. arXiv preprint arXiv:2205. 12677 2022.
[43] Weizhi Wang, Li Dong, Hao Cheng, Xiaodong Liu, Xifeng Yan, Jianfeng Gao, and Furu Wei. Augmenting language models with long-term memory. arXiv preprint arXiv:2306.07174. 2023.
[44] Anthropic. Introducing claude, 2023.
[45] Tianle Cai, Xuezhi Wang, Tengyu Ma. Xinyun Chen, and Denny Zhou. Large language models as tool makers. arXiv preprint 2711 2023.
[46] Alexander I. Hsu and Eric A. Ytri.. B-soid, an open-source unsupervised algorithm for identification and fast prediction of behaviors. Nature Communications, 12, 2019.
[47] Mackenzie Mathis, Jessy Lauer, Tanmay Nath, Kai Sandbink, Michael Beauzile, Sébastien Hausmann, Steffen Schneider, and Alexander Mathis. DLC2Kinematie:: post-deeplabeut module for kinematic analysis. Zenodo, February 2020.
13

---

Appendix
System Prompts
Overall, AmadeusGPT consists of three GPT3.5 GPT4 instances with different system prompts, the pretrained computer vision models (e.g., SAM and SuperAnimal) and our API interface and implementation. For the pretrained computer vision models (SAM. SuperAnimal), training details provided in the original works, and we did not fine-tune them. For GPT3.5, the training details were released by OpenAl the time of this publication, and we also did not fine-tune any GPT model.
Below are the core API system prompts used to create AmadeusGPT; related to Main Section 3.2.
system_prompt Y"uuur name is AmadeusGPT and you are helping users by first + understanding my API docs:
Saaterrfaee_srrnnnccls www.wieeaaoowollleeeoeeeeeeeeeee _odulle__str\nn" system_promps "Now that you understand my API docs, user asks things that are achievable by using my APIs, write and only write function named task_program() with return values. If not achieveable by API docs, explain why. Think step by step when generating code. An
system_promps "Rule 1: (1) Do not access attriibutes objects, unless attributes are + written in the Arributes part of function docs. (2) Do not call any functions or + object methods unless they are written in Methods part in the func docs. An"
system_prompt += "Rule 2: Pay attention to whether the user means for single animal multiple animals. An"
system_prompt "Rule 3: Never call instance functions of matplotlib objects such as ax + and An"
system_prompt += "Rule 4: Do not write any comments in the function. An"
system_prompt += "Rule 5: Must pay attention the typing for the parameters and returns + of functions when writing code.
system_prompt "Rule 6: When generating events, pay attention to whether is simultaneous or sequential from the user prompt. For example, events descirbing multiple bodyparts for one animal must be simultaneous events. Animal moving from one place the other must be sequential events. An"
Below are the Rephraser system prompts used to counter domain shift in the user prompt style; related Main Section 3.2.
system_promps Youu job is to help translate queries into more queries that follow my rules and examples. If you do not know how to translate, just repeat what you are + asked and say sorry. An"
system_prompt "Rule 1: Make sure you only answer with short converted version. explanation or other formatting. An" An" system_prompt += "Rule 2: Make sure you do not change bodypart names and do not change name of the behaviors. Name of the behaviors might be enclosed by <||>.An An" system_prompt "Rule 3: not change temporal conjunctions when rephrasing. For + example, 'and then' cannot be replaced by 'and'.Ani An" system_prompp +=
"Rule 4: Do not change the semantic meaning of the query. Do not remove important details.Ant An"
system_prompt += "Rule 5: Do not rephrase adjective words such as shortest event or longest distance, An"
system_prompt += "Rule 6: when spatial relation is used as adjcetive such as entering
+ from right, add a adjcetive word simultanesouly such as entering from right
+ tiiuusanneuslyy.W An"
system_prompt += "Rule 7: Do not rephrase terms with special meaning such as cebra, umap, deeplabcut.
system_prompt += "Rule 8: If you do not know how to rephrase it, just copy the instruction and write back.
system_prompt += "Rule 9: Keep the adjectives for words like head angle or closest
distance or relative speed. They should not be rephrased such that head angle is confused angle relative_speed confused as speed\n distance confused closest distance\n"
system_prompt "Rule 10: Rephrase specialized animal names such as mouse, mice animal, animals, \n"
14

---

system_prompt += f"Example 1: How much time the animal spends in ROI? -> Give me the amount of time animal spends in ROI and the events where the animal overlaps ROI. An" system_prompt f"Example 2: How much time
animal moves faster than -> Give me + events where the animal moves faster than 3. An"
prstem_prompt += f"Example 3: What is the duration of time animal is outside ROI? >> Get events where the animal is outside ROI. An"
system_prompt += f"Example 4: What is distance travelled in ROI Get distance + travelled of the animal overlaps An"
system_promps += f"Example 5: Get object ethogram +> Plot ethogram for animal overlapping all objects. An"
system_prompt += f"Example 6: Give trajectories of the animal >> Plot trajectories of the animal. An"
system_prompt f"Example 7: The animal enters object and then enters object 2 -> The animal enters object and then enters object. An"
system_prompt += E"Exmmlle 8: Define 

social behavior where distance between animals is less than 260 and distance larger than 50 and angle between - animals
less than 15. Get masks for watching. -> Define <|watching[> as a social behavior where distance between animals is less than 260 and distance larger than 50 and angle between animals \
less than 15. Get events where watching happens. An"
system_prompt f"Example 10: plot object ethogram -> plat object ethogram. An"
system_prompt += f"Example 11: define 
 as a behavior where the animal's bodypart
tail_base moves faster than while animal's 's bodypart nose moves faster than 4, when behavior where the animal's tail_base moves faster than while animal's nose moves faster than 4. Give events where animal does bla. An"
does the animal do bla? -> define 

system_promp += f"Example 12: Give me frames where the animal is on object 17 -> Give me events where the animal is object 17. An" system_prompe
f"Example 13: Give me cebra embedding -> Give 3d cebra embedding and plot embedding.. An"
system_prompt += f"Example 14: plot trajectory when the animal is on object 1 -> plot trajectory for events where the animal overlaps object 1.An"
system_prompt += f"Example 15: Help me define the behavior freeze Help define the behavior freeze. An"
system_prompt += f"Example 16: Where is the animal in the first half of the video? >> Where is the animal in the first half of the frames?\n" system_prompt += f"Example 17: When is the animal above object 3? -> Give events where animal is above object An" system_prompt f"Example
When the closest distance among animals less than 3? - Give me events where the closest_distanc among animals is less than 3.An" system_prompt += f"Example 19: How much time does the mouse spend roi0? -> Give me amount of time animal spends roie and events where the animal overlap with ROIe
system_prompt f"Before rephrasing, keep in mind all those rules need to followed."
Below are the self-correction system prompts used to fix common runtime errors; related to Main Section 3.2.
system_prompe prompt
Your job is to correct code that raised errors. You will be given the user query the code written for, the code, the error message and the - diagnosis for why error happens
system_prompt*= "There are empirical rules that can help you debug: An" system_prompt f"Rule 1: If the code used bodypart that does not exist, replace it with one that is semantically close from suppor ted bodyparts. An"
Dual memory mechanism, reload and relaunch example
Related to Main Section 3.3, we discuss how short-term memory can be supplemented by long-term memory to restore chat history. AmadeusGPT stores information including chat histories both short-term and long-term memory well as segmentation masks and relation look-up tables. As they are all implemented as Python objects, it easy to serialize them into disk and deserialize them from the disk. More specifically, support two special prompts called "save" and "load" respectively.
15

---

Those two special prompts can save the state of AmadeusGPT before exiting and load the state of AmadeusGPT after relaunching.
Query: "save"
AmadeusGPT Output:
Saving state into state pickle
Query: "load"
AmadeusGPT Output:
Loaded state.pick..
API core behavioral modules
This is part of the core API docs that are always sent AmadeusGPT when queried; related Main Section 3.4.
class AndmeelBehaaiiornnallyss::
Methods
ret_object_names List[str] get names of all objects
def s
state_type,
comparison, bodyparts ['all'], )
Parameters
state_types str
Must be one of 'speed', 'acceleration'
comparison: str Must comparison operator followed by number like <50, Examples
>>> # when is the animal moving faster than 3 pixels across frames?
def task_program():
>>>
behavior_analyse animal_faster (atmma___aatee__a_____eeee than_3_eventss
>>>
behavior_analysis. anw.aaalssssaaiimallls....s.. animals_smaalatteeees satllllssttt__eeeeeeeeeeee eeeettseeeeeeeeeeeeeeeee speed', '>3')
>>>
return mnimal_aaster_thn__3.1 3_vents
return animals_state aninaas_staae__vvetttttttttttttd comparison)
superanima_vvdee..iifrrreee(le] None: *au
Examples
extract keypoints (aka pose) from the video file
>>> def task_program(): behavior _analysis
>>>
jahayvioranalysis- = AndimalBehaaiirrknallsss
behavior_analysis. httba//oo.nalllssssooocaaaaallle superaninal_video_infeeeeece
>>>
return superanimaLviddo_iffreeeee
def animals_object.c cnimmlss.ojecct_eeentssseel
16

---

object_name: str, relation_query, comparison None, negate False, bodyparts: List[str] = ['all'], min_window 0, EventDict:
object_name : str. Name of the object
relation_query str. Must be one of 'to_left', 'to_right', 'to_below',
to to_above", 'overlap', 'distance', 'angle', 'orientation"
comparison str, Must be comparison operator followed by number like <50, 4 optional
bodyparts: List[str], optional
sin_window: min length of the event to include
max_window: length of the event to include negate: bool, default false whether to negate the spatial events. For example, if negate is set True, 4 inside roi would be outside roi
Returns:
EventDict Examples
>>>
find where the animal is the left of object def task_program():
>>>
behavior_ane jaaalyisranalyiis knimalBehaaviorknallysss
_eeenttsseeeeesseeooee behavior_ http://yis..aanll.oo...eeeeeeees to tt_llef'', bodyparts = ['all'])
>>>
>>> return seft_too_object__eeettt
>>> how much time the animal spends closed arm?
>>> def task_program():
>>> behavior_analyae
Artmalleehavoorknalyysss
>>> in_closed__rm__eeent behavior_analysee hntnssaaaaaaasseeeeeeeeeeeeeee closedd
4 arm
'overlap', bodyparts ['all'], negate False)
>>> return m_.close_arn__eent. _evetts
>>> get events where animals's 's distance to animal0 is less than 30
>>> def task_program():
behavior_analysis Animaaleehaviooknaalssss
>>> distance_social_eve
4 behavior_analyy www.aaass.aniaalll...a.a.a.aaaae
'disntace', comparison = '<30', wwwiaannnmbaaaaa.. animmlss_ojject_eeeetss ('anima00', ('aiiiaima
bodyparts- ['nose']] 'I)
>>> return distance_social_eve
return anfialls.bheeet..wnnsscheee.... relation.query)
def trft_trajectory((eeff bodyparts: List[str], events: Union[Dict[str, dict],dict]
None, **wargg)):
Parameters
bodyparts List[str]
events: Union[Dict[str, dict],dict], optional
The type must be either dict or dictionary of dict
kwargs parameters for plt.scatter
Returns:
Tuple[plt.Figge Figure, Axes] tuple of figure and axes
Examples
>>> plot trajectory of the animal.
17

---

>>> def task_program():
>>> behavior_amallsis AdimalBeehaviorrnnallssss
>22
fig, ax wethe//w...aaayllssppoorrrreeerr
>22 return fig, ax
>>> # plot trajectory of the nose of the animal with the event that animal
overlaps with object
>>> def task_program():
>>>
behavior_analysis AnimallBehaviorknnllysssSS
>>> overlap_6_events = behavior_analysis. www.ayywws.iiiaal.....aaatetssss
overlap', bodyparts = ['all'])
>>> fig, behavior_analllaeeeetessesssssse
wttpalyyiiisltrrrrjjetooo"""""" events
vverlap___evene
>>> return fig, ax
return plotttrraecctrrytbdyyrrrrts,,,, events)
# do not call saamaalssscialleeeeeeeee if there is multiple animals social events
def mnimals_social_evere (self,
relation_query_llsst
comparison_list,
ortmal__satt_rrllatiooouuur___. [],
ortimal_statte_coorriooo...Ll [], bodyparts = ['all'], otheranimal_.odypartt ['all'], min_window 11,
pixels_per._ 8,
smooth_window.sih 5)
The function is specifically for capturing multiple animals social events Parameters
relation_query_Inn List[str]
list of relation query for animals-animals relationship. Must be chosen from
['to_let',, 'to_right', 'to_below,, "to_above,, 'overlap', 'distance', 'orientation', "closest_distam" 'angle'].
comparison_list: List[str]
list of comparison operator such as '==', '<' '<=', '>='
List[str]
list of relation query for state of the animal that interacts with other animals. Must be chosen from ['speed', 'acceleration' 'confidence'].
List[str]:
list of comparison operator such as '==', '<' '>' 'e', '>!
bodyparts: List[str]
list of bodyparts for the main animal sttheranimll-bddpprre list[str]
list bodyparts for the other animal min_window: int, optional Only include events that are longer than min_window
pixels_per___e int, optional how many pixels for centimer smooth_win serec_window_ste _size: int, optional smooth window size for smoothing the events. Examples
>>> Define 
 as a social behavior where distance between animals are
4 less than 100, one animal is in front of the other animal and the chasing animal has speed faster than 3. Get chases.
>>> def task_program():
chase_events
http://rw.mmlllsss.aaansssse...e dist distance'',
'orientation'].
>>>
[f's 100', (O= [ttpreentttiooooooo]]_..
>>>
aninal_staate_rlaation.gcre = ('speed')
18

---

>5>
>57
bodyparts ['all'],
otheranimal_bone bodypartss ['all'])
222
return chase_events
return oaiiala..ocaaaeeeeessrrrtttto..e comparison_list)
class Orientatiin((ntEEnn)
Attributes
FRONT
BACK LEFT RIGHT НаH
class Event:
@classmethod
def add_ tedlssiiiltteeeoseeeeettcccll *events_list list: List[dict])~> dict:
Parameters
events_list, list: List[dict]
Returns
EventDict
Examples
get events for the animal's nose the roi and the animal's tail base not in
9 roi
>>> def task_progran():
>>>
Dehavior_analysee
AnimalBehaaviorknallsss)
>>>
nose_in_rri_events = behavior httpy/siis.aimmlsoooeeeeeeeeeeee
'overlap', bodyparts ['nose'], negate False)
>>>
tail_base_ntt_inrroii reveteeeesss--
'overlap', bodyparts ['tail base") ]]
negate True)
return Event.add_st sid_:dddldsmateeeeude events (nose ineete_eees_tsss
tail _et_baee__nt_in_oooeeeettss *аu
return add_simul tnt htw.llotetooeeeetssseetttssssss @classmethod
def add__sequentall_evee (clenss(
*events_list, List[dict], continuous False) dict:
Keywords such "then", "later" might suggest it is sequential event Parameters
eisnts_list: List[dict]
Returns
EventDict
Examples
>>>
Get events where the animal enters object from the bottom.
>>> def task_program():
>>>
behavior_analysiis AnimalBehaviornnallyssss
>>>
bottom_to haamm_tt__biiii
behavior_analysiz. annaalsss.animalss.......a.aaaas secrlls_ooeeeeeeeeeeeeee
tto_below', bodyparts ['all'])
>>>
enter_object___eve seeeectt___eeets-e
behavior_analysis. senealysssseetrrooffettt... 6', bodyparts
= ['all'])
>>>
enter wwarr_oeeeeefffooommmmm
wwt.frromooottoeeeseeeooo events
Event.
enter_object___event
19

---

>>>
return enter_ [ooject___from_otttoe
>>>
Find events where animal leaves object 6
>2>
def task_program():
227
behavior_analysii
AnimalBehhaviornnallyssss
>>>
leave ottp: leave_object___ewe j_eeetts = behavior_analysiin
6', bodyparts
['all'])
>>>
return beae_object___evee
>>>
Find events where the animal leaves object and then enters object 3
>>>
def task_program():
>>>
wehavior_analyyee AnimalBehaaiorkknallysss
>>>
@evv__objet___evene = behavior_anallsii..eer Levee
bodyparts
4 ['all'])
>>>
enter_object_ 3 httpriorr..allysss.etter...poppp
bodyparts
+ (' ('all'])
>>>
leave_object__ 6_and_enter, conttrrobject___eeee
serde_______eeee__bbrrer enter_object_3eevnnt find events where the animal moves from left of object to right of object
>>>
return
>>>
6.
>>> def task_program():
behavior_analyse AnimalBehaviorkkaalyysss
>>> Left_to_.bjeet___eeeert
Dehavior_analysis.oo Seeeettt___eeeessttteteeeee
jerject_vveet(("'' 'to_left', bodyparts = ['all'])
>>>
right_to_object___eee
'to_right', bodyparts ['all'])
- Event, 1eft_to_rightteeentts
>>>
right_too.bject___.vens.
>>>
return left_to.right_evenn
add_ hod._eeuuentaleeeeetteeeeeessss
class EventDict(dict):
def sort(self):
Sort the eventdict Return
EventDict
return sort()
Integration of behavioral modules (non-exhaustive)
As described in the Main Section 3.5 and Main Figure 2, AmadeusGPT can use integration modules for task-specinf behavior analysis, and implements dynamic loading by converting API documents into text embedding vectors and caching them in RAM or disk, allowing users to retrieve the most relevant modules by cosine similarity using their prompt modules to integrate functionality. Or, they can write simple modules and contribute them to the AmadeusGPT codebase for other users leverage well. Below provide examples of integration with the newly introduced dimensionality reduction tool called cebra (35), and UMAP (3.. short, to integrate into the codebase the user should write minimal working example in the style of the core API docs. Then, after dynamic loading, queries relating to UMAP or cebra (as shown here) would generate the following code (below) and related output plots (Figure
query. Namely, the user could use these
CEBRA
CEBRA is nonlinear dimensionality reduction tool for joint modeling of neural data and behavioral labels 35 which might be of interest to users of AmadeusGPT. In Appendix Figure 6 we show 3D CEBRA embedding of mouse poses (aligned) colored by when overlaps with different objects. Note that the CEBRA embedding in Figure is 3D n-sphere that is equivalent to 2D Euclidean
20

---

creadnill elevated platforn costos zoll halt tube platforn barrel Food
Annotated MausHaue masks
UMAP embedding plot Cebra 3D enbedding plot
Figure 6: Integration examples. Left to right: MausHaus masks with human annotated names, examples of UMAP and cebra integration modules, plots colored as in annotations. See below for user prompts to produce the plots.
space [35] Below the user query is shown below to produce images in Figure 6 along with the related API docs.
Query: "get 3D embeddings using cebra and plot the embedding."
def compute_embedding_wiii cebrr
(eceeerrradmmmnnniioooeooo 3, max_iterations=100)
Examples
>>> # create dimensional embedding with cebra
>>> def task_program():
behavior_analysi. siineleehaviornnallysss
>>>
embedding = behavior _atiiioranalliis.cmmm
3)
>>>
return embedding
return compute_embeddeeee ewwedding whth (ttp://wwttttthtttnnteaeaee sebra(n_dimensinn didemnsion,, mav.llerraioonsmmeteeettttooss
UMAP
is a popular nonlinear dimensionality reduction tool that is also core to several behavioral analysis tools such B-SOID (46] In Appendix Figure 6 we show 2D UMAP embedding of poses (aligned) when it overlaps with different objects.
UMAP 34
Query: "Get 2D embeddings using umap and plot the embedding"
def compute, prmpute_embedding.om wi th_ umap (n_ dimensino 3):
Examples
>>> # create dimensional embedding umap
def fask_program(featuree
>>>
behavior_analysis Anima Behaahhaooorkklllyssss
>>>
embedding = eeehavior_naalyeee
wwwpllysss.copteeemmm.... compute_embedding wi www.wwthhhhapp.. wh__map(n_dimensiio = 3)
>>>
return embedding
return compute_embeddiingg.u wtw.:ttthwmmmmmmmeeeeeiiiiieeeee n_dimension)
Global spatio-temporal behavioral analyses
Many core behavioral analysis tools rely on computing metrics based on kinematic or movement analysis. For example, time spent locomoting, time in ROI, computing the distance traveled within specific area is highly popular in behavioral neuroscience research. Thus, inspired by DLC2Kinematics 47 we show API docs to integrate such common integration modules:
def calculate_disseeeecc aalculate_distanee.traalleeeed wen
events):
Parameters
21

---

events: EventDict
Return
int
Examples
>>> # what is the distance travelled in closed arm?
def task_program():
>>>
behavior_analysio AdimalBehhavoorAnallysss
roi_events behavior_anaiee bodyparts (]]
>>>
closed arm 'overlap',
distance_in roi
>>>
return iistanceeii_rot
return aal.uulate.distnnn..ttvvlelleedd
def plot.ooject_ethogramseelll object_names):
Parameters
object_names: List[str] list of object names
Return:
MupleppttFiiure, Ale.A tuple of figure axes
Examples
>>> # plot ethogram for animal overlapping all objects def task_program():
behavior_anallsis- AnimaaBBehaiiooAnallysss
>>>
object_names behavior_anallsiie set.aaalllsss.getoo.jeeeeeeeeeee
>>>
fig, = behavior_analysee http://sss.pptttooooeeeeettttrrr
>>>
return fig,
return (aapjjec__ethorrmmbjjeeeeeee
Additional Experiments: behavioral analysis examples
Here, as related to Main Section 4, we provide more example queries and responses from AmadeusGPT support Main Figure 5a (EPM), to Main Figure 5b (MausHaus), and to Main Figure 5c (MABe) and results in Table
Extended Results on EPM with AmadeusGPT
We show additional user queries and generated code from AmadeusGPT related to the EPM data presented in the Main Figure 5a.
Extended Results on MausHaus with AmadeusGPT
We show the user queries and generated code from AmadeusGPT that produce the images shown in the Main Figure 5b.
Query: "Give me the trajectories of the animal entering object 18 from right"
AmadeusGPT Code Output:
def task_program():
dechavor_anallyee = InimalBehaviiorAaaalssss)
www.allli...iilll.l.laao.o...... ("rriiccteeeett("ii 'to_right', behavior_analysit bodyparts ['all'])
enter_object_10 serr_oobect____eeettts 18_events
behavior_analysis
wtt.oraaaallssseeteer.... mnnalysis.ener..bjeee.. www.eeeeeeeeeeee1eeee bodyparts ['all'])
enter_object_ ttr.bbbect_18_froorrriieftt
Event.
to_obbecc__reeeeeeeeeeeeeeeee__e (ecjjett____eeee 18_entts, enter_object_1 Bittr_ooject__8_eeeetts ____evenss
22

---

Query:
Dafinn slbaad éipa|> as a behavior whare the animal's a hodyparta mouse centar and at the sane time neek in the roi and its head, niopoint outside roi, Get head dipa."
Amadous Code Outputs:
progcem()
behavior analyais AutseBBeaarorreeayyya
mouse_ nouse_senter, conter_necke in rov_ennssss Levents behevior, menvior_anallyeee yow..
abt_ettttttteeeeeeeeeeee evente
noi 'ovarlap' bodyparta- mnusse, center' "neck" 1, bagata-Falas) besd_nidpo www.iiddoonn_nie peppoitt_et...o int for___n_oi__aeee _eeeeesssss behavior annalyiie animall objjett objest_eeenter evente(
"Oo"" "overlap", bodyparta- "ead niépoint" ), aegate-Prua)
neac_dipe events Event, add peiieimittaen taneeue evente pt nidpaint, ser_rr__evvtte events)
bead_dips_ dipe_events
Amadeus Chat Outputs (Execution):
for query
video length 15076 frames).
freme nunber
Query:
"Mhen is the animal moving faster than pixele per frame?"
K Amadeus Code Outputs:
def taek progrem(); bebavios analyaia AndralBabarrorraaayyaa
faster_ then_ than___evente behavior_ www.yein...........m. wttplll___p_eeeeeeeeeeeeee "apaed" "
return anins) faster_ then_ B_ eeevent
Amadeus Chat Outputs (Execution):
Results for query 371 frames (total video length is 15078 frames).
5000
8000
20000
12000
14030
Query:
DDfine <| freesing| as behavior where the animal is slover than 0,3, When is the house freezing?"
Amadeus Code Outputs:
def progrem())
behavio tor analyaia Aninal wwwhhvioornalyyan.. ireezing eeenee behavior, anslyeis. seast_oveett "speed", "40.3") return froezing Lcwezing_eewncee events
Amadeus Chat Outputs (Execution):
Results for query 2411 frames (total video length is 15076 frames).
2000
4000
5000
8000
10000
1200n
1400n
frome
bumber
Figure 7: Extended Results on classical behavioral task (EPM) with AmadeusGPT. (a) Using the EPM dataset [13,, we show additional queries and responses from AmadeusGPT: The raster plot an ethogram where there is tick for every related event in time, i.e., the mouse is in the closed arm or in the open arm.
fig, ax behavior www.iior_aaalsiss.ommm wwwppplotooppeeeeeep "rao_trrjjccooo((""l111 events return fig,
Query: "plot trajectories of the animal for the shortest event where it enters object 18 and then its neck enters object 17"
AmadeusGPTT Code Output:
def task_progrant behavior_analysee = AnimalBehaaviorknallysss
23

---

enter_object__8_eventi behavior_analysie enter_o bodyparts-C'all
peck_enter_object__7.. 17_eentt behavior sawawiior_maaysii...m enwelssseetrrobb_eerrre objectr")))
Modypartss='neck"" neck'
Event
tecc__ener_objeet_l1eeeee pooo..triscccoorooooopprrsslall
fig, ax behavior hntaliioraaalysis..ooo plot
events=enter_oojer hataeetet___nnnneeceeeeeeeeeeee return fig, ax
MABe Mouse Triplet Benchmark, extended details
In the MABe 2022 Mouse Triplet Benchmark, participants given dataset of tracking data of triplets of socially interacting mice and asked to submit a frame-by by-ee----aeme representation of the dataset. Here the following user prompts and AmadeusGPT code outputs of the sub-tasks considered and the generated code by AmadeusGPT.
Query: "Define <lchasesl> as a social behavior where the closest distance between animals is less than 40 pixels and the chased animal in front of the chasing animal and the chasing animal has to travel faster than 3.4 pixels per frame. The smooth window size is 25 and min window is 30. Get chases."
AmadeusGPT Code Output:
def task_program():
behavior_analle animelBeetavioornaalssss)
chase_events behavior_analysee sosimasssociaaleeeetss/soo "orientation'],
[' ('<40', '==' (ess(((orienation.FFRNN]
speed ']
animal_state_compprr
Sodypartss=('all11,,
scherranimal_bodyyarttsttt smooth_winddw_sizeer nin_window==0) return chase_events
Query: "Define <loral genital contact|> social behavior where closest distance between this animal's "nose" and other animal's bodyparts "tail base" is less than 15, get the events for oral genital contact."
AmadeusGPT Code Output:
def task_program():
behavior_anee jehayior_anallssi MnimalBehaaiiorMaallysss
aoal_geitaa__conttt__eeee
Satavior_analysii.
sarellsssciilleeeeeeeee closest_distance '],
['< 15'],
bodyparts ['nose'],
otheranimal boodyaaasodyyatsss ['tail base '])
return oral_geni talccottacteeeeeenttt
Query: " Define <loral oral contact|> as a social behavior where closest distance between this animal's "nose" and other animal's "nose" is less than 15, get events for oral oral contact."
AmadeusGPT Code Output:
def task_program():
behavior janallis AnimalBehhvviorknnllysss oral_oral_comee haa___ottttt__eetttttee 4 behavior_analysee
pnimals_socialeeeenteee (ocossssstcossssttddl. closest_distame 'J,
['< 15'J, bodyparts = ['nose'],
24

---

otheraniial__oodparts ['nose') return "eal_oral_contat_eveee
Query:
"Define <lapproachesi> social behavior where the mice move from least 40 pixels apart sequentially to less than 8 pixels apart. Get approaches events." "
AmadeusGPT Code Output:
def task_program():
behavior_ane _analllssana ModmalBehhaviorAnnllysss
eistance_eventee behavior Lamalyis_analysiss. animalll ssros_sooilllleaeeeeeeeeeeseelle "rriiallvvent(("" ("ests((" ['>40'] }]
www.dddittteeeveeeeee = behavior_analyle
saomalsssoilleeeeeeeeee distance 'J, ['<8']]
approaches_event jeveeessseeesseeeeeeeeee = Event. add_ poo___eeenttaleeeeeeee (ditiiiieeeeeeettstsssteeeeeee (distance_evennts
crrse__dstance_evee
return approaches_events
"Define <lhuddlesl> social behavior where the distance between animals is less than 50, and the relative speed between two animals is less than 4 pixels per frame. The smooth window size is 61 and min window is 75. Get huddles."
Query:
AmadeusGPT Code Output:
def
behavior_analysis = Andmalleehaviooknnalysss/
huddle_events behavior httpwiirranallsss..eem
'speed') J]
['<50' '<'',,
bodyparts=['all1]
min sin_window=75,
smooth__windwwssiee:1))
return huddle_events
Query: "Define <lcontact|> as social behavior where the closest distance between animals is less than 12. The smooth window size is and min window is Get contact."
AmadeusGPT Code Output:
def task_program():
dehavior_anallsii Beavaioroloo
contact_events behavior_analysis. animals_social. Dociial_evett(( closest_distance J ['<12'], smooth_windn window_size=11 min_winddo window=5) return contact_events
Query: "Define <iclosel> as a social behavior where the closest distance between animals less than 24. The min window is 5. Get close."
AmadeusGPT Code Output:
def task_program():
behavior_anale _analysis = AndmalBeehaiiorknallyss) glose_events behavior sativiioranalssis..... <a min_window=:) return close_events
closest_distance ['<24'],
"Detine <lwatchingi> as a social behavior where distance between animals less than 260 and larger than 50 and angle between animals is less than 15. The min window is 100 and smooth window 15. Get watching."
Query:
AmadeusGPT Code Output:
def task_program(>>
behavior _analyisss AnimalBeeaaviorknaalysss
watching_ent events eehawior_annllsis, siimallssocial_eveeeee "iistan"" 'distance',
4 'angle' ], ['<260', '>50', '<15'J, bodyparts=[ all'l]
25

---

min_window=108
mmooth__innow__izeee
return watching_events
"Define <loral ear contacti> as social behavior where the closest distance between animal's nose and other animals' left ear and right ear less than 10, The smooth window size and min window is 15. Get oral contact."
Query:
AmadeusGPT Code Output:
def task_program():
behavior_anallsis = AndimalBehhaviooknnalyssss
deal_ear_cnttctteee
behavior_analysis, animalsssocialeevente closest_distanee
"closest_distance J,
['<10', '<10'1,
bodyparts=['nose"]
lleft ear', 'right ear'],
nin_window=155
boboth_window_iizee
return oraa_eer_contactteeents
Prompt robustness testing
Related to Main Section 4, we show the full base questions and generated permutations we tested to gauge robustness.
The input, base out of distribution (OOD) questions
(1) define </ed dips|> behavior where the animal's 's bodypart mouse_center bodypart
bodypart head_midpoint outside roi (2) When is the animal moving faster than 8?
neck in the roi
(3) define 
 as behavior where the animal is slower than 0. When is the
mouse freezing?
How much time the animal spends in the closed arm?
(5) How much time the animal spends outside closed arm?
mun
The 25 generated OOD questions
(1) Dips refer a behavior which an animal's 's bodypart, specifically mouse_center
and neck, are within the ROI while the head_midpoint outside the ROI.
(2) When an animal's 's mouse_center and neck are inside the ROI but the midpoint is outside, this is known as dips.
Dips characterized by the presence of animal's 's mouse_center and neck within ROI, while the head_midpoint
located outside the ROI.
() The behavior known as dips occurs when an animal's mouse_center and neck are within the ROI, but the head_miihhe lend_midee midpoint is outside of it.
(5) In animal behavior, dips
defined the presence of the mouse_center and neck within ROI, while the head_midpoinn situated outside
(6) what point does the animal exceed speed of 8?
(7) When does the animal go faster than 8?
(8) At what moment does the animal surpass speed of 8?
speed greater than 8? (10) At what time does the animal go beyond a speed of
(9) When is the animal traveling
(11) At what point can we say that a mouse is freezing? I define freezing as a behavior
4 where the animal moves slower than 0.5.
(12) When does mouse exhibit freezing behavior? Freezing is defined state where
4 animal's movement slower than 0.5.
(13) What
defini tion freezing in mice? Freezing is characterized by movement speed of less than 0.5. When does a mouse exhibit this behavior?
26

---

(14) When can we say that a mouse is freezing? Freezing is behavior where the animal's movement is slower than 0.5. What the conditions that lead to this behavior?
(15) What does mean when we say that a mouse freezing? Freezing a state where
+ the animal moves slower than 0.5. When does this behavior typically occur in mice?
(16) What is the duration for which the animal remains in the enclosed arm?
(17) How long does the animal spend in the closed arm?
(18) What is the time frame for which the animal stays in the enclosed arm?
(19) What the length time that the animal spends the closed arm?
(20) What is the period of time during which the animal remains in the enclosed arm?
(21) What is the duration for which the animal remains outside the closed arm?
(22) How long does the animal stay outside the closed arm?
(23) What
amount time the animal spends outside the closed arm?
(24) For how much time does the animal remain outside the closed arm?
(25) What is the length of time that the animal spends outside the closed arm?
Questions rephrased automatically by our Rephraser
(1) Define 
 as a behavior where the animal's mouse_center and neck are in ROI and head_midee mead_midpoint is outside ROI.
Define dips as event where the animal's couse_center and neck inside the ROI but the head_midpoint
outside.
(3) Analyze events where the animal's mouse_center and neck are in ROI and head_midpoint is outside ROI.
4) Define </ as behavior where the animal's 's mouse_center and neck are within ROI, but head_midpoint is outside. Define dips as the presence of mouse_center and neck within ROI and head_midpoint outside of it.
6 Give me events where the animal exceeds a speed of 8.
Give events where the animal goes faster than
(8) Give me events where the animal's 's speed greater than
(9) Give me events where the animal is traveling at speed greater than 8.
(10) Give me events where the animal goes beyond a speed of 8.
(11) Define freezing
behavior where the animal moves slower than 0.5. When does the
4 mouse freeze?
(12) Give events where the mouse exhibits freezing behavior, defineed as movement slower
4 than 0.5.
(13) Define freezing as movement speed less than 0.5 in mice. Give events where a mouse
4 exhibits freezing behavior.
(14) Define freezing as a behavior where the animal's movement slower than 0.5. What
4 are the conditions that lead to freezing in a mouse?
(15) what is freezing in mice? Give events where the mouse is freezing.
(16) Get events where the animal is in the enclosed arm and calculate their duration.
(17) Give events where the animal is in the closed
(18) Get events where the animal stays
enclosed arm.
(19) Get events where the animal is in the closed arm.
(20) Get events where the animal remains in the enclosed arm.
(21) Get events where the animal is outside the closed arm.
(22) Get events where the animal outside
closed arm and give me their duration.
(23) Get events where the animal is outside the closed arm.
(24) Give me events where the animal is outside the closed arm.
(25) Get events where the animal is outside the closed arm.
Code generated by AmadeusGPT without Rephraser
#Occurs 5 times
def task_program(> behavior_anale
AnimalBehaaiiorknallyiss
# Get events where mouse nouse_center and neck are within the dip_events = sehavior_analysie wwwallys..aimaals.......aaaa mimmalsobbjet_eeenttt ('ROI', 'overlap', bodyparts=[ mouse_center', 'neck' 'J, negate=False)
# Get events where head_midpoint is outside the ROI
27

---

tooooeeessdeerroeeeetts behavior_analyski http:///.....iilllo...eeeeeeeeea 'overlap',
heartsss"hhaddddppiiiiii 'I, negate=True)
# Combine the events using add_sequentaal_eventee wtw._haaddouttideerooiii. head_outside
Event. Event.add__eeuentiinn www.eeeeenntiileeeessccieeeeeeee
return cettside__ri_eeenee
#0ccurs times
def task_program():
behavior_analyse = MntmalBeehaiiorknallysss
# Get events where the animal's speed exceeds pixels per frame Bes__eexeee___eeeee = eehavior_anallste salmmllsssatt__eenttee eeveess""peed', '>8')
return speed_exceed____ewe
#Occurs times
def task_program():
behavior_analystt = AnimilBeehaiiornnnllysss
# Get events where the animal's speed is less than 0.5 pixels per frame freezing_events behavior_ane sehavior_analysis.o. return freezing_events
speed', '<0.5")
frOccurs 1 times
def task_program():
return "Freezing mice is defined as state where the animal moves at speed of + less than 0.5." "
#Occurs times def task_program(): behavior_ane behavior_analysis AndmalBehaviorknnalyssst
# events where the animal the closed arm
letloseddrrm_eeent behavior_analysis htt.raaallss..aimmls.loooo ans.aaaiisssceeeteeeettttt closed arm', 4 "overlap'' bodyparts=['alll aJ, negate=False)
# Calculate the duration of each event
durations [event. duration() for event in in_closed aerrd__ar__eeeeeeee arm_events(" animale
# Return average duration
return sum(durations) / len(durations) if durations else 0.0
#0ccurs 2 times
def task_program():
behavior_ande _anallisssssssssss MnimalBehaviiorhaallssss) # Get events where the animal is in the closed arm
melclosse_arm_evve
behavior_analysis.. 'overlap's bodyparts-['all1] negate=False)
hatpysiissaimmlssseeeeteeeeeeee closed arm',
# Calculate the duration of each event
durations- [event (urentduration() for event in_closed_armeevente animale ']]
# Return the total duration
return sumudratiioss) if durations else 0.0 #occurs times
def task_program():
behavior_analyse = MiinalBehaviorkhaallsss)) # Get events where the animal is in the closed arm mnlcclosed_arm_eeeeee
anw.aaamammssssoeetteeeesss closed arm', negate=False)
+ "overlap', bodyparts=('alll times of each event # Get the start devenn.sstrtt_raaae start_times [event. start_frame for event in un_cclose__am_eeentte animma0 'J]
end_ so__immes [event. end_frame for event in closed_arm__eennee annmaa0"" ']]
# Return the time frame of the first event if it exists
if start_times:
return (start_times(0) end_times(e])
#Occurs times
def task_program():
behavior_analysis. AnimalBehaaiiorknalyysss/
# Get events where the animal is the closed arm
e'ennss'clooedd closed arm',
"overlap', Dodyparts-''all'1 negate=False)
# Get the start and end times of each event
event_times dlevent.startttiie wtl.cllo__armmeeeeeeee # Return the event times
event. (on_tteendttme()) for event in
animale
return event_times if event_times else []
28

---

#Occurs 5 times
def task_program():
Dehavior_analysis MoimalBehaaiiorAnallysssS
I Get events where the animal is not in the closed arm
www.clooe__arm_eeeneeee = behavior_ane Dehavior_analyiis.o animals closed arm', "overlap', bodyparts-['alll]] negate-True)
# Calculate the duration of each event
durations [event. (ueentduration() for event
htw.:colose__am_eeeeeeeeeee animal0
# Return the total duration
return sumddurations) if durations else 0.0
Code generated by AmadeusGPT with Rephraser
#Occurs times
def
task_program():
behavior_ananl jehavior_analysis AnimalBeehavioornaaayssssS
mouse_ www.cctter_innooeeeveeeeeeeeeeee
criimal_.objecteeeee ('ROI',
'overlap', bodyparts=[" meteemmmuuseeeeeeeeeeeeee ], negate=False)
neck_in_roi_events behavior__mallsiis. stte:o/oor.aallllss..aaalllaeeee animals_.bjeetteeentts ROI', 'overlap',
< bodypartss="nnekk1]] neckk negate=False)
"overlap', pookyaarts="hhedd_midoooi
negate=True)
dips_events Event. add_tadd_simme
+ neck_in_roi_events head_midppin__nn__innrr.....ew
return dips_events
#0ccurs 1 times
def task_program()):
behavior_anallsis AnimalBehaviorrknalysss))
mouse__cetter_n_roi..ee
vehavior_analysis
htt.:oonnnllllls..nnnnnssoeeeeee animalss.bheecteeentts (RROI',
'overlap', bodyparts=[' "oobsaartss("muuus_ccttte""]]] negate=False)
neck_in_roi_events behavior_anallsii. htrplllsiisaaimmls_oojeeeeee Deriect_events"ROO0 'O 'overlap',
bodypaarts=('neckk], negate=False)
htd.mildoonnnnot.nnoooo.eettttt behavior_anallsis. hwwallysis.aimmlsss.j.eee....eee
dips_events poddyaarts("hhea__midppoo negate=True)
= 'overlap's
neck_in_roi_event head_mippcint_nnntii_.........
fig, ax mehavior_analyeee hat.ayssspooottaaaetttrrrooooooe olrrarrrrreettttoooooooorrrsssss return fig, ax #Occurs 5 times
events-dipp_eevnts))
def task_programm):
bshavior_anallsii = AnimalBBehaviiokknaalssss
faster_than_8_ teeer__ttha___eeets __eentss behavior_analysis..o htw.lllsiis.aammmlssstttsttseeee ('sped", '>8')
return faster_than___eww
#0ccurs times
def task_programm):
behavior_analysiis = AnimalBehhavorknnallyss(
freezing_events behavior_analysee (ntinalssstateeeeentssss '<0.5')
return freezing_events
#Occurs times
def task_program():
behavior_analysiis AeiiaalBBhhaviornnaalyssss
in_closed_arm_eeent
wtt.aaallssss.aimasaaeeoeeeeeeee ('closed arm',
"ovorla''''p bodyparts ['all'], negate False)
mnlclosed_arm_duratiie in_closed__rm_eeents duration()
return in_closed_arm_dee aum.uuraiiann
#Occurs times
def task_program():
behavior jehavior_analysis = MnimalBehaviooknnallyss))
in_closed_arm_evest
behavior www.alyyis..ainmlll..oeetteeeeee closed arm',
'overlap', bodyparts=['all '])
return mn_cclose__arn_eeeee
#Occurs 4 times
def task_program():
behavior_analysis = AndmaalBehhaiiookkaalysss
29

---

eehavior_analyiis sehawiio_onalylls..nniilll.../.o andimals_ojject_eeents ('cloeed arm', "overlap', bodyparts-C'all 'I,
return cutsideeclosed__rm_.e.w negate=True)
#Occurs times
def task_program():
Dehavior_analysis = Andmalleehaviloknnallysss
outside_clone ciciiie__lssd__aree @aoosed_arm_eeets behavior_analye _ariloisnnnlliiss... aninmals_.bjeeecteeeettt closed arm',
+ "verlaa', bodyparts="'alll
negate=True)
(animal: [event. [event.duration() for event in events] for closed_arm__eeetts return clwsed_ars......o porssd_armmdduse duration arrerdaatnduuiiionn
outside_cle ciaeee__ose______ aerd__rm_duratee
animal, events in
Additional stress-testing with naive users and GPT4
We additionally stress-tested the system with naive users. We released a fully usable web App to AmadeusGPT using GPT-3.5 with the 3 demo datasets used in the submitted manuscript (EPM, MABe, and MausHaus videos) and asked neuroscience behaviorists to test it - namely, we added example prompts they could run and chat-interface ask their own questions, Here, we now collected extra prompts not generated by the authors for testing and we will include this in the revision. In brief, we sampled 30 naive user prompts at random of the 362 submitted via our App (prompts having caused programming eтror are marked with X), and found roughly 1 18% error rate with GPT3.5 (see more below for GPT4, which was only 10%):
X Get angles and distances between all body parts. Plot a UMAP graph using the resulting data.
Give me the duration of time the animal engages with the object picked and the events where the animal overlaps with the object picked.
X Perform hierarchical clustering and plot the dots with different colors based on their clusters.
Define <lsapl> as behavior where the animal's body length clongates. Do you sap in the epm?
Give me the tracking of all animal's noses with each nose being shown in a different color.
Define < |peckingl > as a behavior where the animal's head_midpone not in ROI5 or RO14 and the mid_back point is in ROI5 ROI4.
X Plot the distance between animals over time.
X What is the speed of the changes from the left the right arm?
Plot regions of interest (ROIs).
X Define < trelative_heald
the angle between the mouse_cent center and the head_midpond Plot the variation of < Irelative_headd_nnglee >> over time.
How many animals are there in this data?
Give me the keypoints.
Give me the total distance traveled by each animal.
Define < |watch| > as social behavior where distance between animals is less than 260 and larger than 50 and head angle between animals is less than 15. The smooth window size is 15.
X Plot animal center of mass x-coordinate, velocity, acceleration, and head direction.
Define < |freezingl > as a behavior where the animal is in the same position for longer than 10 seconds.
Plot figure comparing the time the animal spends in the open arm (ROI0) and the time the animal spends in the closed arm (outside RO10).
X Plot a bar graph with the first bar representing the total time the animal spends in ROI0 (open arm) and the second bar representing the total time the animal spends outside ROI0 (closed arm)
30

---

Calculate the number of successful spontaneous alternations using sliding window calcula- tion. A successful spontaneous alternation is defined when, in the last arm entries, all four arms (ROI0, ROII, ROI2, and ROI3) are visited at least once.
X Plot the euclidean distance between the nose points of animalo and animal2 over time.
Why is the angle limited to 180 degrees instead of 360?
Show which what? Please provide more information or specify your request.
X Define < [leedddirectione > as the orthogonal angle to the line between left_ear and right_ear.
Give me the amount of time the animal spends in ROIO and the events where the animal overlaps ROIO.
Calculate head direction using the ear points and plot it in radians.
X Plot each grooming bout.
Define < thead_scamss > behavior where the animal's head-direction shifts by more than 15 degrees to one side and then the other within 5 seconds. Give events where head scans happen and bouts for head scans.
"Find events where there are errors in tracking due to outliers mean distance between all points, animal acceleration, and velocity. Remove the labeled point causing the error and fill in missing values by interpolation. Plot the and coordinates of the animal's 's center of mass before and after the tracking correction.
Give me the frrquency of rearing events for the animal.
Print overlap_roio___e rea_roeiieeeessssseeeeeeeeeesee events and its items.
We analyzed the logs from 362 queries, 242 of which were from unique prompts (i.e., from manually typed ones rather than executing the demos). Out of the 362 queries, 329 were automatically rephrased. We found AmadeusGPT had reported 129 "errors": 38 were caused by unsupported features or undefined variables, and were thus explained to the user, while 64 originated from programming errors. Therefore, from 362 queries there were an 18% programming error rate, and 11% unsupported feature requests rate.
Then, we took 10 of the failed runs from external users and re-ran them with GPT4 Now, of the 10 that previously failed, only failed, and correctly output clarifying queries to the user"; This means that if GPT4 was used, given the randomly sampled prompts, the error rate would be only 10
X Get angles and distances between all body parts. Plot a UMAP graph using the resulting data.
Perform hierarchical clustering and plot the dots with different colors based on their clusters.
Plot the distance between animals over time.
What is the speed of the changes from the left to the right arm?
Define Irelative head_ head_angle] head midppint. Plot the variation
the angle between the mouse_ center and the |relative_head Laaerive_eed__allee_____ee > over time.
X Plot animal center of mass x-coordinate, velocity, acceleration, and head direction.
Plot a bar graph with the first bar representing the total time the animal spends in ROI0 (open arm) and the second bar representing the total time the animal spends outside of ROI0 (closed arm).
Plot the euclidean distance between the nose points of animal0 and animal2 over time.
X Define right_ear.
< [head_ddrecttonn as the orthogonal angle to line between left_ear and
Plot each grooming bout.
*For (2), (4), (5), (10), the output is message indicating either that the feature is not implemented or that the query does not provide all information, and explains what would needed either in terms of development clarifications from the user.
31

---

Example from Explainer
Here an example query on the EPM dataset and relevant outputs for the explainer module:
Before rephrasing: How much time does the animal spend the closed arm which Roieg
function code
def [ask_program(>> Sahavior_analysiie
AnimalBBehaviiornnaalyssss
parrlap_close__arneeeee behavior_analyyeee hawassaaannnnss..eeeeeeeeeeeeeee ('RO0'',
overlap'' bodyparts=C'all'
return wherlap.close__arm_eeee
generated explanation 1) The code unimalBeehaviornnallyii class, which is tool for analyzing animal behavior.
AninnaBehaaiorrkaaLysSSS[[[ creates an instance of the
The next line Joverlap_closee clopppcccde____eeeeeeeeeee arm_events
saapalls_oeecteeeeetee ('ROO'' 'overlap', bodyparts-C'all ' uses this tool to find events where the animal overlaps with the closed arm region of interest (ROIØ). It does this by calling the [[ninals_objeec_eeenttt method of the animalBehaaiorrnnalysi class, passing in the parameters 'ROIO', 'overlap', and ['all']. This method uses the API documentation access the necessary data and + perform the analysis, returning the events where the animal overlaps the closed arm.
pehavior_analyse
2) The meta answer is: For animal0, the total queried events last 7884 frames (total video length is 15076 frames).
This means that out of the entire video, there were 7884 frames where the animal overlapped with the closed arm ROIO. The total video length is 15076 frames, so the queried events make up significant portion of the video.
Analyzing this in terms animal behavior, it suggests that the animal is spending a considerable amount of time interacting with or being near the closed arm.
Example from Self-correction
Here is an example query on the MausHaus dataset and relevant outputs for the self-correction module:
Query: Plot trajectory of the center the mouse body
def task_program():
bshavior_anallsis = AnimalBehaaiiorknaalysss
traj_plot_info sehavior_analyyee ploc_trajectorr(("ceeet""" return traj_plot_info
ValueError: center not defined in the data. Supported bodyparts are [ L'nose', 'left_ear', "right_ear', 'left_ear_tip', 'right_ear_tip 'left_eye', 'right_eye" 'neck', 'mid_back', "mouse_center 'mid_backend', mid__backedd2.. 'mid_backend3', "tail_base', 'tail1', 'tail2', 'tail3', 'tail4', 'tail5', "left_shoulder', "left_midside', 'left_hip', "right_shoulder', 'right_midside', 'right_hip', 'tail_end', 'head_midpoint""
Diagnosis:
The error message indicates that the variable "center" not defined in the data. The supported body parts for plotting the trajectory are listed in the error message.
To fix this issue, you need make sure that the data contains "center" body part. + You can check if the data has the required body part before plotting the trajectory.
revised function code:
32

---

def task_program()
behavior_analyeee AnimalBeeaaviornnalyysss
traj_plot_info behavior_analysis. htt.aaayyssspppll.rrreoooooooooo return traj_plot_info
Horse Gait Analysis
Horse gait analysis demo
R: Query:
"could you
the daty facsor offfrontfoot
explais
X Chat Output:
duty facsor offf greater than indicating that apeads plase
front eving phace.
Query:
-ak Could you eleo calculate the duty factora uxing the keypoint "gddhindoous' and compare bothe"
Chat Output:
def bask progren():
plu App
anslysie mnialeehwiinnssyss: prappppeaaiiaaeeee soypoint_nazaae nataanttt_teeeiieeeee ("Ottfrotttttttt
offerostient analysia betavier
otthinstoer_ analysia beheviot ansyaasi can gest, melyvasttiin beypoist_ suroooe (OOthinstwoe"))
ofthindfast doty. factore
ansllyylttttttilllla.ao. "orrtll""yyyy... Suryy [antoor""]
coepaziaon tp..me(ottttoetttt oucation) up :0 www.eotttttctttrreeeeeeet
returo expasiton
2: Query:
Could exlsinn what that means?'
K Chat Output:
The duty facsor is a measuze of the smount of tize a lisb spends in contact with the ground duzing gait cycle. It is calculased by dividing the stance ducation Itine when the 1imb is in contact with the ground) by the stride duration (tine for corpiete gait oycle).
By conparing the daty factore of the octtrontfoot and offhindfoot, ve can detarnine i2 there is any difference in the anount of time these lixbs spend in contact with the ground. positive value indicates that the offfrontfoos bas longer duty factor, zeaning agends nore time contact with the ground compared bo the ofthindfont. negative indicates the oppaxite, with the béfsindfoot having longer daty factor.".
2: Query:
aeriee Whfo wil the repitto antiratied baryy sntiuattiio
you previous analysis plos
folloxing keypointsi foot offfronttetlock) Offinee] Blbov
Limb
prograe)):
beharics_ ansiyaia Astaiieeareeeennnyyssee
analycle betavior, wawlloorsssallaaaaooo mallyiia.ron gaft nnalyyiaaliii kwspoint hogpoint_axeesttotttt ptw.r/osssssofffroososoo""""""
resuite(
Li0h, heypoin pamco Potitsontioot. officontieslock. oétinze, Eibow, shoulder"1)
Figure 8: Horse Gait Analysis. Examples of user queries and AmadeusGPT outputs, including example the Web App interface. Data adapted from [36].
33


Reviews:
[{"review_id": "pEUkZbhAS5", "reviewer": "Reviewer_XjKy", "summary": "This submission introduces a system, Amadeus, for performing behavioral analysis on animal videos. This system combines three elements: an API with descriptive docstrings for performing common behavioral analysis tasks, a modified version of GPT3.5 with an enhanced context window, and a prompt tuning of this GPT3.5 to allow for construction of task programs for behavioral analysis using a natural language interface. What is claimed is that (1) the language model can extend beyond its nominal 4096 character token window (2) the system allows for behavioral analysis with no additional code written (3) the system is easily extensible. The authors’ provide schematics of the system architecture and the integration between various components. There are numerous examples for how written code is produced from natural language. There are a few evaluations provided (1) comparing human raters and the amadeus system in computing time spent in arms of an elevated plus maze (2) comparing the performance of the system in a select number of tasks from the MABE challenge and (3) comparing the ability of a trained rephraser model to domain adapt queries to the developer’s syntactical patterns. There is a fairly comprehensive inclusion of the API in the appendix as well as examples of the rephraser module. \n\nOverall I found the contribution novel, of potential immediate utility to academic neuroscience labs performing behavioral analysis, and as an example for other groups hoping to automate analysis in other systems. I however found the evaluations and description of the methods somewhat lacking, and the generalizability and extensibility of the somewhat unclear. These would need to be improved for me to strongly endorse the paper. \n", "questions": "•\tTable 1 only lists a subset of MABE tasks, the model results should be reported across the full benchmark.\n\n•\tThe lack of methods makes it difficult to know the details of the memory augmentation approach and the extent of training of the rephraser module (and how much one expects the later to generalize).\n\n•\tI found the memory augmentation result under-contextualized and other work in this space should be discussed. Augmenting LLM memory window is a subject under intense technological development and the contribution should be put into proper context, especially because it is a secondary result. \n\n•\tThe EPM results are hard to evaluate because of inter-human variability. An experiment with clearer ground truth would help. \n\n•\tIn general, hallucinations are a real problem for LLMs and I am not convinced by the given evaluations that they would not occur (e.g. for what fraction of a given call does the system produce the correct result). This is especially problematic for a system designed to be used by people who do not code and may not be able to debug problems. \n\n•\tThe API is fairly expansive and includes example of most functions call and keyword arguments present in the given examples in the paper. This could mean that the system is limited in functionality to simply modifying parameters and numerical values given as input to the functions. Some statement about the expected generalization ability of the approach and the requirements for adding new functions to the docstring (e.g. does one have to provide examples of every new keyword input) is needed. \n\n•\tThe claim that using the system would obviate the need to be able to code is strong. As the authors’ note, hallucinations are a problem for LLMs. Many in the field regard LLM solutions as an accelerated rough draft to be ‘fact-checked’ by a domain expert. Similar to the rephrasing analysis, it would be nice to have a statement about the reliability of the results, ideally across a pool of fairly naïve users. \n", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "•\tThis is the first LLM integration I have seen in the neuroscience and I think the approach is potentially interesting. In some ways I think it could be even more interesting for codebases that analyze on standardized experimental equipment (e.g. fluorescence imaging, neuropixels) because the set of tasks to perform is arguably more standardized than behavioral analysis, where the approaches, species, and questions can be fairly diverse. \n\n•\tMany researchers in the life sciences do not have a formal background in writing code or have to interact with a fairly specialized API to analyze data, and so assistance is useful here.\n\n•\tThe system schematics and examples are clear. \n\n•\tThe API documentation is helpful, and the integration of state of the art systems like SAM into an open-source (if this is to be distributed) codebase is helpful. \n", "weaknesses": "•\tThe manuscript is missing descriptions of the system architecture and training details. It is difficult to fully understand how the system was trained without this.  \n\n•\tThe evaluations are somewhat cursory. The results on the EPM are hard to evaluate because the ‘ground truth’ from the human raters is quite variable. The MABE evaluation is only presented across a subset of tasks. \n\n•\tThe generalizability and extensibility of the system to new users and new behavioral tasks is unclear. Part of this is the rephraser example is somewhat limited, but it is also unclear how successful new users will be in writing functions with appropriate documentation and that documentation hints are not interdependent. \n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for noting the novelty and potential strength of our application to life sciences, and thanks for the suggestion to consider more experimentally constrained (i.e., imaging) settings, which we can try in the future. Here, we focused on classical behavioral tests, mostly for this reason, as many labs deploy these rather standardized setups worldwide (i.e., EPM). \n\nTo address **weaknesses**: \n\n- **(1)** We do not fine-tune GPT3.5 (or GPT4) or the segmentation (SAM) or pose estimation models (SuperAnimal). Our system consists of three GPT3.5 instances with different system prompts, the pre-trained computer vision models, and our API implementation, we will clarify in revision. Re: system architecture, we hope Figure 2 gives a comprehensive design of the system, and more details about API design and system prompt design are provided in Appendix. \n\n- **(2)** EPM, and mouse behavior in general, is somewhat subjective, hence the absolute ground truth is a mixture of experts. While we fully appreciate this is not always satisfying, this is a real-world application and challenge Amadeus is likely to be faced with. MABe contains the experimentally-defined tasks and expert-annotated behavior tasks, of which 9 of 13 could likely be solved by pose estimation and spatiotemporal reasoning, and the other 4 are not feasible without a ML classifier, such as predicting the animal strain, etc. We now added the 3 other tasks that Amadeus could tackle, which we will add to the final version: \n\n|               | T4 approaches | T10 oral-genital-contact | T11 oral-oral-contact |\n|---------------|---------------|-------------------------|-----------------------|\n| PCA baseline  | 0.00          | 0.00                    | 0.00                  |\n| Top-entry 1   | 0.020         | 0.015                   | 0.013                 |\n| Top-entry 2   | **0.026**         | 0.029                   | 0.023                 |\n| Top-entry 3   | 0.022         | 0.015                   | 0.014                 |\n| BAMS          | 0.02          | 0.0165                  | 0.014                 |\n| Amadeus       | 0.014         | **0.05**                    | **0.05**                  |\n\n- (3) To expand on what the system can do we now have released a fully usable front end user interface App to Amadeus using GPT-3.5 with the three demo datasets used in the submitted manuscript (EPM, MABe, and MausHaus videos) and asked neuroscience behaviorists to test it – namely, we added example prompts they could run and a chat-interface to ask their own questions. We now collected extra prompts not generated by the authors for testing and we will include this in the revision. In brief, we sampled 30 naive user prompts at random out of the 362 submitted via our App (errors= ❌), and found a roughly 18% error rate with GPT3.5 (see more below for GPT4, which was only 10%); due to space limits, we show a few here:\n\n '✅ Give me the duration of time the animal engages with the object I picked and the events where the animal overlaps with the object I picked.',\n\n '❌ Perform hierarchical clustering and plot the dots with different colors based on their clusters.',\n\n \"✅ Define 
 as a behavior where the animal's body length elongates. Do you see sap in the epm?\",\n\n \"✅ Give me the tracking of all 3 animal's noses with each nose being shown in a different color.\",\n\nFrom 362 queries 242 of which were from unique prompts (i.e., from user typed ones rather than executing the demos). 329 were automatically rephrased. We found Amadeus had reported 129 “errors”: 38 were caused by unsupported features or undefined variables, and were thus explained to the user, while 64 originated from programming errors. Therefore, from 362 queries there were an 18% programming error rate, and 11% unsupported feature requests rate.\n\n **Questions**:\n\n- **1 & 2** are hopefully answered above. Note that the Rephraser LLM requires no training and the difference between the code generator and Rephraser is the system prompt, which is used for LLMs to do in-context learning; we will be sure to add the Rephraser prompts in a revised version.\n\n- **3**, we note that of course there is a lot of concurrent work right now, but we did our best to put in context and cite papers that we saw that were closest to ours at the time of submission (even if they did not impact our own ideas), but if you have concrete examples we missed, please let us know!\n\n- **4**, this is why we semi-constrain with the API, but of course we collectively need to understand issues with LLMs. New work that includes GPTs to write executable code are being tested rapidly, and it seems that GPT-4 give less hallucinations than GPT-3.5 (Cai et al. 2023 & Wang et al. 2023). Since our system can easily switch from GPT-3.5 to GPT-4, the hallucinations could be reduced with the improvement of the underlying LLMs used. However, our Rephraser truly does attempt to write code for 90% (ie 329 of 362 from outside users) of the time, i.e., tests if code is runnable and otherwise parses errors and tries again.  Therefore, we are happy to tone down any claims of “no coding ever,” but we do see this as a truly new way to interact with SOTA models (like SAM, etc), so we hope you agree this is still a useful demonstration for the field and neurips community. \n\n- **5**, we note this is still built on GPTs, so it’s not merely bound to our API. Our system prompts and API sets the ground rules to have it understand common behavioral analysis queries, but it doesn’t limit its capacity. As an additional test, we took 10 of the failed runs from external users and re-ran them with GPT4, which reduced errors to 10%, suggesting better LLMs also increase performance/our API is not too limiting.\n\n- **6**, see point 3 in weaknesses above, we have now asked expert mouse behaviorists to use the system.\n\nWe hope this clarifies our novel contributions and improvements, and that you’d consider raising your rating. Thank you!"}, {"review_id": "b9Zqs0mh71", "reviewer": "Reviewer_ZigY", "summary": "This paper presents a novel interface and deep-learning system that enables interactive animal behavioural analysis using only natural language, on tasks that previously required extensive coding expertise and technical machine learning knowledge. The proposed framework integrates various modules based on LLMs and existing vision networks dedicated to various parts of the task, such as understanding the data processing API and rephrasing user prompts. This work also contributes a novel long-short-term memory technique in the context of LLMs. In combination, the contributed interface and systems achieves state-of-the-art performance on established tasks in the domain of animal behavioural analysis, and has the potential to support domain experts in using neural techniques and software frameworks that are currently difficult for them to use.", "questions": "1) What are the general applicability of the long-short-term memory module and the entire framework to domains other than animal behaviour analysis?\n\n2) Did any animal behaviour experts tried using the system / reviewed the prompts in the experiments of this paper? If so, what are their opinions on the ecological validity of them?\n\n3) What is the more low-level success metrics of the code generated by the system, such as the compilation error rate?", "limitations": "The limitations of this work were adequately discussed by the author(s).", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "- Enables the novel user experience of analyzing animal behaviour using only natural language, which could be an important affordance for domain experts \n\n- Well-designed, workable end-to-end system that reasonably combines multiple deep-learning modules, all with dedicated purposes in various aspects of the task.\n\n- Effective use of SoTA techniques in deep learning to achieve high performance on established domain-specific tasks\n\n- Introduced a novel long-short-term memory module that is compatible with LLMs and shown effective usage of such module in the contributed framework", "weaknesses": "- **Unclear general applicability of the long-short-term memory module and the framework in general:** From a machine learning perspective, the most significant technical contribution would be the long-short-term memory module in the context of LLM that the author(s) developed. The current paper discusses this module primary in the context of animal behavioural analysis. I would like to see some discussion of the general applicability of the memory module to common domains utilising LLMs, and also the general applicability of the entire framework to other domains, such that researchers working on those domains can benefit from the technical contributions of this paper.\n\n- **Unclear generalizability of the work due to author-crafted prompts:** This is a limitation the author(s) have acknowledged, such that the experiments performed by the author(s) all followed prompts crafted by the author(s). I also acknowledge that the author(s) have attempted to alleviate this issue by introducing OOD base questions and using similar task descriptions as the original definition in the MABE challenge. However, I am curious about the opinion of the end-user (i.e., animal behaviour analysts/specialists) on the ability of the system in handling their prompts. It would be great if the author(s) can include some study (could be formative/informal) and/or discussion regarding the ecological validity of the system. It would be great if the author(s) can report lower-level success metrics, such as the rate of having syntax/compilation errors of the analysis code generated by Amadeus.\n\n- **Limited technical novelty and contribution for Machine Learning**: Related to the first point, the primary technical contribution of the paper in terms of modelling techniques is the long-short-term memory module, which is limited. I believe more extensive discussion and/or experiments on this module could be helpful for researchers in the ML community to build upon the findings of this paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Firstly we would like to thank the reviewer for pointing out the novelty of our overall system and the novel dual-memory system. We’d like to add that we aimed to be generous in related works, we are one of the first systems to show LLM-generated executable code and error correction, and our Rephraser model we had not seen before (of course since early May more works have come out).  Thus, we want to push back a little that this paper is just a summation of many parts. We hope below (and response globally and to other reviewers) will convince you this is a paper for not only the machine learning crowd, but also for neuroscientists. \n\nTo address **weaknesses**:\n\n- **(1)**  This is a fair point that we only discuss or test our system in the context of animal behavior. We can easily speculate that this is much broader, there is nothing inherent about it being related to behavior, but we wanted to be careful to not over claim this in the manuscript.  To speculate, many applications that demand the system to retain distant context for tasks like answering questions or ongoing learning. Thus, we believe the general capability of this framework is also a good contribution to people working in other domains.  Our modular design and philosophy of the atomic APIs vs. integration APIs can provide interesting references to people who want to build a flexible, extendable analysis framework using LLMs and machine learning models and open source libraries. \n\n- **(2)**  To probe the system we now have released a fully usable front end user interface to Amadeus with the 3 demo datasets used in the submitted manuscript (EPM, MABe, and MausHaus videos) and asked neuroscience behaviorists to test it - -namely, we added example prompts they could run and a chat-interface to ask their own questions. Over 100 signed up in 48 hours, and we admitted 30 as alpha testers who stated they worked with laboratory animal behaviors.\n\nWe analyzed the logs from 362 queries, 242 of which were from unique prompts (i.e., from user typed rather than executing the demos). Out of the 362 queries, 329 were automatically rephrased.  We found Amadeus had reported 129 “errors”: 38 were caused by unsupported features or undefined variables, and were thus explained to the user, while 64 originated from programming errors. Therefore, from 362 queries there were 18% programming errors, and 11% unsupported feature requests.\n\nThe users also had the option to give a “like”  or “dislike” about if the output matched their expectations, and out of 51 responses 6 dislikes reported to the fully-executed outputs, 18 dislikes to code errors, which of course is expected, and 33 likes. Thus, while of course there is room for improvement, we hope this generally addresses your excellent point on generalizability to real-world users.\n\n- **(3)** We do appreciate that our main technical contributions relate to the novel dual-memory system, but we also think the whole integrated system of multi-LLM models working together to write, refine, and explain results after the python interpreter is sufficiently novel, especially at the intersection of neuroscience and machine learning. We certainly will add to the discussion/paper about other domains of interest for this system-level approach too. We will also add more about our memory module: \n\nImplementing short-term memory via a deque for LLMs is straightforward. However, our focus is on long-term memory. Two main approaches exist: 1. Altering the transformer architecture, 2. Using external storage like vector databases seen in generative agents and recent works. Approach 1, altering architecture, is the natural way to extend LLMs' memory span. It requires further investigation to determine if the extended span affects language task performance, a topic we plan to explore. Note that this might incur higher costs compared to GPT-3.5 API calls. For Approach 2, seen in projects like Auto-GPT, a vector database serves as external storage for context embeddings. This method can be expensive due to embedding computations and a vector database lookup operation complexity of O(Nd), N being vectors and d their dimension.\n\nOur long term memory is using a mixed strategy: For long text such as source code, our dynamic module loading is similar to those works that store and look up embeddings using vector databases. The dynamic module loading fetches the most relevant integration module based on the query of the user, providing a flexible way to collect useful integration modules from the community (2). However, we also noted that for interactive behavior analysis, if the user just wants to retrieve some content such as the definition of the behavior,  our symbolic pointer-based long-term memory is also an effective solution. Firstly, using symbolic pointers,  there is no need for LLMs to calculate the query embedding every time, thus lowers financial and computational burdens. Additionally,  as definitions of many behaviors are highly similar in text and fetching context based on embeddings of description can be error prone.  Our symbolic pointer approach uses regular expressions to look for what is in the special tags such as <> and <||>. It guarantees that the fetched description is correct as long as users keep track of the used symbol names, as programmers need to remember the variable names. Secondly, since our symbolic pointer approach implements long-term memory using a hash table, it provides cheap insertion and lookup operation with complexity O(1), a sharp contrast to O(Nd) with a vector database.\n\nIn summary, we are one of the first works that propose long-short-term memory for this kind of system and our mixture strategy for long-term memory is a good practical solution. We hope the above responses answered your **direct questions** (see points **3**, and **2** respectively). We thank you for your expertise, and hope our new analysis and clarifications enable you to find our work now acceptable, thank you!\n"}, {"review_id": "9sz0iRj7IX", "reviewer": "Reviewer_eSvp", "summary": "This paper proposes Amadeus, a novel natural language-based interface that leverages large language models like ChatGPT and vision-language models like SAM for animal behavior tracking and analysis. Amadeus leverages LLMs to generate code outputs, which can be executed to retrieve visual information and memory information and return requested responses from user queries. In particular, Amadeus proposes a novel dual-memory mechanism that combines short-term and long-term memory banks to effectively analyze extended contexts such as long videos. Experiments demonstrate that Amadeus achieve state-of-the-art performance on the MABE Behavior Challenge 2022 benchmark.", "questions": "See \"weaknesses\"", "limitations": "Limitations are mostly sufficiently addressed. Another limitation to add is that the current approach is bottlenecked by the capabilities of LLMs and VLMs (e.g., SAMs) being used. Perception errors can occur due to the limited capabilities of current VLMs, and the ability to produce correct responses given more complex queries and more demanding tasks is bounded by the capabilities of current LLMs.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper is generally well-written, and the figures are generally informative about the details of each module of Amadeus.\n\n- Leveraging recent advancements on LLMs and VLMs to analyze animal behaviors is a novel idea and holds significant potential for scaling up to more complex animal behaviors and a larger number of animals in the future. One of the strengths of this approach is its accessibility and training-free nature, which makes it adaptable to novel scenarios with relative ease. ", "weaknesses": "In Fig. 4, the implementation of long-term memory storing and retrieval processes is a bit unclear, as the generated code does not explicitly store or retrieve information into / from the long-term memory bank.\n\nWhile the proposed approach achieves the state-of-the-art results on the MABE Behavior Challenge 2022 benchmark, it would be beneficial to include discussions on the failure cases, and analyze their source of failures (e.g., due to mistaken VLM perceptions, or due to mistaken code generations, or due to compounding errors caused by the wrong values pushed to the memory banks, etc.)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for noting our novel idea and future potential to scale more broadly, we agree and are excited about this prospect. We also appreciate your noted weaknesses and do our best to address them.\n\nTo address your noted **weaknesses**:\n\n- We will clarify Figure 4 by noting the generated code is examples without the memory (i.e., it makes mistakes) vs. with the memory generated code, and additionally add a diagram that makes the flow clear (add 
, this goes to memory so tokens can be used otherwise, and recall with keyword).  Conceptually, the storing and retrieving process are implemented as a default system behavior that is being executed silently at every iteration of the question-answering and we described the implementation details in section 3.3 and section 3.5. To clarify further, the reason we implemented storing and retrieving as a system behavior instead of API calls in the generated code is that we did not find them necessary to be included into APIs because they do not need to adapt to user inputs and we tried our best to save token consumptions.\n\n- We will also show mistakes in the Appendix, as of course it does not reach 100% performance on the task, and this can come in two flavors: (1) Amadeus outputs unusable code, or (2)  the code cannot capture all the behaviors in the video. Based on external testing, the rate of code failure was 18% and 11% unsupported feature requests with GPT3.5, and only 10% errors with GPT4 (note we used GPT3/5 for MABe results). For MABe results, we ruled out failure cases caused by errors of code generation because we have self-correction mechanism that can fix obvious code generation errors for social behavior APIs used in MABe and we cache the correct code when iterating the code through 3000+ video data (this is also, of course, to save the budget of running tens of thousands of ChatGPT API calls).\n\nWe also agree in general with your noted limitations of LLMs, but hope these additions and clarifications will help you increase your rating of our work! \n"}, {"review_id": "qYZln0drWk", "reviewer": "Reviewer_f7B3", "summary": "This paper proposes AMADEUS, a GPT3.5 powered system to perform animal behavior data analytics given a natural language user-given query and a video depicting animal behavior. The model works by using GPT-3.5 to generate python code which makes calls to a instance segmentation and animal pose model, as well as hard-coded modules to compute animal-object relationships and operates with their outputs to return results according to a given query. The proposed model is tested with 3 popular animal behavior analysis benchmarks, showcasing different domains, queries and tasks, showing behavior consistent with human annotators and surpassing existing baselines.\n\n\n", "questions": "NA", "limitations": "Limitations are mostly addressed. The biggest missing limitation is how general is this approach to other animal behavior analysis, given the strong dependence on the hand-designed modules.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- Related work: The paper is open about closely related work, and assigns credit for the contributions that are used in the AMADEUS system. \n\n- Method: \n\t- I appreciate the discussion about the design of the API. It would be valuable to understand what would be the effect of not having made the specific choices mentioned in lines 150-159.  \n\t- I appreciate the Dual Memory Mechanims proposed (includeing the illustrative example of limitations of short context windows). I would like to understand the main differences between that appraoch and the one used in Generative Agents, and a discussion on why their execution flow is more expensive. The memory mechanism also has valuable usability features, such as being able to store and retreive states.\n- Relying on LLM allows to provide complex data analysis queries that were not possible in previous methods, as well as follow up questions to a given data analysis. \n- The proposed method, besides flexible is high performing in existing benchmarks, consisting on varied types of queries, expected outputs and data domains. The method perorms above the state of the art, both qualitatively adn quantitatively, surpassing all submissions in the MABE 2022 behavior challenge.\n\n", "weaknesses": "Novelty:\n- The work is very similar to existing works using LLMs for computer vision tasks (ViperGPT, VisProg), with the main difference being that they are applied to the animal behavior domain. This is however reported in the related work section, and the work includes other components to handle long duration data. \n\nScope: \n- This looks more like a systems paper, with many of the contributions already existing in previous works, and the main contribution being in combining these existing contributions for a new domain. I think that the paper does have value, but I am wondering if it is the right scope for NeurIPS.\n\nMethod:\n- The performance on the method seems highly dependent on the data analysis modules, which are handwritten. How general are these modules to different kinds of animal behaviors?\n\n- I am suspicious of the robustness and stress testing section. The full system relies on GPT3.5, and that same model is used to stress-test the system. I would like to see how good is the performance (with/without rephraser) from independent researcher-given queries.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for pointing out the high performance, flexibility, and new additions that our work brings that related work have not addressed yet (dual-memory), and also we thank you for praising our related works- its a quickly moving and exciting  time for LLMs+computer vision! We have worked to address your noted weaknesses and address your questions below. \n\nAlso re: relation to Generative Agents by Park et al 2023  came out on arXiv only 30 days before our work, so while it has some similarities we did not compare it head-to-head. The largest difference we see is it builds synthetic agents with code, while we work to analyze real-world video data directly (this is not a critique of that very cool work). Regarding the expense of the execution flow, since our symbolic pointer for long term memory is implemented using Python dictionary (i.e., hash table), the computational complexity for the lookup operation and insertion are O(1). In contrast, generative agents require LLM to calculate the embedding for every query and the embedding will later be used to retrieve relevant content in a vector database where the lookup operation likely takes O(Nd) where N is number of vector in the database and d is dimension of the vectors.\n\nTo address your noted **weaknesses**:\n\n- (1, 2) Novelty & Scope: We agree there are great synergies with ViperGPT and VisProg, but there are some differences and we do cite the work. The technical difference that you also highlight as a strength is our novel dual-memory system, which we do think is broader than animal behavior. Yet, given neurips is at the intersection of ML and neuroscience, we felt this was also the proper venue for our work, we hope you agree!\n\n- (3) Method; we don’t fully constrain the LLMs based only our API, and we wrote the API to be quite “general” for laboratory animal behavior – of course, that is still a specialist approach – but we do think it shows a first applied real-world use of LLMs and SOTA modules in computer vision and machine learning that are linked by our system. Directly, it can be used on other animals, and also to address another reviewer we show this on another species, and note that nothing is hard-coded about mice, the hard-coded nature is about spatio-temporal reasoning, which spans animals. We hope that clarifies the aim and approach. \n\n- (4) We developed a web-App and invited external users to test the system with GPT3.5 models, thereby its OOD and stress tested. \nWe analyzed the logs from 362 queries, 242 of which were from unique prompts (i.e., from manually typed ones rather than executing the demos). Out of the 362 queries, 329 were automatically rephrased. \n\nWe found AmadeusGPT had reported 129 “errors”: 38 were caused by unsupported features or undefined variables, and were thus explained to the user, while 64 originated from programming errors. Therefore, from 362 queries there were an 18% programming error rate, and 11% unsupported feature requests rate. \n\nThen, we took 10 of the failed runs from external users and re-ran them with GPT4. Now, of the 10 that previously failed, only 3 failed, and 4 correctly output clarifying queries to the user*:\n\n1: ['❌ Get angles and distances between all body parts. Plot a UMAP graph using the resulting data.',\n\n2:  '✅ Perform hierarchical clustering and plot the dots with different colors based on their clusters.',\n\n 3: '✅ Plot the distance between animals over time.',\n\n4:  '✅ What is the speed of the changes from the left to the right arm?',\n\n5:  '✅ Define 
 as the angle between the mouse_center and the head_midpoint. Plot the variation of 
 over time.',\n\n6:  '❌ Plot animal center of mass x-coordinate, velocity, acceleration, and head direction.',\n\n7;  '✅ Plot a bar graph with the first bar representing the total time the animal spends in ROI0 (open arm) and the second bar representing the total time the animal spends outside of ROI0 (closed arm).',\n\n8:  '✅ Plot the euclidean distance between the nose points of animal0 and animal2 over time.',\n\n9;  '❌ Define 
 as the orthogonal angle to the line between left_ear and right_ear.',\n\n10:  '✅ Plot each grooming bout.',\n\n*For (2), (4), (5), (10), the output is a message indicating either that the feature is not implemented or that the query does not provide all information, and explains what would be needed either in terms of development or clarifications from the user.\n\nThank you for this suggestion to stress-test further, and hope this overall response enables you to support our work! "}, {"review_id": "268gaYEyyt", "reviewer": "Reviewer_b8BG", "summary": "This paper presented Amadeus, a natural language interface for interactive animal behavior analysis. To accommodate modern LLM (GPT3.5) for behavior analysis, the authors proposed to use an API document to constrain GPT3.5’s knowledge space. Furthermore, the authors proposed Dual Memory Mechanism to read and write behaviors (symbols) to enable correct long-term memory retrieval and overcame the limit of 4096 tokens. The authors demonstrated results on three standard mouse behavior datasets. ", "questions": "(1) Can it extrapolate to behavioral codes that are never seen, and how are the atomic APIs distributed? I mean, the behaviors that could be handled seem to closely related to the capability of atomic API. If the atomic API could not measure animal overlap, would the system work with behavior like \"mount\" which is closely related to pixel/keypoint/area overlap? \n\n(2) What if define a symbol twice with slightly different or totally different descriptions? \n\n(3) Mistake: \nIn Figure.3, “Cmpute the pose from the video” -> “Compute …” \n", "limitations": "The authors have addressed limitations. ", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "(1) The motivation is quite novel. The whole system could provide an unprecedented experience of animal behavior analysis through natural language guidance only. This paper pioneers a practical way to integrate LLM to task programming. \n\n(2) The paper is well written. \n\n(3) The quantitative result on MABE dataset demonstrates the effectiveness of the system. \n", "weaknesses": "(1) Although the paper has claimed “animal behavioral analysis”, the datasets used are mouse only. I wonder whether it is difficult to transfer the experience of this paper to other animals, e.g. monkeys, zebrafish, etc. The authors should discuss its applicability to other animals. \n\n(2) I want to know how many behaviors have been tested on this model. It would be better to list them in the supplementary files. \n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Firstly we greatly thank you for your rating and recognition of the novel approach and performance of AmadeusGPT. Below we address your questions the best we can, and also agree it will be very exciting to see this deployed on more behaviors. \n\nTo address your noted **weaknesses**:\n\n- **(1)** There is nothing limiting the system to mice (i.e. core modules are SAM (generalist), animal pose with SuperAnimal (Ye et al. 2023 released weights that work on over 45 species therefore we used them), and CEBRA with also is a generalist new dimensionality reduction algorithm just published in Nature (Schneider et al 2023, code publicly released March 2023). We do agree perhaps the choice of only mice tasks was limiting, but these are the most common benchmarks. However, we have now tested another species for you. In particular, we provided a new demo with a horse video. We added local maxima detection and some atomic gait analysis and visualization functionalities, and demonstrated that with an addition of <100 lines of code AmadeusGPT can be asked, for example, to evaluate the symmetry of a horse’s stride (see PDF in overall rebuttal for the visual demonstration).\n\n- **(2)** The Sturman et al, benchmark data (EPM), the 9* MABe challenge tasks we show, and the MausHaus video that allows us to demo the SAM + animal pose estimation, and integrations with CEBRA. Of course in the future we aim to open this code up to users and have a web portal – there is nothing inherent in SAM, pose models, or CEBRA that limit this to mice, and we focused on spatio-temporal reasoning as from flies to macaques, interactions with conspecifics and objects are present (and see above). \n\n *-*We added 3 more tasks in response to another reviewer.*\n\n\nTo respond to **questions**:\n\n- For **question (1)**, we agree the way we handle behaviors seems to be closely related to the capability of atomic API. Though it should only augment the underlying LLMs. In the early development stage, we found that many behavior tasks cannot be solved by naive GPT-3.5, possibly due to the lack of high quality training data for behavior analysis. Therefore, we tried to and will cover more atomic APIs by having behaviorists try our software.\n\n- Also, as we briefly mentioned in the text, this is a practical design choice for extensively using those atomic API or combinations of them. This is due to the limited code generation ability of GPT-3.5 and its context window. Adding constraints using atomic APIs, we are able to (1)  reduce code hallucinations, (2) reduce token consumption of the LLMs,  (3) encapsulate lots of pre / post processing that are needed for handling noisy real-world data. While more powerful LLMs such as GPT-4  (context window much bigger) becomes more available and cheaper, we will gradually relax the constraints from atomic APIs in the future. We also note behavior analysis is also subject to domain knowledge and the available LLMs might not be sufficiently trained to know these domains. Therefore, some constraints of atomic APIs will still remain even in the future. To your more specific question about “mounting” and “overlapping”, we do not see any problem for AmadeusGPT to handle overlapping and mounting, as they can be easily captured by the  distance between animals. Though 3D keypoints and other kinematics should be needed to well distinguish two behaviors. However, imagine it could not handle overlap, then it probably wouldn’t work for mounting as mounting and overlapping will involve the generalization or specialization of the same set of atomic APIs.\n\n\n- For **question (2)**, we have a mixture strategy for long term memory, you are welcome to check section 3.3 and section 3.5 for more details. In short, for dynamic module loading, we rely on embedding of the query for storing and retrieving. In that case, a slightly different description might cause false retrieval. But since you are asking about  our symbolic pointer approach for long term memory,  defining a symbol twice is equivalent to overwriting an entry in Python’s dictionary. And whether the descriptions are similar or very different should not impact the retrieval as we used regular expressions to look for the symbol in <> and <||>. This way, the description is guaranteed to be retrieved correctly as long as the symbolic name is given correctly.\n\n- For **question (3)** We will fix the typo in the figure, apologies! Mistake: In Figure.3, “Cmpute the pose from the video” -> “Compute …”\n"}]

Author rebuttal:
Firstly we’d like to thank the reviewers, all who noted novel (or appreciable) advances with our use of LLM for behavior and our dual-memory system for LLMs:
- “Overall I found the contribution novel, of potential immediate utility to academic neuroscience labs performing behavioral analysis, and as an example for other groups hoping to automate analysis in other systems” - Reviewer XjKy
- “novel long-short-term memory module,” … “Enables the novel user experience of analyzing animal behavior using only natural language” - Reviewer ZigY
- “Amadeus proposes a novel dual-memory mechanism that combines short-term and long-term memory banks to effectively analyze extended contexts such as long videos” - Reviewer eSvp
- “The paper is open about closely related work, and assigns credit” … “I appreciate the Dual Memory Mechanisms proposed (including the illustrative example of limitations of short context windows)” … “The method performs above the state of the art, both qualitatively and quantitatively” - Reviewer f7B3
- “The motivation is quite novel. The whole system could provide an unprecedented experience of animal behavior analysis through natural language guidance only. This paper pioneers a practical way to integrate LLM to task programming.” - Reviewer b8BG

Secondly, we thank the reviewers for providing constructive feedback, which we address individually below. 
In summary, the major points are:
- we provided a new analysis of the system from external users (here anonymous, but from behavioral experts) 
- we added more behavioral examples beyond mice, i.e., we added horses, which is also attached as a new figure PDF 
- we added three additional MABe challenge tasks

Overall, we are thankful for the reviewers support, and we hope these additions will improve your impression of our new system.


Meta-review: The reviewers's sentiments are overall positive towards the paper. On the positive side, all the reviewers think the work is novel and addresses an interesting domain-specific problem (animal behavior analysis) with LLMs enhanced with long-and-short-term memory. Reviewer b8BG is the most enthusiastic, and commended that "The motivation is quite novel. The whole system could provide an unprecedented experience of animal behavior analysis through natural language guidance only. This paper pioneers a practical way to integrate LLM to task programming." Reviewer ZigY noted that (the work) "Enables the novel user experience of analyzing animal behaviour using only natural language, which could be an important affordance for domain experts". The reviewers also think the paper is well written. On the negative side, the reviewers have reservations with the generalizability, evaluation design and technical descriptions. Particularly, the work is only based on mouse data and its applicability to general animal behavior is not demonstrated. Nevertheless, the reviewers think the work makes a nice contribution to the problem.
Decision: Accept (poster)