Title: AmadeusGPT: a natural language interface for interactive animal behavioral analysis
Venue: NeurIPS 2023
Accepted: yes
Keywords: ChatGPT

GPT3.5

GPT4

behavioral analysis

LLMs

human-AI interaction

behavioral neuroscience
Abstract: The process of quantifying and analyzing animal behavior involves translating the naturally occurring descriptive language of their actions into machine-readable code. Yet, codifying behavior analysis is often challenging without deep understanding of animal behavior and technical machine learning knowledge. To limit this gap, we introduce AmadeusGPT: a natural language interface that turns natural language descriptions of behaviors into machine-executable code. Large-language models (LLMs) such as GPT3.5 and GPT4 allow for interactive language-based queries that are potentially well suited for making interactive behavior analysis. However, the comprehension capability of these LLMs is limited by the context window size, which prevents it from remembering distant conversations. To overcome the context window limitation, we implement a novel dual-memory mechanism to allow communication between short-term and long-term memory using symbols as context pointers for retrieval and saving. Concretely, users directly use language-based definitions of behavior and our augmented GPT develops code based on the core AmadeusGPT API, which contains machine learning, computer vision, spatio-temporal reasoning, and visualization modules. Users then can interactively refine results, and seamlessly add new behavioral modules as needed. We used the MABe 2022 behavior challenge tasks to benchmark AmadeusGPT and show excellent performance. Note, an end-user would not need to write any code to achieve this. Thus, collectively AmadeusGPT presents a novel way to merge deep biological knowledge, large-language models, and core computer vision modules into a more naturally intelligent system. Code and demos can be found at: https://github.com/AdaptiveMotorControlLab/AmadeusGPT

AmadeusGPT: a natural language interface for interactive animal behavioral analysis

Abstract

The process of quantifying and analyzing animal behavior involves translating the naturally occurring descriptive language of their actions into machine-readable code. Yet, codifying behavior analysis is often challenging without deep understanding of animal behavior and technical machine learning knowledge. To limit this gap, we introduce AmadeusGPT: a natural language interface that turns natural language descriptions of behaviors into machine-executable code. Large-language models (LLMs) such as GPT3.5 and GPT4 allow for interactive language-based queries that are potentially well suited for making interactive behavior analysis. However, the comprehension capability of these LLMs is limited by the context window size, which prevents it from remembering distant conversations. To overcome the context window limitation, we implement a novel dual-memory mechanism to allow communication between short-term and long-term memory using symbols as context pointers for retrieval and saving. Concretely, users directly use languagebased definitions of behavior and our augmented GPT develops code based on the core AmadeusGPT API, which contains machine learning, computer vision, spatio-temporal reasoning, and visualization modules. Users then can interactively refine results, and seamlessly add new behavioral modules as needed. We used the MABe 2022 behavior challenge tasks to benchmark AmadeusGPT and show excellent performance. Note, an end-user would not need to write any code to achieve this. Thus, collectively AmadeusGPT presents a novel way to merge deep biological knowledge, large-language models, and core computer vision modules into a more naturally intelligent system. Code and demos can be found at: https://github.com/AdaptiveMotorControlLab/AmadeusGPT .

1 Introduction

Efficiently describing and analyzing animal behavior offers valuable insights into their motivations and underlying neural circuits, making it a critical aspect of modern ethology, neuroscience, medicine, and technology [1, 2, 3, 4]. Yet behavior is complex, often multi-faceted, and context-dependent, making it challenging to quantify and analyze [5, 6]. The process of translating animal behavior into machine-readable code often involves handcrafted features, unsupervised pre-processing, or neural network training, which may not be intuitive to develop for life scientists.

To understand animal behavior one needs to complete a series of sub-tasks, such as obtaining animal poses and identities, object locations and segmentation masks, and then specifying events or actions the animal performs. Significant progress has been made in automating sub-tasks of behavioral analysis such as animal tracking [7, 8, 9], object segmentation [10, 11], and behavior classification [3, 12, 13, 14, 4], yet behavioral phenotyping requires additional analysis and reasoning [2, 15, 16, 13,

17, 8]. This is typically done with feature computations such as measuring time spent in regions of interest or general statistics of locomotion [13, 8, 18].

The challenge of making behavior analysis accessible and interpretable is hindered by the difficulty of combining task-specific models and the lack of an intuitive natural language interface to produce machine code. In an ideal scenario, a behavior analysis practitioner would be able to explore behavioral data, define their desired actions using natural language, and visualize captured behaviors without needing to learn how to train models, write scripts, or integrate code bases. Our framework, AmadeusGPT, takes the first step towards achieving this goal. AmadeusGPT provides a natural language interface that bridges users' prompts and behavioral modules designed for sub-tasks of behavior analysis. Our results show that AmadeusGPT outperforms machine learning-based behavior analysis classification tasks in the MABe [4, 19] benchmark by using prompts highly similar to their official definitions of each behavior, namely with small modifications and only three tunable parameters.

AmadeusGPT offers a novel human-computer interaction ap-

Query

"When is the mouse

on the treadmill?"

AmadeusGPT

SOpenAI

GPT3.5LLM

core behavior API

Generate Code based on

class AnimalBehaviorAnalysis:

CoreAPI&Integrations

Methods

def task_program():

behavior_analysis = AnimalBehaviorAnalysis()

get_object_names()->List[str]

object_names =['18']

get names of all objects

fig, ax =\

def animals_state_events(self,

behavior_analysis.plot_object_ethogram(object_names)

state_type,

comparison,

return fig, ax

['a11'],

bodyparts =

Python Interpreter

)：

pose estimation

object segmentation

8

SAM

use pretrained DLC model

use pretrained SAM model

Xoutput

"Result for query : 12,836 frames

the mouse is on the treadmill (total

video length is 107,800 frames)."

proach to behavior analysis, providing a unique user experience for those interested in exploring their behavioral data. Through natural language prompts, users can ask questions, define behaviors on-the-fly, and visualize the resulting analyses plus the language output (Figure 1). We show that with our novel dual-memory mechanism, defined behaviors are not lost (when exceeding the token limit), wording can be automatically rephrased for robustness, and the state of the application can be restored when relaunched, providing seamless and intuitive use (Figures 1-4).

To capture animal-environment states, AmadeusGPT leverages state-of-the-art pretrained models, such as SuperAnimals [17] for animal pose estimation and Segment-Anything (SAM) for object segmentation [11]. The platform enables spatio-temporal reasoning to parse the outputs of computer vision models into quantitative behavior analysis. Additionally, AmadeusGPT simplifies the integration of arbitrary behavioral modules, making it easier to combine tools for task-specific models and interface with machine code.

2 Related Work

Large Language Models. In recent years there has been a surge in the development of large language models (LLMs) [20, 21, 22, 23] that show exceptional abilities in natural language processing tasks such as language generation, interaction, and reasoning. One notable example is ChatGPT, which is built on GPT-3+ and trained on a massive corpus of text data [22]. ChatGPT has demonstrated impressive performance in a wide range of natural language processing tasks, including text classification, language modeling, and question-answering. In addition to language processing, LLMs have

also been applied to other domains such as image and video processing [24], audio processing, and natural language generation for multi-modal inputs [24, 25].

LLM Integrations. Recent works leverage LLMs to generate code for computers to execute, including visual programming with VisProg [26] and ViperGPT [27], and HuggingGPT [28] that exploits pre-trained AI models and smartly handle API-GPT interactions [26, 27]. However, animal behavior analysis lacks a natural language-computer vision pipeline and requires diverse open-source code-base to complete the sub-tasks. Concurrent work by Park et al. [29] on Generative Agents uses a dual-memory system and iterative chatting, but has a more expensive execution flow compared to our approach, which uses a memory mechanism with symbols as pointers.

Behavioral Analysis. Machine learning and computer vision techniques have increasingly been employed in animal behavior analysis in recent years [7, 9, 12, 30, 31, 32]. These techniques offer advantages such as automation, high throughput, and objectivity, compared to traditional methods that rely on manual observation and annotation [1]. Deep learning models, such as convolutional neural networks (CNNs) and transformers, have been utilized for feature extraction and classification of animal behaviors in various domains such as social behavior, locomotion, and posture analysis (reviewed in [2]). Yet, universal interfaces to the plethora of pose estimation tools and downstream analysis methods are lacking.

We used a rule-based approach that leveraged the pose estimation outputs to compute behaviors, similar to the approach used in LiveMouseTracker (LMT) [33]. However, unlike LMT, which requires users to interact with code and requires a specific hardware to collect data, AmadeusGPT is agnostic to data collection and provides a natural language interface that is easy to use and customize. Additionally, unlike LMT, AmadeusGPT includes object segmentation, enabling it to capture animal-object interactions.

Task Programming for Behavior. Task programming for behavior analysis [30] aims to leverage the domain knowledge from experts to help extract the most relevant features that are useful for the downstream behavior classification task. However, task programs were only considered as Python functions that help extract better features (i.e., features built from poses). In this work, we generalize the scope of task programming: any sub-task that can be achieved as machine-executable Python code is considered task programs in AmadeusGPT, and can be queried and developed with natural language. Thus, we name the generated function in AmadeusGPT (target) task programs.

3 AmadeusGPT

Long termmemory

Datasetloader

Shorttermmemory

Memory

Visualization

SAM

Errorhandling

Embeddings

SuperAnimal

Integration

Robustnesstesting

Reflection LLM

QAmadeusGPT

modules

Matplotlibpolygons

Animal&objects

Rephraser

Core modules

Datafiltering

Selfcorrection

Kinematics

Behavior LLM

Bodypartkinematics

Dynamicmodule

matching

Spatio-temporal

reasoning

behavior capturing

Rule-based

Embedding LLM

AmadeusGPT is an interactive human-computer platform that allows users to describe in natural language the behavioral analysis they want to be performed. It utilizes ChatGPT as the user-guided controller and a range of machine learning and computer vision models as collaborative executors to analyze animal behavior, starting from raw video and leveraging new pretrained pose estimation models that can run inference across species and settings [17], and powerful objection segmentation models (SAM [11]) (Figure 2). Here we focus primarily on mice, but also show it works for horses to demonstrate that nothing precludes AmadeusGPT to be used on a range of animals.

AmadeusGPT uses LLMs to generate Python executable code that fulfills user-specified queries in the prompt. Building such a system requires LLMs to learn to manipulate core process resources in a constrained way. If the user's prompt is unclear or beyond the system's capacity, the generated code might result in errors that require programming expertise. Intuitive error messages are therefore essential for ensuring a consistent natural language experience, while maintaining the ability for users

(a)Short questions leading to active refinement and decomposition into to multipletaskprograms

Queries

AmadeusGPT

Short-term

long-term

runs computer vision models

memory

memory

toextractposeandobjects

Query 1

"When is the mouse

on the treadmill?"

User specifies Object 18 is

LLM

Generated code

LLM

the "treadmill"

Query 2

X Chat Output:

"Plot the

corresponding

Result for query:12,836 frames

object ethogram"

the mouse is onthe treadmill

Query3

"plot the

API DoCS

Python Interpreter

Error

frames).

(total video length is 107,999

API Implementation

trajectory of the

center of animal"

X Visual Outputs:

X Code Outputs:

For query 1:

def task_program():

behavior_analysis = AnimalBehaviorAnalysis()

object_18_events =\

(.e.]=sxedpoq ',dexaAo,

object_18_mask=\

Event.events2onemask(object_18_events)

frames_on_object_18 = np.sum(object_18_mask)

return frames_on_object_18

For query 2: def task_program():

behavior_analysis = AnimalBehaviorAnalysis()

For query 2:

fig, ax = behavior_analysis.plot_object_ethogram(object_names)

object_names =['18']

Ethogram for Object + Animal overlap

return fig, ax

not defined in the data.Check if "center"is a valid

bodypart for the animal in the data.2.If "center"is not a

valid bodypart，try using a different bodypart that is

supported by thedata.3.Once you have identifieda valid

0

2000 4000 6000 8000 100001200014000

bodypart，update the task_program(）function touse that

frame number

bodypart instead of "center".

(b) Long description decomposed to multipletaskprograms

Featureextraction

Behavior definition

Visualization

Query

"Get the pose from the video and define <|head dips|> as a behavior where animal's mouse_center and neck roil. Then print the count of how many head dips are in the roio and its head_midpoint is outside the happens. At the end, plot the trajectory of the animal's head_midpoint when head_dips happens."

Cmputetheposefrom

thevideo

"Define </head dips|>

animal mouse_center

asabehaviorwhere

"Plottrajectory"

Preprocessing

"in the roi"

and its head_midpoint

andneck are in theroi

is outside the roi"

Behavior counting

manyhead dips"

"Count of how

X Code Outputs:

def task_program():

behavior_analysis = AnimalBehaviorAnalysis()

#get events where head dips happens

ROI1

head_dips_events =\

behavior_analysis.animals_object_events('Ro1o',\

xau,xuso,]=sxedpoqdexo,

negate=False)

head_outside_roi_events =\

behavior_analysis.animals_object_events('RoI1',\

'overlap'，bodyparts=['head_midpoint']，\

negate=True)

head_dips_events =\

Event.add_simultaneous_events(head_dips_events,\

head_outside_roi_events)

# count how many times head dips happens

Interactive:

Run pose estimation

Plot trajectory of

count_head_dips =\

Drawing ROIs

SuperAnimal model

sdp peay

Event.count_events(head_dips_events)

#plot the trajectory of head dips

matpl&tlib

fig, ax =\

★= start of head dip bout

behavior_analysis.plot_trajectory(\

X = end of head dip bout

to iteratively refine generated code with language. Therefore, to build AmadeusGPT we leveraged GPT3.5, augmented it, developed a dual-memory system and core behavioral utilities that together make AmadeusGPT an end-to-end, human language-AI system (Figure 2).

3.1 LLMs as natural interfaces for code development and execution

ChatGPT (i.e., with GPT3.5 or 4) [23] is able to generate code that corresponds to users' prompts due to instruction tuning paired with access to a large amount of code in its training data. However, the native ChatGPT/GPT3.5 or 4 API cannot serve as the language interface for behavior analysis for the following reasons: (1) it cannot work with private APIs that are not in its training data; (2) it

hallucinates functions if too many implementation details are exposed or asked to be provided; (3) the context window does not have a sufficient capacity size for complex tasks that require reading large source code files; (4) it has no Python interpreter to execute the code it suggests. Therefore, we built an augmented version of GPT to overcome these limitations (see Sections 3.2 and 3.3).

3.2 Augmented-GPT and API design

AmadeusGPT sends its API documentation to ChatGPT in order to smartly constrain the output code and immediate Python execution and thereby always augments the data available to GPT3.5 or GPT4. In the API documentation, implementation details are encapsulated and only the function documentation is exposed (see Appendix for example API docs). Importantly, there is an 'explanation prompt" with each function example that serves as a hint for GPT3.5 to understand what the API is for. This separation of API documentation and implementation has two advantages: it reduces token use, and prevents hallucinating resources that do not exist in the core functions or modules linked to AmadeusGPT. Concretely, we prompt GPT3.5 to strictly follow our API (see Appendix). This design improves the reliability of code generation while imposing quality control.

We followed three core principles when designing the API. Firstly, we developed a code with an atomic API that consists of simple functions designed to perform specific operations (Figure 2). These functions are modular, reusable, and composable, making them easy to combine with other API functions to create more complex functionality. The use of atomic API improves the readability of the code for users and developers and reduces the chance of LLMs confusing ambiguous API calls. Secondly, we leverage polymorphism in code design to generalize our API to variants of a sub-task routed by parameter values and input types since it is not desirable nor realistic to provide an example code for every possible sub-task. Thirdly, to make AmadeusGPT cover a range of behavioral analysis tasks, we identify core behavioral modules that cover common behavior analysis sub-tasks and additionally use integration behavioral modules for task-specific sub-tasks.

3.3 Dual Memory Mechanism

As GPT3.5 is implemented with a transformer architecture that has a context window size limitation of 4,096 tokens for hosting the history of conversations [22]. This would limit how far back AmadeusGPT can use the context, such as a user-defined behavior or its own generated code. We demonstrate in Figure 4 that without tackling the context window limitation, forgetting will happen, which results in wrong code that hallucinates functions or variables that do not exist.

To tackle this limitation, we implemented a dual-memory mechanism that consists of both short-term memory and long-term memory banks. In this work, short-term memory is implemented as a dynamic deque that holds the chat history, pulling in new tokens and removing older ones when full (i.e., beyond the 4,096 tokens). At inference time, all text in the deque is sent to the ChatGPT API call.

Long-term memory is implemented as a dictionary in RAM. The keys of such a dictionary are called symbols in this work. We define read and write symbols to instruct the communication between short-term memory and long-term memory. With the help of regular expression, a symbol that is enclosed by < || > triggers a memory writing operation that writes the context of the corresponding chat into the dictionary using the symbol name as the key. As in Figure 3, keyword < | head dips | > triggers memory writing that stores the whole sentence under the key name 'head dips". Similarly, <> triggers a reading operation that pushes the stored sentence into the front of short-term memory with an additional prompt to remind GPT3.5 that the retrieved memory is used for context instead of a formal instruction. This allows a read operation retrieval from the long-term memory to be used as a context for a new instruction. Additionally, with the long-term memory, AmadeusGPT can store its states into disk and the user can restore the states after re-launching the application (See Appendix). Collectively, we call this augmented-GPT3.5.

3.4 Core Behavioral Modules

The language query feature of ChatGPT inspired us to provide a more natural way for humans to perform behavior analysis. Namely, it provides users with a platform to perform behavior analysis by asking questions or instructing tasks in an interactive manner. We imagine that users will either ask a question or provide longer instructions to complete a task (Figure 3). In addition, if users ask followup questions to a previous answer from AmadeusGPT, it attempts to answer with executable code

(a) Memory operations

Copyingfrom short-term

Retrievalfromlongtermmemory

Activeforgetting in short-termmemory

Dynamicallyloadingintegrationmodules

memoryto long-termmemory

<contact>

4096

Compute 3D embedding using cebra

Write symbol

initialization

are less than 1 cm apart

behaviorwhere twoanimals

Define 
 as a

Text embedding

Short-term memory

Long-term memory

<contact>

<contact>

Garbage collection

0.35

</contact|>o

<|contactl>

0.95

Long termmemory

Shorttermmemory

<ll>

Write Symbol

text history

text embedding

implemented by

implemented by

<>

Read symbol

process memory

dequeue andcontext

or hard disk

window of GPT3.5

(b) Short term memory retrieval

(d) Long term memory retrieval

called 'task programs" that are executed by the backend Python interpreter. In the API documentation we mostly specify code examples for uni-purpose task programs (such as counting events). However, we show in Figure 3 that our augmented-GPT3.5 API is able to compose multi-purpose task programs (such as computing events and interactions with objects over time to produce plots).

Figure 3b shows how the user can give a complex instruction that covers multiple sub-tasks, including pose extraction, behavioral definitions, interactively drawing regions of interest (ROIs), then visualizing and performing tasks, such as behavior event counting. In this case, AmadeusGPT is able to decompose the description into multiple task programs and assemble the final program. Alternatively, the user can also ask a question 'When is the mouse on the treadmill?' or 'Count the number of head dips per ROI' as follow-up queries to AmadeusGPT's answer, or change the color map of the previous plot, etc. This all relies on the core behavioral modules and their ability to be combinatorically used (Figure 2).

Our core behavioral modules try to cover the most common behavioral sub-tasks. Generally speaking, a typical behavior analysis task asks question about what animals do through a spatio-temporal space. This can simply be the amount of time that an animal spends moving in a particular ROI, to more advanced analysis like measuring animal-animal or animal-object interactions. This requires varying levels of key-point tracking, object segmentation, and video (temporal) reasoning. Thus, we implemented the following core modules and always send them with the queries during augmentedGPT API calls (Figure 2).

-Kinematic feature analysis. As many behaviors are concerned with the kinematics of bodyparts, we provide a module for kinematics analysis. This includes filtering the pose data, calculating speed, velocity, and acceleration. It also includes movement-based statistical analysis, such as queries related to computing the amount of time spent locomoting (i.e., a velocity over some minimal threshold computed across the video, or time spent in an ROI).

-Animal-object environment states. To capture animals' and environment states, we deploy pretrained computer vision models such as pose estimation with SuperAnimal models [17, 7] and objects with SAM [11]. Note they can be easily substituted by tailored supervised models. To support end-to-end behavior analysis with raw video only, we use them as the default models for their wide coverage of animals and objects. We also implemented code infrastructure that abstracts 'static objects' and 'moving objects' to represent objects in the environment as well as customized ROI objects and animals respectively. The animal-object relation and animal-animal relation are modeled and saved in lookup tables. These relations mostly cover binary spatial relations such as 'to the left', 'to the right', 'overlap', 'orientation', and numerical spatial relations such as 'distance', 'angle' (i.e., the angle between animals, based on computed neck-to-tailbase body axes) and 'gazing angle' (i.e., the angle between animals' head coordinate systems, determined from the nose-neck line) (see Appendix).

-Spatio-temporal reasoning. After computer vision models track animals and segment objects, their outputs are used to build internal spatio-temporal relation tables among animals-animals and animalsobjects. We provide code infrastructures to support queries of events (i.e., sequential spatio-temporal relations and simultaneous spatio-temporal relations). Users can query events where animals move from one state to the other (see also Figure 5b).

3.5 Integrations Beyond Core Behavioral Modules

Integration modules aim to cover task-specific behavior analysis sub-tasks such as dataset loading, embedding calculations, and visualization tools. Because they are task-specific, we do not send the API documentation of these modules. Instead, we rely on 'dynamic loading' to load only few of the selected API documents for integration modules at inference time. To allow for dynamic loading, we use the embedding API from OpenAI to turn API documents of integration modules into text embedding vectors and cache them in RAM or disk. The user's prompt then acts as a query to retrieve k (a hyperparameter) most relevant integration modules by cosine similarity (Figure 4). We also allow users to manually load integration modules by prompting 'loading module \module-path' if users want to save the cost of using the embedding API and/or they are creating their own modules.

Error handling. We include error handling to help users understand when instructions are beyond the system's capabilities or ambiguous. Here, AmadeusGPT forwards prompts, error messages, and API docs to the ChatGPT API for natural language explanations (Figure 3a, Query 3 example).

Rephraser. Users can ask questions with language that might be very different from those in our API docs, which can cause performance degradation due to out-of-distribution prompts (See Section 4). To overcome this we leverage a native GPT3.5 (i.e., without our augmentation) that we call 'Rephraser' that is tasked to turn users' expressions into a style that is similar to that found in our API docs. We wrote in the system prompt of Rephraser a few examples of such rephrasing, hoping the GPT3.5 can learn it via few-shot learning. Therefore, Rephraser is tasked to act as test-time domain adaptation component for our system.

Self-correction. There are incidences where ChatGPT API makes obvious mistakes that cause runtime errors, therefore we implemented a self-correction mechanism to help ameliorate this. When there is an error executing the generated python code, we forward the error message and the output of our error handler to an independent ChatGPT API connection with a system prompt that encourages it to revise the function. The number of times for such retrying is flexible, but setting this to three in practice generally improves the success rates (See Appendix for an example). We keep the history of retries in the context window to help it to learn from failures if it takes more than one trial to correct the code. Note that ChatGPT API does not have read/write access to the implementation of our APIs thus self-correction cannot correctly revise the code if the errors stem from our code.

Explainer module. AmadeusGPT takes users' queries and generates Python code to address the queries. The executed function returns results of multiple data types, including plots, strings, numpy arrays, etc. However, it might not always be straightforward for users to link those returned results to the queries. Moreover, the users do not know how much they can trust the returned results. In many cases, checking the generated code can help. However, inspecting the code requires python programming knowledge. Therefore we add and explainer module, which is another LLM whose job is to explain to the users how to link the results to the queries. The explainer is implemented as an independent ChatGPT API connection with its independent system prompt. In the system prompt, we ask the explainer to take the thought process parsed from the code generator, the return values

(a) EPM

X Code Outputs 1:

(a.1) EPM: GT vs. x

L: Query 1:

def task_program():

100

behavior_analysis = AnimalBehaviorAnalysis()

"How much time does the mouse spend in the closed arm which is Rolo?"

in_ROIo_events =\

80

behavior_analysis.animals_object_events(\

ROI0

negate=False)

'ROIo','overlap'，bodyparts=['all']，\

Counts

60

return in_Rolo_arm_events

40

X Chat Output 1:

20

Results for query:12376 frames (total videolengthis15076frames)

0

2000

4000

6000

8000

10000

12000

14000

frame number

: Query 2:

X Code Outputs 2:

"How much time does the mouse spend outside the closed arm?"

def task_program():

behavior_analysis = AnimalBehaviorAnalysis()

outside_RoIo_events =

behavior_analysis.animals_object_events('RoIo'，'overlap'，\

X Chat Output 2:

bodyparts=['all'],negate=True)

return outside_Rolo_events

Results for query : 2700 frames (total video length is 15076 frames)

(b) MausHaus

0

2000

4000

6000

8000

10000

12000

14000

frame number

:Query:

1:Query:

:Query:

"When is the animal on the treadmill，which is object 5?"

"Plot the trajectory of the animal."

"When is the animal close to the object 35，if I define close as less than 50 pixels?"

time

def task_program():

behavior_analysis = AnimalBehaviorAnalysis()

overlap_object_5_events = behavior_analysis.animals_object_events('5'，'overlap',bodyparts=['all'])

fig, ax = behavior_analysis.plot_trajectory(bodyparts=['all']，events=overlap_object_5_events)

return fig, ax

(c) MABE Challenge dataset

Official definition

Our prompt

Chase

Watching

Chase:Mice are moving above 15 cm/sec, with closest points less than 5 cm apart, less than 30 degrees,for at least 80% of and angular deviationbetweenmice is frames within at least one second.Merge bouts less than 0.5 seconds apart.

"Define 
 as a social behavior where closest distance between this 40 pixels and the angle between this animal and other animals is less than and other animals have to be less than 30 and this animal has to travel faster than 2 pixels.When do chases happen?"

Watching:Mice aremore than5cm apart but less than 20 cm apart, and gaze offset of onemouse is less than 15 degrees from body of other mouse,fora minimum duration of 3 seconds.Mergebouts less than 0.5 seconds apart.

"Define 
 as a social behaviorwheredistancebetweenanimals is less than 260 pixels and larger than 50 and head angle between animals is less than 15. The smooth_window_size is 15.when does watching happen?"

Table 1: F1 scores by AmadeusGPT on several tasks from the MABe Behavior Challenge 2022 [4]. We do not use representation learning (the aim of the benchmark), only rule-based task programming. *T9: oral-ear-contact; T10: oral-genital-contact; T11: oral-oral-contact.

from python as well as the user queries to explain whether the code and its return values meet the queries (See Appendix for an example).

4 Experiments

To evaluate AmadeusGPT we ran a series of qualitative and quantitative experiments. We used three datasets that highlight standard behavioral neuroscience settings. The first is an open-access Elevated Plus Maze (EPM) dataset from Sturman et al. [13]. The second is called MausHaus and is a video of a mouse for one hour (108K frames) in an enriched home-cage setting [17]. The third is video data of three mice interacting from the MABe 2022 Challenge [4]. We also include a fourth in-the-wild horse dataset [36].

EPM. The EPM is a classical task in neuroscience research to test an animal's anxiety about being in exposed 'open arms" vs. 'closed arms" [13]. We show that with minimal prompts and ROI interactive plotting a user can measure these events (Figure 5a, also see Figure 3). Here, we query AmadeusGPT to report both open and closed arms and note that the resulting raster plots (ethograms) do not identify the same frames, as one expects if it is correct (i.e., the mouse cannot be in both states at once). We also show that AmadeusGPT counts the number of events similar to those reported in ground truth annotations by three raters across five different videos (Figure 5a.1).

MausHaus: enriched mouse-cage monitoring. Studying more natural behavior is becoming important in research settings. Here, we demonstrate intuitive prompts that run pretrained models (SuperAnimal-TopViewMouse and SAM) then query spatio-temporal relationships between the model outputs (Figure 5b). As SAM only provides object labels, we use object number or interactive clicking on the object to ground the analysis (see also Figure 3).

Horse gait analysis. AmadeusGPT has no innate mouse preference. Therefore to show another application we performed equine gait analysis [36]. We leveraged horse videos with ground truth keypoint annotations on every frame and used SuperAnimal-Quadruped [17], and found comparable results within AmadeusGPT to human-level labeling and stride analysis (see Appendix Figure 8).

MABe 2022 Benchmark. The benchmark had two rounds for the three mouse dataset. We used the more popular first round (evaluation split) and therefore provided the pre-computed keypoints as inputs to AmadeusGPT. In Figure 5c, we show how AmadeusGPT captures two representative tasks from MABe benchmark, Chase and Watching. We use text that is close to the original definition to define the behaviors we want to capture. Note the units in our prompt are pixels for distance, radians for degree and pixel per frame for speed. We tested nine behaviors and report the F1 score as computed in the MABe evaluation (Table 1). Our approach is purely rule-based, thus no machine learning is needed and only three parameters are needed to be given or tuned: a smoothing window size for merging neighboring bouts, a minimal window size for dropping short events, and the pixel per centimeter. Note that tasks that are hard for machine learning models are also hard for our rule-based approach (Table 1, Tasks 9-11).

We do not intend to formally claim state-of-the-art on the benchmark, as the goal was to evaluate unsupervised representational learning models. Nevertheless, we show that in practice one can use a text definition of behavior to capture behaviors that are on par with, or better than, the competitive representation learning approaches.

Robustness tests and stress-testing AmadeusGPT. Users might query AmadeusGPT with expressions that are very different from the explanation text we provided in the API documentation. A potential pitfall is that AmadeusGPT overfits to our (the developers) expressions and biases. To test how robust AmadeusGPT is, we crafted five out-of-distribution (OOD) base questions for an EPM video. Then we asked a native GPT3.5 (part of our reflection module, see Figure 2) to generate five variants of each OOD question. We manually checked whether AmadeusGPT generated consistently correct results on those 25 generated questions, and it did 88% of the time with our Rephraser module vs. 32% without the Rephraser. The total number of tokens consumed using the ChatGPT-API was 4,322 and 5,002 with and without using Rephraser, respectively. Note that in both cases, the consumed number of tokens is larger than the maximal context window size of 4,096. This also shows that AmadeusGPT passes the stress-test we set for it in two key ways: (1) The short-term memory deque correctly maintains the constrained size without getting an error from the OpenAI API due to maximal token size error; (2) The diverse questions in the short-term memory do not result in mode collapse or severe performance degradation.

In additional testing with naive users we found that better LLMs boost performance. In brief, we sampled 30 naive user prompts at random out of 362 submitted via an App we developed, and found an 18% error rate with GPT3.5 yet with GPT4 [38] only 10%). See the Appendix for examples.

5 Discussion

AmadeusGPT is a novel system for interactive behavior analysis. By providing a user-friendly interface and leveraging the power of language models, AmadeusGPT enables researchers to more efficiently and effectively study animal behavior. We believe this system offers a significant contribution to the field of behavioral analysis. The natural language interface of AmadeusGPT empowers non-experts to conduct advanced behavioral analysis with cutting-edge computer vision (such as with SAM [11] and SuperAnimal [17]), but its reliance on LLMs raises potential ethical concerns such as bias amplification. The use of a custom-designed API module in AmadeusGPT helps limit potential biases in outputs, but future work should consider how integration modules can introduce biases [39]. It also allows for domain-specific human-AI interactions [40, 41].

Our proposed natural language interface with the augmented-GPT3.5 shows promise, but there are limitations that need to be addressed in future work. These include enhancing robustness by reducing bias in prompt writing, enabling more goal-driven code reasoning to make AmadeusGPT more collaborative, and improving generalization by extending the available APIs and preventing ChatGPT from writing incorrect code. We also only support the English language, but multi-lingual support with be explored in the future. Collectively, we believe AmadeusGPT promises to open new avenues for both leveraging the utility of, and developing on, cutting-edge computer vision and machine learning tools by using intuitive language.

It is also worth noting that AmadeusGPT leverages ChatGPT API calls, which means we do not have access to the weights of the underlying LLMs thus we cannot yet fine-tune or augment the LLMs. Therefore, our dual-memory leverages the process memory and disks to augment the language model, in contrast to works that augment LLMs by assuming the access to the LLMs [42, 43]. Similarly, without fine-tuning the underlying LLMs, we constrain the behaviors of the LLMs only via in-context learning for both the code generator and rephraser. While recent models such as GPT4 [38], CLAUDE [44] continue to make context window bigger and in-context learning more powerful, deploying them can be slower and more expensive and a strategy of combining a smaller model and a bigger model is worth exploring [45]. In the future, it would be interesting to compare our in-process-memory with in-context memory in terms of reliability and cost. It is also interesting to compare fine-tuned LLMs vs. LLMs that are constrained via in-context learning for future works that use natural language interface for behavior analysis.

Of course, AmadeusGPT is not limited to using GPT3.5 or GPT-4, and new models such as CLAUDE [44] or others could be used. However, the larger the model the slower the response time and the higher the computational cost. Thus, while these excellent works have started to overcome token limits, there are still limits such that our computationally efficient dual-memory system will be of broad interest to those developing both domain-specific solutions (such as AmadeusGPT) and generalist models (the base GPT models). Collectively, we believe AmadeusGPT promises to open new avenues for both leveraging the utility of, and developing on, cutting-edge computer vision and machine learning tools with minimal to no coding - namely, by only using intuitive language.

References

David J Anderson and Pietro Perona. Toward a science of computational ethology. Neuron , 84(1):18-31, 2014.

Mackenzie Weygandt Mathis and Alexander Mathis. Deep learning tools for the measurement of animal behavior in neuroscience. Current opinion in neurobiology , 60:1-11, 2020.

Mayank Kabra, Alice Robie, Marta Rivera-Alba, Steve Branson, and Kristin Branson. Jaaba: interactive machine learning for automatic annotation of animal behavior. Nature Methods , 10:64-67, 2013.

Jennifer J Sun, Andrew Ulmer, Dipam Chakraborty, Brian Geuther, Edward Hayes, Heng Jia, Vivek Kumar, Zachary Partridge, Alice Robie, Catherine E Schretter, et al. The mabe22 benchmarks for representation learning of multi-agent behavior. arXiv preprint arXiv:2207.10553 , 2022.

Konrad Lorenz. The foundations of ethology . Springer verlag, 1981.

Patrick C. Friman. Cooper, heron, and heward's applied behavior analysis (2nd ed.): Checkered flag for students and professors, yellow flag for the field. Journal of Applied Behavior Analysis , 43:161-174, 2010.

Alexander Mathis, Pranav Mamidanna, Kevin M Cury, Taiga Abe, Venkatesh N Murthy, Mackenzie Weygandt Mathis, and Matthias Bethge. Deeplabcut: markerless pose estimation of user-defined body parts with deep learning. Nature neuroscience , 21(9):1281-1289, 2018.

Fabrice de Chaumont, Elodie Ey, Nicolas Torquet, Thibault Lagache, Stéphane Dallongeville, Albane Imbert, Thierry Legou, Anne-Marie Le Sourd, Philippe Faure, Thomas Bourgeron, and Jean-Christophe Olivo-Marin. Live mouse tracker: real-time behavioral analysis of groups of mice. bioRxiv , 2018.

Talmo D Pereira, Diego E Aldarondo, Lindsay Willmore, Mikhail Kislin, Samuel S-H Wang, Mala Murthy, and Joshua W Shaevitz. Fast animal pose estimation using deep neural networks. Nature methods , 16(1):117-125, 2019.

Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In Proceedings of the IEEE international conference on computer vision , pages 2961-2969, 2017.

Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan-Yen Lo, Piotr Dollár, and Ross Girshick. Segment anything. arXiv:2304.02643 , 2023.

James P Bohnslav, Nivanthika K Wimalasena, Kelsey J Clausing, Yu Y Dai, David A Yarmolinsky, Tomás Cruz, Adam D Kashlan, M Eugenia Chiappe, Lauren L Orefice, Clifford J Woolf, et al. Deepethogram, a machine learning pipeline for supervised behavior classification from raw pixels. Elife , 10:e63377, 2021.

Oliver Sturman, Lukas von Ziegler, Christa Schläppi, Furkan Akyol, Mattia Privitera, Daria Slominski, Christina Grimm, Laetitia Thieren, Valerio Zerbi, Benjamin Grewe, et al. Deep learning-based behavioral analysis reaches human accuracy and is capable of outperforming commercial solutions. Neuropsychopharmacology , 45(11):1942-1952, 2020.

Simon RO Nilsson, Nastacia L Goodwin, Jia Jie Choong, Sophia Hwang, Hayden R Wright, Zane C Norville, Xiaoyu Tong, Dayu Lin, Brandon S Bentzley, Neir Eshel, et al. Simple behavioral analysis (simba)-an open source toolkit for computer classification of complex social behaviors in experimental animals. BioRxiv , pages 2020-04, 2020.

Markus Marks, Qiuhan Jin, Oliver Sturman, Lukas M. von Ziegler, Sepp Kollmorgen, Wolfger von der Behrens, Valerio Mante, Johannes Bohacek, and Mehmet Fatih Yanik. Deep-learning based identification, tracking, pose estimation, and behavior classification of interacting primates and mice in complex environments. Nature machine intelligence , 4:331 - 340, 2020.

Cristina Segalin, Jalani Williams, Tomomi Karigo, May Hui, Moriel Zelikowsky, Jennifer J Sun, Pietro Perona, David J Anderson, and Ann Kennedy. The mouse action recognition system (mars) software pipeline for automated analysis of social behaviors in mice. Elife , 10, 2021.

Shaokai Ye, Anastasiia Filippova, Jessy Lauer, Maxime Vidal, Steffen Schneider, Tian Qiu, Alexander Mathis, and Mackenzie Weygandt Mathis. Superanimal pretrained pose estimation models for behavioral analysis. arXiv preprint arXiv:2203.07436 , 2023.

Zachary T. Pennington, Zhe Dong, Yu Feng, Lauren M. Vetere, Lucia Page-Harley, Tristan Shuman, and Denise J. Cai. eztrack: An open-source video analysis pipeline for the investigation of animal behavior. Scientific Reports , 9, 2019.

Mehdi Azabou, Michael Mendelson, Maks Sorokin, Shantanu Thakoor, Nauman Ahad, Carolina Urzay, and Eva L Dyer. Learning behavior representations through multi-timescale bootstrapping. arXiv preprint arXiv:2206.07041 , 2022.

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 , 2018.

Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. The Journal of Machine Learning Research , 21(1):5485-5551, 2020.

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems , 33:1877-1901, 2020.

Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems , 35:27730-27744, 2022.

Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances in Neural Information Processing Systems , 35:23716-23736, 2022.

Yaru Hao, Haoyu Song, Li Dong, Shaohan Huang, Zewen Chi, Wenhui Wang, Shuming Ma, and Furu Wei. Language models are general-purpose interfaces. arXiv preprint arXiv:2206.06336 , 2022.

Tanmay Gupta and Aniruddha Kembhavi. Visual programming: Compositional visual reasoning without training. arXiv preprint arXiv:2211.11559 , 2022.

Dídac Surís, Sachit Menon, and Carl Vondrick. Vipergpt: Visual inference via python execution for reasoning. arXiv preprint arXiv:2303.08128 , 2023.

Yongliang Shen, Kaitao Song, Xu Tan, Dongsheng Li, Weiming Lu, and Yueting Zhuang. Hugginggpt: Solving ai tasks with chatgpt and its friends in huggingface. arXiv preprint arXiv:2303.17580 , 2023.

Joon Sung Park, Joseph C O'Brien, Carrie J Cai, Meredith Ringel Morris, Percy Liang, and Michael S Bernstein. Generative agents: Interactive simulacra of human behavior. arXiv preprint arXiv:2304.03442 , 2023.

Jennifer J Sun, Ann Kennedy, Eric Zhan, David J Anderson, Yisong Yue, and Pietro Perona. Task programming: Learning data efficient behavior representations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 2876-2885, 2021.

Kevin Luxem, Petra Mocellin, Falko Fuhrmann, Johannes Kürsch, Stefan Remy, and Pavol Bauer. Identifying behavioral structure from deep variational embeddings of animal motion. Communications Biology , 5, 2020.

Eleanor Batty, Matthew R Whiteway, Shreya Saxena, Dan Biderman, Taiga Abe, Simon Musall, Winthrop F. Gillis, Jeffrey E. Markowitz, Anne K. Churchland, John P. Cunningham, Sandeep Robert Datta, Scott W. Linderman, and Liam Paninski. Behavenet: nonlinear embedding and bayesian neural decoding of behavioral videos. In Neural Information Processing Systems , 2019.

Fabrice De Chaumont, Elodie Ey, Nicolas Torquet, Thibault Lagache, Stéphane Dallongeville, Albane Imbert, Thierry Legou, Anne-Marie Le Sourd, Philippe Faure, Thomas Bourgeron, et al. Real-time analysis of the behaviour of groups of mice via a depth-sensing camera and machine learning. Nature biomedical engineering , 3(11):930-942, 2019.

Leland McInnes and John Healy. Umap: Uniform manifold approximation and projection for dimension reduction. ArXiv , abs/1802.03426, 2018.

Steffen Schneider, Jin Hwa Lee, and Mackenzie Weygandt Mathis. Learnable latent embeddings for joint behavioural and neural analysis. Nature , May 2023.

Alexander Mathis, Thomas Biasi, Steffen Schneider, Mert Yuksekgonul, Byron Rogers, Matthias Bethge, and Mackenzie W Mathis. Pretraining boosts out-of-domain robustness for pose estimation. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision , pages 1859-1868, 2021.

Mehdi Azabou, Michael Mendelson, Maks Sorokin, Shantanu Thakoor, Nauman Ahad, Carolina Urzay, and Eva L. Dyer. Learning behavior representations through multi-timescale bootstrapping. ArXiv , abs/2206.07041, 2022.

OpenAI. Gpt-4 technical report, 2023.

Hanzhou Li, John T. Moon, Saptarshi Purkayastha, Leo Anthony Celi, Hari Trivedi, and Judy Wawira Gichoya. Ethics of large language models in medicine and medical research. The Lancet. Digital health , 2023.

Tongshuang Sherry Wu, Michael Terry, and Carrie J. Cai. Ai chains: Transparent and controllable human-ai interaction by chaining large language model prompts. Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems , 2021.

April Yi Wang, Dakuo Wang, Jaimie Drozdal, Michael Muller, Soya Park, Justin D Weisz, Xuye Liu, Lingfei Wu, and Casey Dugan. Documentation matters: Human-centered ai system to assist data science code documentation in computational notebooks. ACM Transactions on Computer-Human Interaction , 29(2):1-33, 2022.

Zexuan Zhong, Tao Lei, and Danqi Chen. Training language models with memory augmentation. arXiv preprint arXiv:2205.12674 , 2022.

Weizhi Wang, Li Dong, Hao Cheng, Xiaodong Liu, Xifeng Yan, Jianfeng Gao, and Furu Wei. Augmenting language models with long-term memory. arXiv preprint arXiv:2306.07174 , 2023.

Anthropic. Introducing claude, 2023.

Tianle Cai, Xuezhi Wang, Tengyu Ma, Xinyun Chen, and Denny Zhou. Large language models as tool makers. arXiv preprint arXiv:2305.17126 , 2023.

Alexander I. Hsu and Eric A. Yttri. B-soid, an open-source unsupervised algorithm for identification and fast prediction of behaviors. Nature Communications , 12, 2019.

Mackenzie Mathis, Jessy Lauer, Tanmay Nath, Kai Sandbink, Michael Beauzile, Sébastien Hausmann, Steffen Schneider, and Alexander Mathis. DLC2Kinematics: a post-deeplabcut module for kinematic analysis. Zenodo , February 2020.

Appendix

System Prompts

Overall, AmadeusGPT consists of three GPT3.5 or GPT4 instances with different system prompts, the pretrained computer vision models (e.g., SAM and SuperAnimal) and our API interface and implementation. For the pretrained computer vision models (SAM, SuperAnimal), training details are provided in the original works, and we did not fine-tune them. For GPT3.5, the training details were not released by OpenAI at the time of this publication, and we also did not fine-tune any GPT model.

Below are the core API system prompts used to create AmadeusGPT; related to Main Section 3.2.

Below are the Rephraser system prompts used to counter domain shift in the user prompt style; related to Main Section 3.2.

Dual memory mechanism, reload and relaunch example

Related to Main Section 3.3, we discuss how short-term memory can be supplemented by long-term memory to restore chat history. AmadeusGPT stores information including chat histories in both short-term and long-term memory as well as segmentation masks and relation look-up tables. As they are all implemented as Python objects, it is easy to serialize them into disk and deserialize them from the disk. More specifically, we support two special prompts called 'save' and 'load' respectively.

Those two special prompts can save the state of AmadeusGPT before exiting and load the state of AmadeusGPT after relaunching.

API core behavioral modules

This is part of the core API docs that are always sent to AmadeusGPT when queried; related to Main Section 3.4.

Integration of behavioral modules (non-exhaustive)

As described in the Main Section 3.5 and Main Figure 2, AmadeusGPT can use integration modules for task-specific behavior analysis, and implements dynamic loading by converting API documents into text embedding vectors and caching them in RAM or disk, allowing users to retrieve the most relevant modules by cosine similarity using their prompt as a query. Namely, the user could use these modules to integrate functionality. Or, they can write simple modules and contribute them to the AmadeusGPT codebase for other users to leverage as well. Below we provide examples of integration with the newly introduced dimensionality reduction tool called cebra [35], and UMAP [34]. In short, to integrate into the codebase the user should write a minimal working example in the style of the core API docs. Then, after dynamic loading, queries relating to UMAP or cebra (as shown here) would generate the following code (below) and related output plots (Figure 6).

CEBRA

CEBRA is a nonlinear dimensionality reduction tool for joint modeling of neural data and behavioral labels [35], which might be of interest to users of AmadeusGPT. In Appendix Figure 6 we show a 3D CEBRA embedding of mouse poses (aligned) colored by when it overlaps with different objects. Note that the CEBRA embedding in Figure 6 is on a 3D n-sphere that is equivalent to a 2D Euclidean

treadmill

water basin

mouse hut

elevated platform

cotton roll

half tube

platform barrel

food pellet

latent 1

latent 1

latent 3

latent 2

latent 2

Annotated MausHaus masks

UMAP 2D embedding plot

Cebra 3Dembeddingplot

space [35]. Below the user query is shown below to produce images in Figure 6, along with the related API docs.

Query: 'get 3D embeddings using cebra and plot the embedding."

UMAP

UMAP[34] is a popular nonlinear dimensionality reduction tool that is also core to several behavioral analysis tools such as B-SOID [46]. In Appendix Figure 6 we show 2D UMAP embedding of poses (aligned) when it overlaps with different objects.

Query: 'Get 2D embeddings using umap and plot the embedding"

Global spatio-temporal behavioral analyses

Many core behavioral analysis tools rely on computing metrics based on kinematic or movement analysis. For example, time spent locomoting, time in an ROI, or computing the distance traveled within a specific area is highly popular in behavioral neuroscience research. Thus, inspired by DLC2Kinematics [47], we show API docs to integrate two such common integration modules:

Additional Experiments: behavioral analysis examples

Here, as related to Main Section 4, we provide more example queries and responses from AmadeusGPT to support Main Figure 5a (EPM), to Main Figure 5b (MausHaus), and to Main Figure 5c (MABe) and results in Table 1.

Extended Results on EPM with AmadeusGPT

We show additional user queries and generated code from AmadeusGPT related to the EPM data presented in the Main Figure 5a.

Extended Results on MausHaus with AmadeusGPT

We show the user queries and generated code from AmadeusGPT that produce the images shown in the Main Figure 5b.

Query: 'Give me the trajectories of the animal entering object 18 from right"

AmadeusGPT Code Output:

Query:

"Define <|head dips|> as a behavior where the animal's bodyparts mouse_center and at the same time neck in the roi and its head_midpoint outside roi.Get head dips."

Amadeus Code Outputs:

Amadeus Chat Outputs (Execution):

Result for query : 910 frames (total video length is 15076 frames).

0

2000

4000

6000

8000

10000

12000

14000

frame number

1 Query:

"When is the animal moving faster than 8 pixels per frame?"

X Amadeus Code Outputs:

Amadeus Chat Outputs (Execution):

Query:

"Define
 as a behavior where the animal is slower than0.5.When is the mouse freezing?"

Amadeus Code Outputs:

AmadeusChat Outputs(Execution):

Figure 7: Extended Results on classical behavioral task (EPM) with AmadeusGPT . (a) Using the EPM dataset [13], we show additional queries and responses from AmadeusGPT. The raster plot is an ethogram where there is a tick for every related event in time, i.e., the mouse is in the closed arm or in the open arm.

Query: 'plot trajectories of the animal for the shortest event where it enters object 18 and then its neck enters object 17'

AmadeusGPT Code Output:

MABe Mouse Triplet Benchmark, extended details

In the MABe 2022 Mouse Triplet Benchmark, participants are given a dataset of tracking data of triplets of socially interacting mice and asked to submit a frame-by-frame representation of the dataset. Here are the following user prompts and AmadeusGPT code outputs of the sub-tasks we considered and the generated code by AmadeusGPT.

Query: 'Define 
 as a social behavior where the closest distance between animals is less than 40 pixels and the chased animal is in front of the chasing animal and the chasing animal has to travel faster than 3.4 pixels per frame. The smooth window size is 25 and min window is 30. Get chases.'

AmadeusGPT Code Output:

Query: 'Define <|oral genital contact|> as a social behavior where closest distance between this animal's "nose" and other animal's bodyparts "tail base" is less than 15, get the events for oral genital contact.'

AmadeusGPT Code Output:

Query: ' Define <|oral oral contact|> as a social behavior where closest distance between this animal's "nose" and other animal's "nose" is less than 15, get events for oral oral contact.'

AmadeusGPT Code Output:

Query: 'Define 
 as a social behavior where the mice move from at least 40 pixels apart sequentially to less than 8 pixels apart. Get approaches events.'

AmadeusGPT Code Output:

Query: 'Define 
 as a social behavior where the distance between animals is less than 50, and the relative speed between two animals is less than 4 pixels per frame. The smooth window size is 61 and min window is 75. Get huddles.'

AmadeusGPT Code Output:

Query: 'Define 
 as a social behavior where the closest distance between animals is less than 12. The smooth window size is 11 and min window is 5. Get contact.'

AmadeusGPT Code Output:

Query: 'Define 
 as a social behavior where the closest distance between animals is less than 24. The min window is 5. Get close.'

AmadeusGPT Code Output:

Query: 'Define 
 as a social behavior where distance between animals is less than 260 and larger than 50 and angle between animals is less than 15. The min window is 100 and smooth window is 15. Get watching.'

AmadeusGPT Code Output:

Query: 'Define <|oral ear contact|> as a social behavior where the closest distance between animal's nose and other animals' left ear and right ear is less than 10. The smooth window size is 5 and min window is 15. Get oral ear contact.'

AmadeusGPT Code Output:

Prompt robustness testing

Related to Main Section 4, we show the full base questions and generated permutations we tested to gauge robustness.

The input, base out of distribution (OOD) questions

""" (1) define <|head dips|> as a behavior where the animal's bodypart mouse_center bodypart neck in the roi and its bodypart head_midpoint outside roi ↪ → (2) When is the animal moving faster than 8? (3) define 
 as a behavior where the animal is slower than 0.5. When is the mouse freezing? ↪ → (4) How much time the animal spends in the closed arm? (5) How much time the animal spends outside the closed arm? """ The 25 generated OOD questions """ (1) Dips refer to a behavior in which an animal's bodypart, specifically the mouse_center and neck, are within the ROI while the head_midpoint is outside the ROI. ↪ → (2) When an animal's mouse_center and neck are inside the ROI but the head_midpoint is outside, this is known as dips. ↪ → (3) Dips are characterized by the presence of an animal's mouse_center and neck within the ROI, while the head_midpoint is located outside the ROI. ↪ → (4) The behavior known as dips occurs when an animal's mouse_center and neck are within the ROI, but the head_midpoint is outside of it. ↪ → (5) In animal behavior, dips are defined as the presence of the mouse_center and neck within the ROI, while the head_midpoint is situated outside of it. ↪ → (6) At what point does the animal exceed a speed of 8? (7) When does the animal go faster than 8? (8) At what moment does the animal surpass a speed of 8? (9) When is the animal traveling at a speed greater than 8? (10) At what time does the animal go beyond a speed of 8? (11) At what point can we say that a mouse is freezing? I define freezing as a behavior where the animal moves slower than 0.5. ↪ → (12) When does a mouse exhibit freezing behavior? Freezing is defined as a state where the animal's movement is slower than 0.5. ↪ → (13) What is the definition of freezing in mice? Freezing is characterized by a movement speed of less than 0.5. When does a mouse exhibit this behavior? ↪ →

Questions rephrased automatically by our Rephraser

Code generated by AmadeusGPT without Rephraser

Code generated by AmadeusGPT with Rephraser

Additional stress-testing with naive users and GPT4

We additionally stress-tested the system with naive users. We released a fully usable web App to AmadeusGPT using GPT-3.5 with the 3 demo datasets used in the submitted manuscript (EPM, MABe, and MausHaus videos) and asked neuroscience behaviorists to test it - namely, we added example prompts they could run and a chat-interface to ask their own questions. Here, we now collected extra prompts not generated by the authors for testing and we will include this in the revision. In brief, we sampled 30 naive user prompts at random out of the 362 submitted via our App (prompts having caused a programming error are marked with a X ), and found a roughly 18% error rate with GPT3.5 (see more below for GPT4, which was only 10%):

X Get angles and distances between all body parts. Plot a UMAP graph using the resulting data.

Give me the duration of time the animal engages with the object I picked and the events where the animal overlaps with the object I picked.

X Perform hierarchical clustering and plot the dots with different colors based on their clusters.

Define 
 as a behavior where the animal's body length elongates. Do you see sap in the epm?

Give me the tracking of all 3 animal's noses with each nose being shown in a different color.

Define < | peeking | > as a behavior where the animal's head_midpoint is not in ROI5 or ROI4 and the mid_back point is in ROI5 or ROI4.

X Plot the distance between animals over time.

X What is the speed of the changes from the left to the right arm?

Plot regions of interest (ROIs).

X Define < | relative _ head a ngle | > as the angle between the mouse_center and the head_midpoint. Plot the variation of < | relative _ head _ angle | > over time.

How many animals are there in this data?

Give me the keypoints.

Give me the total distance traveled by each animal.

Define < | watch | > as a social behavior where distance between animals is less than 260 and larger than 50 and head angle between animals is less than 15. The smooth_window_size is 15.

X Plot animal center of mass x-coordinate, velocity, acceleration, and head direction.

Define < | freezing | > as a behavior where the animal is in the same position for longer than 10 seconds.

Plot a figure comparing the time the animal spends in the open arm (ROI0) and the time the animal spends in the closed arm (outside ROI0).

X Plot a bar graph with the first bar representing the total time the animal spends in ROI0 (open arm) and the second bar representing the total time the animal spends outside of ROI0 (closed arm).

Calculate the number of successful spontaneous alternations using a sliding window calculation. A successful spontaneous alternation is defined as when, in the last 5 arm entries, all four arms (ROI0, ROI1, ROI2, and ROI3) are visited at least once.

X Plot the euclidean distance between the nose points of animal0 and animal2 over time.

Why is the angle limited to 180 degrees instead of 360?

Show which what? Please provide more information or specify your request.

X Define < | head _ direction | > as the orthogonal angle to the line between left_ear and right_ear.

Give me the amount of time the animal spends in ROI0 and the events where the animal overlaps ROI0.

Calculate head direction using the ear points and plot it in radians.

X Plot each grooming bout.

Define < | head _ scans | > as a behavior where the animal's head-direction shifts by more than 15 degrees to one side and then the other within 5 seconds. Give events where head_scans happen and bouts for head_scans.

"Find events where there are errors in tracking due to outliers in mean distance between all points, animal acceleration, and velocity. Remove the labeled point causing the error and fill in missing values by interpolation. Plot the x and y coordinates of the animal's center of mass before and after the tracking correction.

Give me the frequency of rearing events for the animal.

Print overlap_roi0_events and its items.

We analyzed the logs from 362 queries, 242 of which were from unique prompts (i.e., from manually typed ones rather than executing the demos). Out of the 362 queries, 329 were automatically rephrased. We found AmadeusGPT had reported 129 'errors': 38 were caused by unsupported features or undefined variables, and were thus explained to the user, while 64 originated from programming errors. Therefore, from 362 queries there were an 18% programming error rate, and 11% unsupported feature requests rate.

Then, we took 10 of the failed runs from external users and re-ran them with GPT4. Now, of the 10 that previously failed, only 3 failed, and 4 correctly output clarifying queries to the user*; This means that if GPT4 was used, given the randomly sampled prompts, the error rate would be only 10

X Get angles and distances between all body parts. Plot a UMAP graph using the resulting data.

Perform hierarchical clustering and plot the dots with different colors based on their clusters.

Plot the distance between animals over time.

What is the speed of the changes from the left to the right arm?

Define < | relative _ head _ angle | > as the angle between the mouse_center and the head_midpoint. Plot the variation of < | relative _ head _ angle | > over time.

X Plot animal center of mass x-coordinate, velocity, acceleration, and head direction.

Plot a bar graph with the first bar representing the total time the animal spends in ROI0 (open arm) and the second bar representing the total time the animal spends outside of ROI0 (closed arm).

Plot the euclidean distance between the nose points of animal0 and animal2 over time.

X Define < | head _ direction | > as the orthogonal angle to the line between left_ear and right_ear.

Plot each grooming bout.

*For (2), (4), (5), (10), the output is a message indicating either that the feature is not implemented or that the query does not provide all information, and explains what would be needed either in terms of development or clarifications from the user.

Example from Explainer

Here is an example query on the EPM dataset and relevant outputs for the explainer module:

Example from Self-correction

Here is an example query on the MausHaus dataset and relevant outputs for the self-correction module:

task_program():

Horse Gait Analysis

Horse gait analysis demo

:Query:

"Could you tell me what the duty factor of the offfrontfoot is and explain its value to me?"

Could you tell ne what the duty factor of the Nearfrontfot s and explain its value

X Chat Output:

0.59

Thedutyfactor ofthe Offfrontfoot is greater than 0.5，indicating that the animal spends more time in the stance phase with its frontfoot than in the swing phase.

0.5936869705526422

The duty factor of the Nearfrontfoot is greater than .5, 1ndicating that the aninal

Keypoints:

spends more time in the stance phase with its front foot than in the swing phase

lyssatayss(1ts

:Query:

2：

n(duty_factors)

"ok! Could you also calculate the duty factors using the keypoint 'offhindfoot'，and compare both?"

elifavtyfactor0.5

else:

ity factor of the Nearfrontfoot is less than 0.5,

X Chat Output:

average_duty_factor ; explanat1on

exampleof theWebAppinterface

def task_program():

behavior_analysis = AnimalBehaviorAnalysis()

offfrontfoot_analysis = behavior_analysis.run_gait_analysis(limb_keypoint_names=["offfrontfoot"])

offfrontfoot_duty_factors=offfrontfoot_analysis["duty_factors"]

offhindfoot_duty_factors=offhindfoot_analysis["duty_factors"]

return comparison

:Query:

"Could you explain what that means?"

X Chat Output:

The duty factor is a measure of the amount of time a limb spends in contact with the ground during a gait cycle.It is calculated by dividing the stance duration (time when the limb is in contact with the ground）by the stride duration (time for a complete gait cycle).

By comparing the duty factors of the Offfrontfoot and offhindfoot,we can determine if there is any difference in the amount of time these limbs spend in contact with the ground.A positive value indicates that the Offfrontfoot has a longer duty factor，meaning it spends more time in contact with the ground compared to the Offhindfoot.A negative value indicates the opposite，with the Offhindfoot having a longer duty factor."""

:Query:

Could you rerun the previous analysis but plot 1t this tine? The Linb 1s

defined with the folloe

ring keypoints: offfrontfoot; offfrontfetlock;

"Could you rerun the previous analysis but plot it this time? The limb is defined with the following keypoints:Offfrontfoot; Offfrontfetlock;Offknee; Elbow and Shoulder."

Offknee; ELbow and Shoulder,

Limb

analysis = AninalBehaviorAnalysfs()

gait_analysis(tin

fig. ax

def task_program():

behavior_analysis = AnimalBehaviorAnalysis()

analysis = behavior_analysis.run_gait_analysis(limb_keypoint_names=["offfrontfoot"])

fix, ax = behavior_analysis.plot_gait_analysis_results(

analysis,

limb_keypoint_names=["offfrontfoot， Offfrontfetlock， Offknee，Elbow，Shoulder"])

return fig, ax

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presented Amadeus, a natural language interface for interactive animal behavior analysis. To accommodate modern LLM (GPT3.5) for behavior analysis, the authors proposed to use an API document to constrain GPT3.5’s knowledge space. Furthermore, the authors proposed Dual Memory Mechanism to read and write behaviors (symbols) to enable correct long-term memory retrieval and overcame the limit of 4096 tokens. The authors demonstrated results on three standard mouse behavior datasets.\n\nSTRENGTHS:\n(1) The motivation is quite novel. The whole system could provide an unprecedented experience of animal behavior analysis through natural language guidance only. This paper pioneers a practical way to integrate LLM to task programming. \n\n(2) The paper is well written. \n\n(3) The quantitative result on MABE dataset demonstrates the effectiveness of the system.\n\nWEAKNESSES:\n(1) Although the paper has claimed “animal behavioral analysis”, the datasets used are mouse only. I wonder whether it is difficult to transfer the experience of this paper to other animals, e.g. monkeys, zebrafish, etc. The authors should discuss its applicability to other animals. \n\n(2) I want to know how many behaviors have been tested on this model. It would be better to list them in the supplementary files.\n\nQUESTIONS:\n(1) Can it extrapolate to behavioral codes that are never seen, and how are the atomic APIs distributed? I mean, the behaviors that could be handled seem to closely related to the capability of atomic API. If the atomic API could not measure animal overlap, would the system work with behavior like \"mount\" which is closely related to pixel/keypoint/area overlap? \n\n(2) What if define a symbol twice with slightly different or totally different descriptions? \n\n(3) Mistake: \nIn Figure.3, “Cmpute the pose from the video” -> “Compute …”\n\nLIMITATIONS:\nThe authors have addressed limitations.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-06-16"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes AMADEUS, a GPT3.5 powered system to perform animal behavior data analytics given a natural language user-given query and a video depicting animal behavior. The model works by using GPT-3.5 to generate python code which makes calls to a instance segmentation and animal pose model, as well as hard-coded modules to compute animal-object relationships and operates with their outputs to return results according to a given query. The proposed model is tested with 3 popular animal behavior analysis benchmarks, showcasing different domains, queries and tasks, showing behavior consistent with human annotators and surpassing existing baselines.\n\nSTRENGTHS:\n- Related work: The paper is open about closely related work, and assigns credit for the contributions that are used in the AMADEUS system. \n\n- Method: \n\t- I appreciate the discussion about the design of the API. It would be valuable to understand what would be the effect of not having made the specific choices mentioned in lines 150-159.  \n\t- I appreciate the Dual Memory Mechanims proposed (includeing the illustrative example of limitations of short context windows). I would like to understand the main differences between that appraoch and the one used in Generative Agents, and a discussion on why their execution flow is more expensive. The memory mechanism also has valuable usability features, such as being able to store and retreive states.\n- Relying on LLM allows to provide complex data analysis queries that were not possible in previous methods, as well as follow up questions to a given data analysis. \n- The proposed method, besides flexible is high performing in existing benchmarks, consisting on varied types of queries, expected outputs and data domains. The method perorms above the state of the art, both qualitatively adn quantitatively, surpassing all submissions in the MABE 2022 behavior challenge.\n\nWEAKNESSES:\nNovelty:\n- The work is very similar to existing works using LLMs for computer vision tasks (ViperGPT, VisProg), with the main difference being that they are applied to the animal behavior domain. This is however reported in the related work section, and the work includes other components to handle long duration data. \n\nScope: \n- This looks more like a systems paper, with many of the contributions already existing in previous works, and the main contribution being in combining these existing contributions for a new domain. I think that the paper does have value, but I am wondering if it is the right scope for NeurIPS.\n\nMethod:\n- The performance on the method seems highly dependent on the data analysis modules, which are handwritten. How general are these modules to different kinds of animal behaviors?\n\n- I am suspicious of the robustness and stress testing section. The full system relies on GPT3.5, and that same model is used to stress-test the system. I would like to see how good is the performance (with/without rephraser) from independent researcher-given queries.\n\nQUESTIONS:\nNA\n\nLIMITATIONS:\nLimitations are mostly addressed. The biggest missing limitation is how general is this approach to other animal behavior analysis, given the strong dependence on the hand-designed modules.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-19"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes Amadeus, a novel natural language-based interface that leverages large language models like ChatGPT and vision-language models like SAM for animal behavior tracking and analysis. Amadeus leverages LLMs to generate code outputs, which can be executed to retrieve visual information and memory information and return requested responses from user queries. In particular, Amadeus proposes a novel dual-memory mechanism that combines short-term and long-term memory banks to effectively analyze extended contexts such as long videos. Experiments demonstrate that Amadeus achieve state-of-the-art performance on the MABE Behavior Challenge 2022 benchmark.\n\nSTRENGTHS:\n- The paper is generally well-written, and the figures are generally informative about the details of each module of Amadeus.\n\n- Leveraging recent advancements on LLMs and VLMs to analyze animal behaviors is a novel idea and holds significant potential for scaling up to more complex animal behaviors and a larger number of animals in the future. One of the strengths of this approach is its accessibility and training-free nature, which makes it adaptable to novel scenarios with relative ease.\n\nWEAKNESSES:\nIn Fig. 4, the implementation of long-term memory storing and retrieval processes is a bit unclear, as the generated code does not explicitly store or retrieve information into / from the long-term memory bank.\n\nWhile the proposed approach achieves the state-of-the-art results on the MABE Behavior Challenge 2022 benchmark, it would be beneficial to include discussions on the failure cases, and analyze their source of failures (e.g., due to mistaken VLM perceptions, or due to mistaken code generations, or due to compounding errors caused by the wrong values pushed to the memory banks, etc.)\n\nQUESTIONS:\nSee \"weaknesses\"\n\nLIMITATIONS:\nLimitations are mostly sufficiently addressed. Another limitation to add is that the current approach is bottlenecked by the capabilities of LLMs and VLMs (e.g., SAMs) being used. Perception errors can occur due to the limited capabilities of current VLMs, and the ability to produce correct responses given more complex queries and more demanding tasks is bounded by the capabilities of current LLMs.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a novel interface and deep-learning system that enables interactive animal behavioural analysis using only natural language, on tasks that previously required extensive coding expertise and technical machine learning knowledge. The proposed framework integrates various modules based on LLMs and existing vision networks dedicated to various parts of the task, such as understanding the data processing API and rephrasing user prompts. This work also contributes a novel long-short-term memory technique in the context of LLMs. In combination, the contributed interface and systems achieves state-of-the-art performance on established tasks in the domain of animal behavioural analysis, and has the potential to support domain experts in using neural techniques and software frameworks that are currently difficult for them to use.\n\nSTRENGTHS:\n- Enables the novel user experience of analyzing animal behaviour using only natural language, which could be an important affordance for domain experts \n\n- Well-designed, workable end-to-end system that reasonably combines multiple deep-learning modules, all with dedicated purposes in various aspects of the task.\n\n- Effective use of SoTA techniques in deep learning to achieve high performance on established domain-specific tasks\n\n- Introduced a novel long-short-term memory module that is compatible with LLMs and shown effective usage of such module in the contributed framework\n\nWEAKNESSES:\n- **Unclear general applicability of the long-short-term memory module and the framework in general:** From a machine learning perspective, the most significant technical contribution would be the long-short-term memory module in the context of LLM that the author(s) developed. The current paper discusses this module primary in the context of animal behavioural analysis. I would like to see some discussion of the general applicability of the memory module to common domains utilising LLMs, and also the general applicability of the entire framework to other domains, such that researchers working on those domains can benefit from the technical contributions of this paper.\n\n- **Unclear generalizability of the work due to author-crafted prompts:** This is a limitation the author(s) have acknowledged, such that the experiments performed by the author(s) all followed prompts crafted by the author(s). I also acknowledge that the author(s) have attempted to alleviate this issue by introducing OOD base questions and using similar task descriptions as the original definition in the MABE challenge. However, I am curious about the opinion of the end-user (i.e., animal behaviour analysts/specialists) on the ability of the system in handling their prompts. It would be great if the author(s) can include some study (could be formative/informal) and/or discussion regarding the ecological validity of the system. It would be great if the author(s) can report lower-level success metrics, such as the rate of having syntax/compilation errors of the analysis code generated by Amadeus.\n\n- **Limited technical novelty and contribution for Machine Learning**: Related to the first point, the primary technical contribution of the paper in terms of modelling techniques is the long-short-term memory module, which is limited. I believe more extensive discussion and/or experiments on this module could be helpful for researchers in the ML community to build upon the findings of this paper.\n\nQUESTIONS:\n1) What are the general applicability of the long-short-term memory module and the entire framework to domains other than animal behaviour analysis?\n\n2) Did any animal behaviour experts tried using the system / reviewed the prompts in the experiments of this paper? If so, what are their opinions on the ecological validity of them?\n\n3) What is the more low-level success metrics of the code generated by the system, such as the compilation error rate?\n\nLIMITATIONS:\nThe limitations of this work were adequately discussed by the author(s).", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis submission introduces a system, Amadeus, for performing behavioral analysis on animal videos. This system combines three elements: an API with descriptive docstrings for performing common behavioral analysis tasks, a modified version of GPT3.5 with an enhanced context window, and a prompt tuning of this GPT3.5 to allow for construction of task programs for behavioral analysis using a natural language interface. What is claimed is that (1) the language model can extend beyond its nominal 4096 character token window (2) the system allows for behavioral analysis with no additional code written (3) the system is easily extensible. The authors’ provide schematics of the system architecture and the integration between various components. There are numerous examples for how written code is produced from natural language. There are a few evaluations provided (1) comparing human raters and the amadeus system in computing time spent in arms of an elevated plus maze (2) comparing the performance of the system in a select number of tasks from the MABE challenge and (3) comparing the ability of a trained rephraser model to domain adapt queries to the developer’s syntactical patterns. There is a fairly comprehensive inclusion of the API in the appendix as well as examples of the rephraser module. \n\nOverall I found the contribution novel, of potential immediate utility to academic neuroscience labs performing behavioral analysis, and as an example for other groups hoping to automate analysis in other systems. I however found the evaluations and description of the methods somewhat lacking, and the generalizability and extensibility of the somewhat unclear. These would need to be improved for me to strongly endorse the paper.\n\nSTRENGTHS:\n•\tThis is the first LLM integration I have seen in the neuroscience and I think the approach is potentially interesting. In some ways I think it could be even more interesting for codebases that analyze on standardized experimental equipment (e.g. fluorescence imaging, neuropixels) because the set of tasks to perform is arguably more standardized than behavioral analysis, where the approaches, species, and questions can be fairly diverse. \n\n•\tMany researchers in the life sciences do not have a formal background in writing code or have to interact with a fairly specialized API to analyze data, and so assistance is useful here.\n\n•\tThe system schematics and examples are clear. \n\n•\tThe API documentation is helpful, and the integration of state of the art systems like SAM into an open-source (if this is to be distributed) codebase is helpful.\n\nWEAKNESSES:\n•\tThe manuscript is missing descriptions of the system architecture and training details. It is difficult to fully understand how the system was trained without this.  \n\n•\tThe evaluations are somewhat cursory. The results on the EPM are hard to evaluate because the ‘ground truth’ from the human raters is quite variable. The MABE evaluation is only presented across a subset of tasks. \n\n•\tThe generalizability and extensibility of the system to new users and new behavioral tasks is unclear. Part of this is the rephraser example is somewhat limited, but it is also unclear how successful new users will be in writing functions with appropriate documentation and that documentation hints are not interdependent.\n\nQUESTIONS:\n•\tTable 1 only lists a subset of MABE tasks, the model results should be reported across the full benchmark.\n\n•\tThe lack of methods makes it difficult to know the details of the memory augmentation approach and the extent of training of the rephraser module (and how much one expects the later to generalize).\n\n•\tI found the memory augmentation result under-contextualized and other work in this space should be discussed. Augmenting LLM memory window is a subject under intense technological development and the contribution should be put into proper context, especially because it is a secondary result. \n\n•\tThe EPM results are hard to evaluate because of inter-human variability. An experiment with clearer ground truth would help. \n\n•\tIn general, hallucinations are a real problem for LLMs and I am not convinced by the given evaluations that they would not occur (e.g. for what fraction of a given call does the system produce the correct result). This is especially problematic for a system designed to be used by people who do not code and may not be able to debug problems. \n\n•\tThe API is fairly expansive and includes example of most functions call and keyword arguments present in the given examples in the paper. This could mean that the system is limited in functionality to simply modifying parameters and numerical values given as input to the functions. Some statement about the expected generalization ability of the approach and the requirements for adding new functions to the docstring (e.g. does one have to provide examples of every new keyword input) is needed. \n\n•\tThe claim that using the system would obviate the need to be able to code is strong. As the authors’ note, hallucinations are a problem for LLMs. Many in the field regard LLM solutions as an accelerated rough draft to be ‘fact-checked’ by a domain expert. Similar to the rephrasing analysis, it would be nice to have a statement about the reliability of the results, ideally across a pool of fairly naïve users.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-06"}