SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Chat Output 2:
Results for query : 2700 frames (total video length is 15076 frames)
Code Outputs 1:
def task-program(: behavior_analysis  $=$  AnimalBehaviorAnalysis() in_ROIO_events  $\equiv$  \\ behavior_analysis.animals_object_events(\\ 'ROIO','overlap', bodyparts=['all'],  $\backslash$  negate=True) return in_ROIO_arm_events
Chat Output 1:
Results for query : 12376 frames(total video length is 15076 frames)
(a.1) EPM: GT vs.
Code Outputs 2:
def task_program():
    behavior_analysis = AnimalBehaviorAnalysis()
    outside_ROIO_events = \
        behavior_analysis.Animals_object_events('ROI0', 'overlap', \ bodyparts=['all'], negate=True)
    return outside_ROIO_events
0 2000 4000 6000 8000 10000 12000 14000
(b)MausHaus
:Query:
"When is the animal on the treadmill, which is object 5?"
Query:
"Plot the trajectory of the animal."
"When is the animal close to the object 35, if I define close as less than 50 pixels?"
def task_program(): behavior_analys overlap_object fig, ax = behavior return fig, ax
Official definition
Chase: Mice are moving above  $15\mathrm{cm / sec}$  with closest points less than 5 cm apart, and angular deviation between mice is less than 30 degrees, for at least  $80\%$  of frames within at least one second. Merge bouts less than 0.5 seconds apart.
Watching: Mice are more than  $5\mathrm{cm}$  apart but less than  $20~\mathrm{cm}$  apart, and gaze offset of one mouse is less than 15 degrees from body of other mouse, for a minimum duration of 3 seconds. Merge bouts less than 0.5 seconds apart.
Our prompt
"Define  $<$  |chases> as a social behavior where closest distance between this animal and other animals is less than 40 pixels and the angle between this and other animals have to be less than 30 and this animal has to travel faster than 2 pixels. When do chases happen?"
"Define  $<$  |watching|> as a social behavior where distance between animals is less than 260 pixels and larger than 50 and head angle between animals is less than 15. The smooth_window_size is 15. When does watching happen?"
(c) MABE Challenge dataset
Figure 5: Results on classical behavioral tasks with AmadeusGPT. (a) Result on EPM showing limited time in the open arms by the mouse. The raster plot is an ethogram where there is a tick for every related event in time, i.e., the mouse is in the closed arm or in the open arm. (a.1) shows AmadeusGPT counts vs. three human raters [13] across five videos (colored dots). (b) Animal-object interactions can be computed in natural home cage settings. (c) Behavioral images and original description given in MABe, vs. our prompts that produce the quantitative results shown in Table 1. For Chase, we visualize the chasing animal's trajectory that overlaps with the predicted mask and the chasing animal's trajectory for the ground-truth mask. For Watching, we visualize the visual cone for each animal from each animal's head coordinate (axis by neck-nose vector and its normal vector).
In the open arm (a.1) shows
Tasks	T4 approach	T5 chase	T6 close	T7 contact	T8 huddles	T9 *	T10 *	T11 *	T12 watch
PCA baseline	0	0	0.13	0.0008	0	0	0	0	0
Top-entry 1 [4]	0.020	0.010	0.708	0.558	0.310	0.0056	0.015	0.013	0.182
Top-entry 2 [4]	0.026	0.050	0.7072	0.561	0.214	0.0084	0.029	0.023	0.159
Top-entry 3 [4]	0.022	0.029	0.655	0.515	0.249	0.0062	0.015	0.014	0.162
BAMS [37]	0.02	0.023	0.664	0.533	0.302	0.0045	0.0165	0.014	0.191
AmadeusGPT	0.014	0.274	0.700	0.572	0.380	0.05	0.05	0.024	0.600
Table 1: F1 scores by AmadeusGPT on several tasks from the MABe Behavior Challenge 2022 [4]. We do not use representation learning (the aim of the benchmark), only rule-based task programming. *T9: oral-ear-contact; T10: oral-genital-contact; T11: oral-oral-contact.
from python as well as the user queries to explain whether the code and its return values meet the queries (See Appendix for an example).
To evaluate AmadeusGPT we ran a series of qualitative and quantitative experiments. We used three datasets that highlight standard behavioral neuroscience settings. The first is an open-access Elevated Plus Maze (EPM) dataset from Sturman et al. [13]. The second is called MausHaus and is a video of a mouse for one hour (108K frames) in an enriched home-cage setting [17]. The third is video data of three mice interacting from the MABe 2022 Challenge [4]. We also include a fourth in-the-wild horse dataset [36].
EPM. The EPM is a classical task in neuroscience research to test an animal's anxiety about being in exposed "open arms" vs. "closed arms" [13]. We show that with minimal prompts and ROI interactive plotting a user can measure these events (Figure 5a, also see Figure 3). Here, we query AmadeusGPT to report both open and closed arms and note that the resulting raster plots (ethograms) do not identify the same frames, as one expects if it is correct (i.e., the mouse cannot be in both states at once). We also show that AmadeusGPT counts the number of events similar to those reported in ground truth annotations by three raters across five different videos (Figure 5a.1).
MausHaus: enriched mouse-cage monitoring. Studying more natural behavior is becoming important in research settings. Here, we demonstrate intuitive prompts that run pretrained models (SuperAnimal-TopViewMouse and SAM) then query spatio-temporal relationships between the model outputs (Figure 5b). As SAM only provides object labels, we use object number or interactive clicking on the object to ground the analysis (see also Figure 3).
Horse gait analysis. AmadeusGPT has no innate mouse preference. Therefore to show another application we performed equine gait analysis [36]. We leveraged horse videos with ground truth keypoint annotations on every frame and used SuperAnimal-Quadruped [17], and found comparable results within AmadeusGPT to human-level labeling and stride analysis (see Appendix Figure 8).
MABe 2022 Benchmark. The benchmark had two rounds for the three mouse dataset. We used the more popular first round (evaluation split) and therefore provided the pre-computed keypoints as inputs to AmadeusGPT. In Figure 5c, we show how AmadeusGPT captures two representative tasks from MABe benchmark, Chase and Watching. We use text that is close to the original definition to define the behaviors we want to capture. Note the units in our prompt are pixels for distance, radians for degree and pixel per frame for speed. We tested nine behaviors and report the F1 score as computed in the MABe evaluation (Table 1). Our approach is purely rule-based, thus no machine learning is needed and only three parameters are needed to be given or tuned: a smoothing window size for merging neighboring bouts, a minimal window size for dropping short events, and the pixel per centimeter. Note that tasks that are hard for machine learning models are also hard for our rule-based approach (Table 1, Tasks 9-11).
We do not intend to formally claim state-of-the-art on the benchmark, as the goal was to evaluate unsupervised representational learning models. Nevertheless, we show that in practice one can use a text definition of behavior to capture behaviors that are on par with, or better than, the competitive representation learning approaches.
Robustness tests and stress-testing AmadeusGPT. Users might query AmadeusGPT with expressions that are very different from the explanation text we provided in the API documentation. A potential pitfall is that AmadeusGPT overfits to our (the developers) expressions and biases. To test how robust AmadeusGPT is, we crafted five out-of-distribution (OOD) base questions for an EPM video. Then we asked a native GPT3.5 (part of our reflection module, see Figure 2) to generate five variants of each OOD question. We manually checked whether AmadeusGPT generated consistently correct results on those 25 generated questions, and it did  $88\%$  of the time with our Rephraser module vs.  $32\%$  without the Rephraser. The total number of tokens consumed using the ChatGPT-API was 4,322 and 5,002 with and without using Rephraser, respectively. Note that in both cases, the consumed number of tokens is larger than the maximal context window size of 4,096. This also shows that AmadeusGPT passes the stress-test we set for it in two key ways: (1) The short-term memory deque correctly maintains the constrained size without getting an error from the OpenAI API due to maximal token size error; (2) The diverse questions in the short-term memory do not result in mode collapse or severe performance degradation.
In additional testing with naive users we found that better LLMs boost performance. In brief, we sampled 30 naive user prompts at random out of 362 submitted via an App we developed, and found an  $18\%$  error rate with GPT3.5 yet with GPT4 [38] only  $10\%$ . See the Appendix for examples.
AmadeusGPT is a novel system for interactive behavior analysis. By providing a user-friendly interface and leveraging the power of language models, AmadeusGPT enables researchers to more efficiently and effectively study animal behavior. We believe this system offers a significant contribution to the field of behavioral analysis. The natural language interface of AmadeusGPT empowers non-experts to conduct advanced behavioral analysis with cutting-edge computer vision (such as with SAM [11] and SuperAnimal [17]), but its reliance on LLMs raises potential ethical concerns such as bias amplification. The use of a custom-designed API module in AmadeusGPT helps limit potential biases in outputs, but future work should consider how integration modules can introduce biases [39]. It also allows for domain-specific human-AI interactions [40, 41].
Our proposed natural language interface with the augmented-GPT3.5 shows promise, but there are limitations that need to be addressed in future work. These include enhancing robustness by reducing bias in prompt writing, enabling more goal-driven code reasoning to make AmadeusGPT more collaborative, and improving generalization by extending the available APIs and preventing ChatGPT from writing incorrect code. We also only support the English language, but multi-lingual support with be explored in the future. Collectively, we believe AmadeusGPT promises to open new avenues for both leveraging the utility of, and developing on, cutting-edge computer vision and machine learning tools by using intuitive language.
It is also worth noting that AmadeusGPT leverages ChatGPT API calls, which means we do not have access to the weights of the underlying LLMs thus we cannot yet fine-tune or augment the LLMs. Therefore, our dual-memory leverages the process memory and disks to augment the language model, in contrast to works that augment LLMs by assuming the access to the LLMs [42, 43]. Similarly, without fine-tuning the underlying LLMs, we constrain the behaviors of the LLMs only via in-context learning for both the code generator and rephraser. While recent models such as GPT4 [38], CLAude [44] continue to make context window bigger and in-context learning more powerful, deploying them can be slower and more expensive and a strategy of combining a smaller model and a bigger model is worth exploring [45]. In the future, it would be interesting to compare our in-process-memory with in-context memory in terms of reliability and cost. It is also interesting to compare fine-tuned LLMs vs. LLMs that are constrained via in-context learning for future works that use natural language interface for behavior analysis.
Of course, AmadeusGPT is not limited to using GPT3.5 or GPT-4, and new models such as CLAude [44] or others could be used. However, the larger the model the slower the response time and the higher the computational cost. Thus, while these excellent works have started to overcome token limits, there are still limits such that our computationally efficient dual-memory system will be of broad interest to those developing both domain-specific solutions (such as AmadeusGPT) and generalist models (the base GPT models). Collectively, we believe AmadeusGPT promises to open new avenues for both leveraging the utility of, and developing on, cutting-edge computer vision and machine learning tools with minimal to no coding – namely, by only using intuitive language.
This work was supported by The Vallee Foundation Scholar award to MWM. We thank NeuroX at EPFL for travel support. We thank members of the M.W.Mathis Lab and A.Mathis Group at EPFL for feedback, A. Iqbal for computing animal-object interactions used in Appendix Figure 6, St. Schneider and Kinematik AI for assistance and hosting our demo, and the alpha testers. COI: MWM is a co-founder and equity holder in Kinematik AI. Author Contributions: Conceptualization: SY, MWM; Methodology & Software: SY, JL, MWM, MZ, AM; Writing: MWM, SY; Writing-Editing: JL, AM; Visualization: MWM, SY, MZ; Funding acquisition: MWM.
[34] Leland McInnes and John Healy. Umap: Uniform manifold approximation and projection for dimension reduction. ArXiv, abs/1802.03426, 2018.
[35] Steffen Schneider, Jin Hwa Lee, and Mackenzie Weygandt Mathis. Learnable latent embeddings for joint behavioural and neural analysis. Nature, May 2023.
[36] Alexander Mathis, Thomas Biasi, Steffen Schneider, Mert Yuksekgonul, Byron Rogers, Matthias Bethge, and Mackenzie W Mathis. Pretraining boosts out-of-domain robustness for pose estimation. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 1859-1868, 2021.
[37] Mehdi Azabou, Michael Mendelson, Maks Sorokin, Shantanu Thakoor, Nauman Ahad, Carolina Urzay, and Eva L. Dyer. Learning behavior representations through multi-timescale bootstrapping. ArXiv, abs/2206.07041, 2022.
[38] OpenAI. Gpt-4 technical report, 2023.
[39] Hanzhou Li, John T. Moon, Saptarshi Purkayastha, Leo Anthony Celi, Hari Trivedi, and Judy Wawira Gichoya. Ethics of large language models in medicine and medical research. The Lancet. Digital health, 2023.
[40] Tongshuang Sherry Wu, Michael Terry, and Carrie J. Cai. Ai chains: Transparent and controllable human-ai interaction by chaining large language model prompts. Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems, 2021.
[41] April Yi Wang, Dakuo Wang, Jaimie Drozdal, Michael Muller, Soya Park, Justin D Weisz, Xuye Liu, Lingfei Wu, and Casey Dugan. Documentation matters: Human-centered ai system to assist data science code documentation in computational notebooks. ACM Transactions on Computer-Human Interaction, 29(2):1-33, 2022.
[42] Zexuan Zhong, Tao Lei, and Danqi Chen. Training language models with memory augmentation. arXiv preprint arXiv:2205.12674, 2022.
[43] Weizhi Wang, Li Dong, Hao Cheng, Xiaodong Liu, Xifeng Yan, Jianfeng Gao, and Furu Wei. Augmenting language models with long-term memory. arXiv preprint arXiv:2306.07174, 2023.
[44] Anthropic. Introducing claude, 2023.
[45] Tianle Cai, Xuezhi Wang, Tengyu Ma, Xinyun Chen, and Denny Zhou. Large language models as tool makers. arXiv preprint arXiv:2305.17126, 2023.
[46] Alexander I. Hsu and Eric A. Yttri. B-soid, an open-source unsupervised algorithm for identification and fast prediction of behaviors. Nature Communications, 12, 2019.
[47] Mackenzie Mathis, Jessy Lauer, Tanmay Nath, Kai Sandbink, Michael Beauzile, Sébastien Hausmann, Steffen Schneider, and Alexander Mathis. DLC2Kinematics: a post-deeplabcut module for kinematic analysis. Zenodo, February 2020.
System Prompts
Overall, AmadeusGPT consists of three GPT3.5 or GPT4 instances with different system prompts, the pretrained computer vision models (e.g., SAM and SuperAnimal) and our API interface and implementation. For the pretrained computer vision models (SAM, SuperAnimal), training details are provided in the original works, and we did not fine-tune them. For GPT3.5, the training details were not released by OpenAI at the time of this publication, and we also did not fine-tune any GPT model.
Below are the core API system prompts used to create AmadeusGPT; related to Main Section 3.2.
system_prompt = f"Your name is AmadeusGPT and you are helping our users by first  
 $\rightarrow$  understanding my API docs: \n{interface_str}\n{clsbehavior Modules_str}\n"  
system_prompt += "Now that you understand my API docs, If user asks things that are  
 $\rightarrow$  achievable by using my APIs, write and only write a function named task_program()  
 $\rightarrow$  with return values. If not achievable by API docs, explain why. Think step by step  
 $\rightarrow$  when generating code.\n"  
system_prompt += "Rule 1: (1) Do not access attributes of objects, unless attributes are  
 $\rightarrow$  written in the Arribles part of function docs. (2) Do not call any functions or  
 $\rightarrow$  object methods unless they are written in Methods part in the func docs.\n"  
system_prompt += "Rule 2: Pay attention to whether the user means for single animal or  
 $\rightarrow$  multiple animals.\n"  
system_prompt += "Rule 3: Never call instance functions of matplotlib objects such as ax  
 $\rightarrow$  and plt.\n"  
system_prompt += "Rule 4: Do not write any comments in the function.\n"  
system_prompt += "Rule 5: Must pay attention to the typing for the parameters and returns  
 $\rightarrow$  of functions when writing code.\n"  
system_prompt += "Rule 6: When generating events, pay attention to whether it is  
 $\rightarrow$  simultaneous or sequential from the user prompt. For example, events describing  
 $\rightarrow$  multiple bodyparts for one animal must be simultaneous events. Animal moving from one  
 $\rightarrow$  place to the other must be sequential events.\n"
Below are the Rephraser system prompts used to counter domain shift in the user prompt style; related to Main Section 3.2.
system_prompt = f"Your job is to help translate queries into more queries that follow my rules and examples. If you do not know how to translate, just repeat what you are asked and never say sorry. \n"
system_prompt += "Rule 1: Make sure you only answer with a short converted version. No explanation or other formatting.\n"
system_prompt += "Rule 2: Make sure you do not change bodypart names and do not change name of the behaviors. Name of the behaviors might be enclosed by  $<\|\>$ ."
system_prompt += "Rule 3: Do not change temporal conjunctions when rephrasing. For example, 'and then' cannot be replaced by 'and'.\n"
system_prompt += (
    "Rule 4: Do not change the semantic meaning of the query. Do not remove important details.\n"
)
system_prompt += "Rule 5: Do not rephrase adjective words such as shortest event or longest distance.\n"
system_prompt += "Rule 6: when a spatial relation is used as adjunctive such as entering from right, add a adjunctive word simultanesously such as entering from right simultaneously.\n"
system_prompt += "Rule 7: Do not rephrase terms with special meaning such as cebra, umap, or deepabcut.\n"
system_prompt += "Rule 8: If you do not know how to rephrase it, just copy the instruction and write back.\n"
system_prompt += "Rule 9: Keep the adjectives for words like head angle or closest distance or relative speed. They should not be rephrased such that head angle is confused as angle or relative_speed confused as speed\n or distance confused as closest distance\n"
system_prompt += "Rule 10: Rephrase specialized animal names such as mouse, mice to animal, animals.\n"
system_prompt += f"Example 1: How much time the animal spends in ROI? -> Give me the amount of time animal spends in ROI and the events where the animal overlaps ROI.\n"
system_prompt += f"Example 2: How much time the animal moves faster than 3? -> Give me events where the animal moves faster than 3.\n"
system_prompt += f"Example 3: What is the duration of time animal is outside ROI? -> Get events where the animal is outside ROI.\n"
system_prompt += f"Example 4: What is distance travelled in ROI -> Get me distance travelled of the animal overlaps ROI.\n"
system_prompt += f"Example 5: Get object ethogram -> Plot ethogram for animal overlapping all objects.\n"
system_prompt += f"Example 6: Give me trajectories of the animal -> Plot me trajectories of the animal.\n"
system_prompt += f"Example 7: The animal enters object 1 and then enters object 2 -> The animal enters object 1 and then enters object.\n"
system_prompt += f"Example 8: Define 
 as a social behavior where distance between animals is less than 260 and distance larger than 50 and angle between animals \nless than 15. Get masks for watching. -> Define 
 as a social behavior where distance between animals is less than 260 and distance larger than 50 and angle between animals \nless than 15. Get events where watching happens.\n"
system_prompt += f"Example 10: plot object ethogram -> plot object ethogram.\n"
system_prompt += f"Example 11: define 
 as a behavior where the animal's bodypart tail_base moves faster than 3 while animal's bodypart nose moves faster than 4, when does the animal do bla? -> define 
 as a behavior where the animal's tail_base moves faster than 3 while animal's nose moves faster than 4. Give events where animal does bla.\n"
system_prompt += f"Example 12: Give me frames where the animal is on object 17 -> Give me events where the animal is on object 17.\n"
system_prompt += f"Example 13: Give me 3d cebra embedding -> Give me 3d cebra embedding and plot the embedding.\n"
system_prompt += f"Example 14: plot trajectory when the animal is on object 1 -> plot trajectory for events where the animal overlaps object 1.\n"
system_prompt += f"Example 15: Help me define the behavior freeze -> Help me define the behavior freeze.\n"
system_prompt += f"Example 16: Where is the animal in the first half of the video? -> Where is the animal in the first half of the frames?\n"
system_prompt += f"Example 17: When is the animal above object 3? -> Give me events where animal is above object 3.\n"
system_prompt += f"Example 18: When is the closest distance among animals is less than 3? -> Give me events where the closest distance among animals is less than 3.\n"
system_prompt += f"Example 19: How much time does the mouse spend on roi? -> Give me amount of time animal spends on roi and events where the animal overlap with ROI0 \n"
system_prompt += f"Before rephrasing, keep in mind all those rules need to be followed."
Below are the self-correction system prompts used to fix common runtime errors; ; related to Main Section 3.2.
system_prompt = "" Your job is to correct a code that raised errors. You will be given  $\leftrightarrow$  the user query the code was written for, the code, the error message and the  $\leftrightarrow$  diagnosis for why the error happens
system_prompt+= "There are empirical rules that can help you debug:\n"  
system_prompt+= f"Rule 1: If the code used a bodypart that does not exist, replace it  $\leftrightarrow$  with one that is semantically close from supported bodyparts.\n"
Dual memory mechanism, reload and relaunch example
Related to Main Section 3.3, we discuss how short-term memory can be supplemented by long-term memory to restore chat history. AmadeusGPT stores information including chat histories in both short-term and long-term memory as well as segmentation masks and relation look-up tables. As they are all implemented as Python objects, it is easy to serialize them into disk and deserialization them from the disk. More specifically, we support two special prompts called "save" and "load" respectively.
Those two special prompts can save the state of AmadeusGPT before exiting and load the state of AmadeusGPT after relaunching.
Query: "save"
AmadeusGPT Output:
Saving state into state.pickle
Query: "load"
Loaded state.pickle
API core behavioral modules
This is part of the core API docs that are always sent to AmadeusGPT when queried; related to Main Section 3.4.