SlowGuess/NeurIPS_2023_OCR huggingface dataset:
1
(1) Dips refer to a behavior in which an animal's bodypart, specifically the mouse_center  $\leftrightarrow$  and neck, are within the ROI while the head_midpoint is outside the ROI.
(2) When an animal's mouse_center and neck are inside the ROI but the head_midpoint is  $\rightarrow$  outside, this is known as dips.
(3) Dips are characterized by the presence of an animal's mouse_center and neck within  $\rightarrow$  the ROI, while the head_midpoint is located outside the ROI.
(4) The behavior known as dips occurs when an animal's mouse_center and neck are within  $\rightarrow$  the ROI, but the head_midpoint is outside of it.
(5) In animal behavior, dips are defined as the presence of the mouse_center and neck  $\leftrightarrow$  within the ROI, while the head_midpoint is situated outside of it.
(6) At what point does the animal exceed a speed of 8?
(7) When does the animal go faster than 8?
(8) At what moment does the animal surpass a speed of 8?
(9) When is the animal traveling at a speed greater than 8?
(10) At what time does the animal go beyond a speed of 8?
(11) At what point can we say that a mouse is freezing? I define freezing as a behavior  $\leftrightarrow$  where the animal moves slower than 0.5.
(12) When does a mouse exhibit freezing behavior? Freezing is defined as a state where  $\rightarrow$  the animal's movement is slower than 0.5.
(13) What is the definition of freezing in mice? Freezing is characterized by a movement  $\leftrightarrow$  speed of less than 0.5. When does a mouse exhibit this behavior?
(14) When can we say that a mouse is freezing? Freezing is a behavior where the animal's  $\leftrightarrow$  movement is slower than 0.5. What are the conditions that lead to this behavior?
(15) What does it mean when we say that a mouse is freezing? Freezing is a state where  $\leftrightarrow$  the animal moves slower than 0.5. When does this behavior typically occur in mice?
(16) What is the duration for which the animal remains in the enclosed arm?
(17) How long does the animal spend in the closed arm?
(18) What is the time frame for which the animal stays in the enclosed arm?
(19) What is the length of time that the animal spends in the closed arm?
(20) What is the period of time during which the animal remains in the enclosed arm?
(21) What is the duration for which the animal remains outside the closed arm?
(22) How long does the animal stay outside the closed arm?
(23) What is the amount of time the animal spends outside the closed arm?
(24) For how much time does the animal remain outside the closed arm?
(25) What is the length of time that the animal spends outside the closed arm?
Questions rephrased automatically by our Rephraser
1 1
(1) Define  $< |dips|>$  as a behavior where the animal's mouse_center and neck are in ROI and  $\leftrightarrow$  head_midpoint is outside ROI.
(2) Define dips as an event where the animal's mouse_center and neck are inside the ROI  $\rightarrow$  but the head_midpoint is outside.
(3) Analyze events where the animal's mouse_center and neck are in ROI and head_midpoint  $\leftrightarrow$  is outside ROI.
(4) Define  $
$  as a behavior where the animal's mouse_center and neck are within  $\leftrightarrow$  ROI, but head_midpoint is outside.
(5) Define dips as the presence of mouse_center and neck within ROI and head_midpoint  $\leftrightarrow$  outside of it.
(6) Give me events where the animal exceeds a speed of 8.
(7) Give me events where the animal goes faster than 8.
(8) Give me events where the animal's speed is greater than 8.
(9) Give me events where the animal is traveling at a speed greater than 8.
(10) Give me events where the animal goes beyond a speed of 8.
(11) Define freezing as a behavior where the animal moves slower than 0.5. When does the  $\leftrightarrow$  mouse freeze?
(12) Give events where the mouse exhibits freezing behavior, defined as movement slower  $\rightarrow$  than 0.5.
(13) Define freezing as movement speed less than 0.5 in mice. Give events where a mouse  $\leftrightarrow$  exhibits freezing behavior.
(14) Define freezing as a behavior where the animal's movement is slower than 0.5. What  $\leftrightarrow$  are the conditions that lead to freezing in a mouse?
(15) What is freezing in mice? Give events where the mouse is freezing.
(16) Get events where the animal is in the enclosed arm and calculate their duration.
(17) Give me events where the animal is in the closed arm.
(18) Get events where the animal stays in the enclosed arm.
(19) Get events where the animal is in the closed arm.
(20) Get events where the animal remains in the enclosed arm.
(21) Get events where the animal is outside the closed arm.
(22) Get events where the animal is outside the closed arm and give me their duration.
(23) Get events where the animal is outside the closed arm.
(24) Give me events where the animal is outside the closed arm.
(25) Get events where the animal is outside the closed arm.
Code generated by AmadeusGPT without Rephraser
#Occurs 5 times
def task_program():
    behavior_analysis = AnimalBehaviorAnalysis()
    # Get events where mouse_center and neck are within the ROI
    dip_events = behavior_analysisanimals_object_events('ROI', 'overlap',
                   $\leftrightarrow$  bodyparts=['mouse_center', 'neck'], negate=False)
    # Get events where head_midpoint is outside the ROI
head_outside_roi_events = behavior_analysisanimals_object_events('ROI', 'overlap',  $\leftrightarrow$  bodyparts=['head_midpoint'], negate=True) #Combine the events using addsequential_events() dip_with_head_outside_roi_events  $=$  Event.addsequential_events(dip_events,  $\leftrightarrow$  head_outside_roi_events) return dip_with_head_outside_roi_events   
#Occurs 5 times   
def taskProgram(): behavior_analysis  $=$  AnimalBehaviorAnalysis() #Get events where the animal's speed exceeds 8 pixels per frame speed_exceed_8_events  $=$  behavior_analysisanimals_state_events('speed','>8') return speed_exceed_8_events   
#Occurs 3 times   
def taskProgram(): behavior_analysis  $=$  AnimalBehaviorAnalysis() #Get events where the animal's speed is less than O.5 pixels per frame freezing_events  $=$  behavior_analysisanimals_state_events('speed','<0.5') return freezing_events   
#Occurs 1 times   
def taskProgram(): return "Freezing in mice is defined as a state where the animal moves at a speed of  $\leftrightarrow$  less than 0.5."   
#Occurs 1 times   
def taskProgram(): behavior_analysis  $=$  AnimalBehaviorAnalysis() #Get events where the animal is in the closed arm in Closed_arm_events  $=$  behavior_analysisanimals_object_events('closed arm',  $\leftrightarrow$  'overlap', bodyparts=['all'], negate=False) #Calculate the duration of each event durations  $=$  [event.duration() for event in in Closed_arm_events['animal0']) #Return the average duration return sum(durations) / len(durations) if durations else 0.0   
#Occurs 2 times   
def taskProgram(): behavior_analysis  $=$  AnimalBehaviorAnalysis() #Get events where the animal is in the closed arm in Closed_arm_events  $=$  behavior_analysisanimals_object_events('closed arm',  $\leftrightarrow$  'overlap', bodyparts=['all'], negate=False) #Calculate the duration of each event durations  $=$  [event.duration() for event in in Closed_arm_events['animal0']) #Return the total duration return sum(durations) if durations else 0.0   
#Occurs 1 times   
def taskProgram(): behavior_analysis  $=$  AnimalBehaviorAnalysis() #Get events where the animal is in the closed arm in Closed_arm_events  $=$  behavior_analysisanimals_object_events('closed arm',  $\leftrightarrow$  'overlap', bodyparts=['all'], negate=False) #Get the start and end times of each event start-times  $=$  [event.start_frame for event in in Closed_arm_events['animal0']) end-times  $=$  [event.end_frame for event in in Closed_arm_events['animal0']) #Return the time frame of the first event if it exists if start-times: return (start(times[0], end(times[0])   
#Occurs 1 times   
def taskProgram(): behavior_analysis  $=$  AnimalBehaviorAnalysis() #Get events where the animal is in the closed arm in Closed_arm_events  $=$  behavior_analysisanimals_object_events('closed arm',  $\leftrightarrow$  'overlap', bodyparts=['all'], negate=False) #Get the start and end times of each event event-times  $=$  [(event.start_time(), event.end_time()) for event in  $\leftrightarrow$  in Closed_arm_events['animal0'])] #Return the event times return event-times if event-times else []
#Occurs 5 times
def task_program():
    behavior_analysis = AnimalBehaviorAnalysis()
    # Get events where the animal is not in the closed arm
    not_in Closed_arm_events = behavior_analysis.animals_object_events('closed arm',
                   $\leftrightarrow$  'overlap', bodyparts=['all'], negate=True)
    # Calculate the duration of each event
    durations = [event.duration() for event in not_in Closed_arm_events['animal0']
    # Return the total duration
    return sum(durations) if durations else 0.0
Code generated by AmadeusGPT with Rephraser
#Occurs 4 times
def task-program(): behavior_analysis  $=$  AnimalBehaviorAnalysis() mouse_center_in_roi_events  $=$  behavior_analysis.animals_object_events('ROI',  $\nrightarrow$  'overlap'，bodyparts  $= [{}^{\prime}$  mouse_center']，negate  $=$  False) neck_in_roi_events  $=$  behavior_analysis.animals_object_events('ROI'，'overlap'，  $\nrightarrow$  bodyparts  $= [{}^{\prime}$  neck']，negate  $=$  False) head_midpoint_not_in_roi_events  $=$  behavior_analysis.animals_object_events('ROI'，  $\nrightarrow$  'overlap'，bodyparts  $= [{}^{\prime}$  head_midpoint']，negate  $=$  True) dips_events  $=$  Event.add_simultaneous_events(mouse_center_in_roi_events,  $\nrightarrow$  neck_in_roi_events，head_midpoint.not_in_roi_events) return dips_events   
#0occurs 1 times   
def task-program(): behavior_analysis  $=$  AnimalBehaviorAnalysis() mouse_center_in_roi_events  $=$  behavior_analysis.animals_object_events('ROI'，  $\nrightarrow$  'overlap'，bodyparts  $= [{}^{\prime}$  mouse_center']，negate  $=$  False) neck_in_roi_events  $=$  behavior_analysis.animals_object_events('ROI'，'overlap'，  $\nrightarrow$  bodyparts  $= [{}^{\prime}$  neck']，negate  $=$  False) head_midpoint_not_in_roi_events  $=$  behavior_analysis.animals_object_events(' ROI'，  $\nrightarrow$  'overlap'，bodyparts  $= [{}^{\prime}$  head_midpoint']，negate  $=$  True) dips_events  $=$  Event.add_simultaneous_events(mouse_center_in_roi_events,  $\nrightarrow$  neck_in_roi_events，head_midpoint.not_in_roi_events) fig，ax  $=$  behavior_analysis.plot_trjectory(bodiesparts  $= [{}^{\prime}$  all']，events=dips_events) return fig，ax   
#0occurs 5 times   
def task-program(): behavior_analysis  $=$  AnimalBehaviorAnalysis() faster Than_8_events  $=$  behavior_analysis.animals_state_events('speed'，'>8') return faster Than_8_events   
#0occurs 4 times   
def task-program(): behavior_analysis  $=$  AnimalBehaviorAnalysis() freezing_events  $=$  behavior_analysis.animals_state_events('speed'，'<0.5') return freezing_events   
#0occurs 1 times   
def task-program(): behavior_analysis  $=$  AnimalBehaviorAnalysis() in Closed_arm_events  $=$  behavior_analysis.animals_object_events('closed arm'，  $\nrightarrow$  'overlap'，bodyparts  $= [{}^{\prime}$  all']，negate  $=$  False) in Closed_arm_duration  $=$  in Closed_arm_events.duration() return in Closed_arm_duration   
#0occurs 4 times   
def task-program(): behavior_analysis  $=$  AnimalBehaviorAnalysis() in Closed_arm_events  $=$  behavior_analysis.animals_object_events('closed arm'，  $\nrightarrow$  'overlap'，bodyparts  $= [{}^{\prime}$  all']) return in Closed_arm_events   
#0occurs 4 times   
def task-program(): behavior_analysis  $=$  AnimalBehaviorAnalysis()
outside Closed_arm_events = behavior_analysis.animals_object_events('closed arm',  $\leftrightarrow$  'overlap', bodyparts=['all'], negate=True) return outside Closed_arm_events   
#Occurs 1 times   
def task_program(): behavior_analysis  $=$  AnimalBehaviorAnalysis() outside Closed_arm_events  $=$  behavior_analysis.animals_object_events('closed arm',  $\leftrightarrow$  'overlap', bodyparts=['all'], negate=True) outside Closed_arm_duration  $=$  {animal: [event.duration() for event in events] for  $\leftrightarrow$  animal, events in outside Closed_arm_events.items()} return outside Closed_arm_duration
Additional stress-testing with naive users and GPT4
We additionally stress-tested the system with naive users. We released a fully usable web App to AmadeusGPT using GPT-3.5 with the 3 demo datasets used in the submitted manuscript (EPM, MABe, and Haus videos) and asked neuroscience behaviorists to test it - namely, we added example prompts they could run and a chat-interface to ask their own questions. Here, we now collected extra prompts not generated by the authors for testing and we will include this in the revision. In brief, we sampled 30 naive user prompts at random out of the 362 submitted via our App (prompts having caused a programming error are marked with a  $\mathbf{X}$ ), and found a roughly  $18\%$  error rate with GPT3.5 (see more below for GPT4, which was only  $10\%$ ):
- X Get angles and distances between all body parts. Plot a UMAP graph using the resulting data.
- Give me the duration of time the animal engages with the object I picked and the events where the animal overlaps with the object I picked.
- X Perform hierarchical clustering and plot the dots with different colors based on their clusters.
- Define  $\langle |\mathrm{sap}| \rangle$  as a behavior where the animal's body length elongates. Do you see sap in the epm?
- Give me the tracking of all 3 animal's noses with each nose being shown in a different color.
- Define  $< |peeking|>$  as a behavior where the animal's head_midpoint is not in ROI5 or ROI4 and the mid_back point is in ROI5 or ROI4.
- X Plot the distance between animals over time.
- X What is the speed of the changes from the left to the right arm?
- Plot regions of interest (ROIs).
- X Define  $<$  |relative_head_angle| > as the angle between the mouse_center and the head_midpoint. Plot the variation of  $<$  |relative_head_angle| > over time.
- How many animals are there in this data?
- Give me the keypoints.
- Give me the total distance traveled by each animal.
- Define  $< |watch| >$  as a social behavior where distance between animals is less than 260 and larger than 50 and head angle between animals is less than 15. The smooth_window_size is 15.
- X Plot animal center of mass x-coordinate, velocity, acceleration, and head direction.
- Define  $< |freezing| >$  as a behavior where the animal is in the same position for longer than 10 seconds.
- Plot a figure comparing the time the animal spends in the open arm (ROI0) and the time the animal spends in the closed arm (outside ROI0).
- X Plot a bar graph with the first bar representing the total time the animal spends in ROI0 (open arm) and the second bar representing the total time the animal spends outside of ROI0 (closed arm).
- Calculate the number of successful spontaneous alternations using a sliding window calculation. A successful spontaneous alternation is defined as when, in the last 5 arm entries, all four arms (ROI0, ROI1, ROI2, and ROI3) are visited at least once.
- X Plot the euclidean distance between the nose points of animal0 and animal2 over time.
- Why is the angle limited to 180 degrees instead of 360?
- Show which what? Please provide more information or specify your request.
- X Define  $< |head\_direction| >$  as the orthogonal angle to the line between left_ear and right_ear.
- Give me the amount of time the animal spends in ROI0 and the events where the animal overlaps ROI0.
- Calculate head direction using the ear points and plot it in radians.
- X Plot each grooming bout.
- Define  $< |head\_ \text{scans}|>$  as a behavior where the animal's head-direction shifts by more than 15 degrees to one side and then the other within 5 seconds. Give events where head_scans happen and bouts for head_scans.
- "Find events where there are errors in tracking due to outliers in mean distance between all points, animal acceleration, and velocity. Remove the labeled point causing the error and fill in missing values by interpolation. Plot the x and y coordinates of the animal's center of mass before and after the tracking correction.
- Give me the frequency of rearing events for the animal.
- Print overlap_roi0_events and its items.
We analyzed the logs from 362 queries, 242 of which were from unique prompts (i.e., from manually typed ones rather than executing the demos). Out of the 362 queries, 329 were automatically rephrased. We found AmadeusGPT had reported 129 "errors": 38 were caused by unsupported features or undefined variables, and were thus explained to the user, while 64 originated from programming errors. Therefore, from 362 queries there were an  $18\%$  programming error rate, and  $11\%$  unsupported feature requests rate.
Then, we took 10 of the failed runs from external users and re-ran them with GPT4. Now, of the 10 that previously failed, only 3 failed, and 4 correctly output clarifying queries to the user*; This means that if GPT4 was used, given the randomly sampled prompts, the error rate would be only 10
- X Get angles and distances between all body parts. Plot a UMAP graph using the resulting data.
- Perform hierarchical clustering and plot the dots with different colors based on their clusters
- Plot the distance between animals over time.
- What is the speed of the changes from the left to the right arm?
- Define  $<$  |relative_head_angle| > as the angle between the mouse_center and the head_midpoint. Plot the variation of  $<$  |relative_head_angle| > over time.
- X Plot animal center of mass x-coordinate, velocity, acceleration, and head direction.
- Plot a bar graph with the first bar representing the total time the animal spends in ROI0 (open arm) and the second bar representing the total time the animal spends outside of ROI0 (closed arm).
- Plot the euclidean distance between the nose points of animal0 and animal2 over time.
- X Define  $< |head\_direction|>$  as the orthogonal angle to the line between left_ear and right_ear.
- Plot each grooming bout.
*For (2), (4), (5), (10), the output is a message indicating either that the feature is not implemented or that the query does not provide all information, and explains what would be needed either in terms of development or clarifications from the user.
Example from Explainer
Here is an example query on the EPM dataset and relevant outputs for the explainer module:
Before rephrasing: How much time does the animal spend in the closed arm which is ROI0?
---- function code ----
behavior_analysis = AnimalBehaviorAnalysis()
overlap Closed_arm_events = behavior_analysis.animals_object_events('ROI0',  $\leftrightarrow$  'overlap', bodyparts=['all'])
generated explanation
1) The code [behavior_analysis = AnimalBehaviorAnalysis()] creates an instance of the  $\hookrightarrow$  AnimalBehaviorAnalysis class, which is a tool for analyzing animal behavior.
The next line `overlap Closed arm events =
$\hookrightarrow$  behavior_analysisanimals_object_events('ROI0','overlap',bodyparts=['all'])] uses
$\leftrightarrow$  this tool to find events where the animal overlaps with the closed arm region of
$\hookrightarrow$  interest (ROI0). It does this by calling the [animals_object_events] method of the
$\hookrightarrow$  AnimalBehaviorAnalysis class, passing in the parameters 'ROI0', 'overlap', and
$\hookrightarrow$  ['all']. This method uses the API documentation to access the necessary data and
$\hookrightarrow$  perform the analysis, returning the events where the animal overlaps the closed arm.
2) The meta answer is: For animal0, the total queried events last 7884 frames (total
$\hookrightarrow$  video length is 15076 frames).
This means that out of the entire video, there were 7884 frames where the animal
$\hookrightarrow$  overlapped with the closed arm ROI0. The total video length is 15076 frames, so the
$\hookrightarrow$  queried events make up a significant portion of the video.
Analyzing this in terms of animal behavior, it suggests that the animal is spending a
$\hookrightarrow$  considerable amount of time interacting with or being near the closed arm.
Example from Self-correction
Here is an example query on the MausHaus dataset and relevant outputs for the self-correction module:
Query: Plot trajectory of the center of the mouse body
behavior_analysis = AnimalBehaviorAnalysis()
traj_plot_info = behavior_analysis.plot_trjectory(['center'])
ValueError: center not defined in the data. Supported bodyparts are ['nose', 'left_ear',
$\hookrightarrow$  'right_ear', 'left_ear_tip', 'right_ear-tip', 'left_eye', 'right_eye', 'neck',
$\hookrightarrow$  'mid_back', 'mouse_center', 'mid�建end', 'mid�建end2', 'mid�建end3',
$\hookrightarrow$  'tail_base', 'tail1', 'tail2', 'tail3', 'tail4', 'tail5', 'left_shoulder',
$\hookrightarrow$  'left_midside', 'left_hip', 'right_shoulder', 'right_midside', 'right_hip',
tail_end', 'head_midpoint']
Diagnosis: The error message indicates that the variable "center" is not defined in the
$\leftrightarrow$  data. The supported body parts for plotting the trajectory are listed in the error
$\hookrightarrow$  message.
To fix this issue, you need to make sure that the data contains the "center" body part.
$\hookrightarrow$  You can check if the data has the required body part before plotting the trajectory.
revised function code:
traj.plot_info = behavior_analysis.plot_trjectory(['mouse_center'])
return traj_plot_info
Horse Gait Analysis
Horse gait analysis demo
Could you tell me what the duty factor of the Offfrontfoot is and explain its value to me?
Chat Output:
The duty factor of the Offfrontfoot is greater than 0.5, indicating that the animal spends more time in the stance phase with its front foot than in the swing phase.
"ok! Could you also calculate the duty factors using the keypoint 'Offhandfoot', and compare both?"
def task_program()
behavior_analysis = AnimalBehaviorAnalysis()  
offfrontfoot_analysis = behavior_analysis.run_gait_analysis(limb_keypoint_names=['Offfrontfoot'])  
offhindfoot_analysis = behavior_analysis.run_gait_analysis(limb_keypoint_names=['Offhindfoot'])  
offfrontfoot_duty Factors = offfrontfoot数据分析["duty Factors"]  
offhindfoot_duty Factors = offhindfoot数据分析["duty Factors"]  
comparison = np.mean(offfrontfoot_duration) - np.mean(offhindfoot_duration)  
return comparison
"Could you explain what that means?"
The duty factor is a measure of the amount of time a limb spends in contact with the ground during a gait cycle. It is calculated by dividing the stance duration (time when the limb is in contact with the ground) by the stride duration (time for a complete gait cycle).
By comparing the duty factors of the Offfrontfoot and Offhindfoot, we can determine if there is any difference in the amount of time these limbs spend in contact with the ground. A positive value indicates that the Offfrontfoot has a longer duty factor, meaning it spends more time in contact with the ground compared to the Offhindfoot. A negative value indicates the opposite, with the Offhindfoot having a longer duty factor."
2:Query:
"Could you rerun the previous analysis but plot it this time? The limb is defined with the following keypoints: Offfrontfoot; Offfrontfetlock; Offknee; Elbow and Shoulder."
Figure 8: Horse Gait Analysis. Examples of user queries and AmadeusGPT outputs, including example of the Web App interface. Data adapted from [36].
behavior_analysis  $=$  AnimalBehaviorAnalysis()   
analysis  $=$  behavior_analysis.run_gait_analysis(limb_keypoint_names  $\equiv$  ["Offfrontfoot)])   
fix,ax  $=$  behavior_analysis.plot_gait_analysis_results( analysis, limb_keypoint_names  $\equiv$  ["Offfrontfoot, Offfrontfetlock, Offknee, Elbow, Shoulder"]
return fig, ax