Animal-Bench: Benchmarking Multimodal Video Models for Animal-centric Video Understanding
Authors: Yinuo Jing
Ruxu Zhang
Kongming Liang
Yongxiang Li
Zhongjiang He
Zhanyu Ma
Jun Guo
Venue: NeurIPS 2024 poster (2024)
Keywords: Multimodal video model
Evaluation benchmark
Robustness testing
Primary area: evaluation

Abstract: With the emergence of large pre-trained multimodal video models, multiple benchmarks have been proposed to evaluate model capabilities. However, most of the benchmarks are human-centric, with evaluation data and tasks centered around human applications. Animals are an integral part of the natural world, and animal-centric video understanding is crucial for animal welfare and conservation efforts. Yet, existing benchmarks overlook evaluations focused on animals, limiting the application of the models. To address this limitation, our work established an animal-centric benchmark, namely Animal-Bench, to allow for a comprehensive evaluation of model capabilities in real-world contexts, overcoming agent-bias in previous benchmarks. Animal-Bench includes 13 tasks encompassing both common tasks shared with humans and special tasks relevant to animal conservation, spanning 7 major animal categories and 819 species, comprising a total of 41,839 data entries. To generate this benchmark, we defined a task system centered on animals and proposed an automated pipeline for animal-centric data processing. To further validate the robustness of models against real-world challenges, we utilized a video editing approach to simulate realistic scenarios like weather changes and shooting parameters due to animal movements. We evaluated 8 current multimodal video models on our benchmark and found considerable room for improvement. We hope our work provides insights for the community and opens up new avenues for research in multimodal video models. Our data and code will be released at https://github.com/PRIS-CV/Animal-Bench.

Animal-Bench: Benchmarking Multimodal Video Models for Animal-centric Video Understanding
Yinuo Jing', Ruxu Zhang', Kongming Liang'; Yongxiang Li², Zhongjiang He? Zhanyu Ma', Jun Guo
School of Artificial Intelligence, Beijing University of Posts and Telecommunications CCina Telecom Artificial Intelligence Technology Co. Ltd
{jingyinuo, zhangruxu, liangkongming, mazhanyu, (liyx25,
edu. cn
Abstract
With the emergence of large pre- pre-trained multimodal video models, multiple bench- marks have been proposed to evaluate model capabilities. However, most of the benchmarks are humann-cnttic with evaluation data and tasks centered around human applications. Animals are an integral part the natural world, and animal- centric video understanding is crucial for animal welfare and conservation efforts. Yet, existing benchmarks overlook evaluations focused on animals, limiting the application of the models. To address this limitation, our work established unimal-centric benchmark, namely Animal- -enchh, to allow for a comprehensive evaluation of model capabilities in real-world contexts, overcoming agent-bias in previous benchmarks. Animal-Bas -ench includes 13 tasks encompassing both com- mon tasks shared with humans and special tasks relevant to animal conservation, spanning 7 major animal categories and 819 species, comprising total of 41,839 data entries. To generate this benchmark, animals and proposed an automated pipeline for animal-centrie data processing. To further validate the robustness of models against real-world challenges, we utilized a video editing approach to simulate realistic scenarios like weather changes and shooting parameters due to animal movements. We evaluated 8 current multimodal video models on our benchmark and found considerable room for improvement. We hope our work provides insights for the community and opens up new avenues for research in multimodal video models. Our data and code will be released at https://gi giggttht..... www./cosc//./eee. CV/ Animal-Bemeeeaeeee
defined task system centered
Introduction
With the rapid advancement of artificial intelligence technology, multimodal video large models I 3 5
have bridged the gap between video and language modalities. Leveraging extensive knowledge and powerful comprehension capabilities, these models are being applied across various areas, ushering in new era of intelligence. The emergence accompanied by the development of new evaluation benchmarks. In contrast to conventional single- task benchmarks, the new henchmarks incorporate a variety of tasks and seek evaluate the model's intelligence across multiple dimensions. Current evaluation studies [3] 9 I0 [1 12 13] primarily
new era of intelligence is also
arise from human daily needs, with a focus on human-centric application tasks, aiming to assess
Corresponding author
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Previous Benchmarks TREALARA
Animal-Rence
Limited Agent
Various Animal Agent
E.g.
47%
S2%
mainly Humans & Otgects
Various und Reulistic Scenarios
Objizt
ONLY 1% animal datn
Simplified Unrealistic Scemarios
Ediiing-bsed benchmarks are ISITALI and have limitations PRACTICAL applications
Orgind
Prost
bireztion
Various Tasks
Common Tasks
Special Tasks
Okjeet
Artian
Time
Coner
Resoeiin
No Mecakctng
Figure Previous benchmarks (left) relied on limited agent and the scenarios of editing-base benchmarks unrealistic. Our proposed Animal-Bec Bench (right) includes diverse animal agents, various realistic scenarios, and encompasses 13 different tasks.
model performance in common tasks encountered in human life. However, this approach confines their applicative scope tasks centered around humans. In the broader real-world context, animals represent indispensable constituents of ecosystems (14]. They participate in vital processes such as pollination, seed dispersal, and nutrient cycling, which essential for environmental conservation and the maintenance of biodiversity [15] 16, 17). However, evaluations that focus on animal- centric tasks entirely divergent from the current frameworks. As illustrated in Table [ existing benchmarks that comprehensively evaluate model capabilities predominantly emphasize humans or objects, whereas animal-centric evaluation datasets can only assess the model's performance few aspects. As shown in Figure I taking the popular benchmark MVBench ( as an example, the main agents in the videos humans and objects, with animal data accounting for only 1%. The inherent agent bias in these comprehensive benchmark datasets hampers our understanding of large models' ability to comprehend animal agents.
Research focusing on animal-centric evaluation overcomes the agent bias present in previous bench- marks, allowing us assess model capabilities in broader real-world framework, further exploring the potential applications of models and providing more valuable guidance for model optimization. The inherent diversity among animal species and the complexity of their habitats result in rich variability within animal videos, making animal-centia tasks highly challenging [18]. Evaluations on these demanding tasks can reveal the weaknesses of models in complex environments and analyze the robustness of multimodal video models against significant intra-class variations. Moreover, the applications of artificial intelligence the field of animal studies are extensive 19
20
21
22
24
26 27 aids in tracking population dynamics in natural reserves, assessing overall ecosystem health, and significantly reducing human effort. Automated
For instance, automated species counting 255
detection of animal stress and pain 228 29 30 31
33 enables timely identification of potential issues, facilitating early treatment and ensuring animal welfare. Therefore, evaluations of models focused on animals are beneficial for further advancing the practical applications of artificial intelli- gence in the animal world. In summary, conducting animal-centric model evaluation plays a pivotal role in both model development and conservation efforts.
32
In this work, we propose Animal-Bench Bench, benchmark for evaluating multimodal video models in animal-centric video understanding. We choose tasks from two broad aspects: common tasks shared with human-centre benchmarks, covering aspects such as "object", "action", "time", "count", and "reasoning", and special tasks relevant to animal conservation. In total, we include 13 tasks spanning 7 major animal categories and 819 species, comprising total of 41,839 data entries. To construct Animal-Benche we devise a pipeline for automated data filtering and question-answer pair generation, reducing human effort and mitigating potential biases from human intervention. Furthermore, since our data primarily originates from the web, which typically features favorable

---

recording conditions, while real-world filming scenarios may involve harsh weather conditions such as snowy or frosty, or changes in shooting parameters, such as variations in camera distance and direction due to animal movements. To accommodate potential variations in model applications and simulate real-world filming conditions, we employ video editing methods for simulation, As illustrated in Figure previous editing- baaed evaluation benchmark (34] generated images that did not correspond to real situations and were not applicable to our setting. We utilize a video editing approach based on the diffusion model [35] to simulate videos captured in realistic scenarios, thereby evaluating the robustness of multimodal video models. We evaluate 8 existing multimodal video models our benchmark, identifying significant room for improvement. We hope our work can inspire ad advancements in multimodal video models development.
Dataset Properties
Benchmarks
Tasks
Label
QA Size
Agent(main)
object, action, attributic position, count, time, reasoning, summarization, etc.
Video-MME 36
Multi-Choioo QA
2700 QAs
Human & Object
action, object, attribute, position, count, reasoning, etc.
Video-Bench
I
Multi- Multi-Choice QA
15,033 QAs
Human & Object
action, object, position, coant, secne, posc, attribute, character and cognition
MVBench X
Multi-Choice QA
4000 QAs
Human Object
Animal Kingdom
Classification
object, action,
MammalNet 38
Classification
N/A
Animal
object, action
Common object, action, time, count, reasoning
Spocial tasks: predator-prey behavior monitoring, social interaction analysis, breeding behavior monitoring and stress and pain detection
Animal-Bench
Multi-Choioe QA
41,839 QAs
Animal
Table I: Comparison of existing video understanding benchmarks. In contrast to other benchmarks, Animal-Bench mitigates the limitations of prior video sovetiin-answeeet benchmarks that lack animal agents. The dataset is characterized by its richness and diversity, facilitating comprchensive evaluation of models across multiple dimensions of performance.
2 Related Work
MLLM Benchmarks Traditional evaluation benchmarks [39 40
41
typically only test 42
the ability of models by a single task. With the rise of multi-modal large language models (MLLMs), aimed to cover wider range of evaluation aspects. For instance, Video-Bench [11] categorized the video language models' comprehension abilities into three levels: video-exclusive understanding, prior knowledge-based question-ansseerigg, and comprehension and decision-making. The Perception test (45) focused on skills (memory, abstraction, physics, semantics) and types of reasoning (descriptive, explanatory, predictive, counterfactual) across video, audio, and text modalities. These benchmarks all evaluated whether the models' capabilities were comprehensive enough by carefully choosing and dividing tasks, while overlooking the importance of data selection for evaluation. Other works considered how data selection influenced the evaluation process. For Image-LL.N LLMM evaluation, MMBench 10] hierarchically subdivided the MILM models' perception and cognition abilities and redesigned the QA pairs to better reflect capabilities of the models. For Video-LLM evaluation, MVBench B considered 20 challenging video tasks in its evaluations,
new benchmarks 44 13
selecting 200 test instances from open datasets for each task and redesigning the QAs. However, most existing benchmarks used human-centric data, predominantly featuring humans as the main agents and neglecting others. Our Animal-Bence believes that MILMss should demonstrate good generalization abilities across different agents, thus designing a new benchmark and assessing the performance of MILMs animal-cente cemmal-centric data.
Editing- based Benchmarks recent years, some works have employed editing techniques to pro- cess benchmarks, enhancing data diversity and evaluating model robustness. Regarding benchmarks based on image editing, LANCE (46) utilized image editing techniques to augment the test set with a suite of diverse, challenging, yet counterfactual examples for diagnosing the image recognition abilities of different models. D. Hendrycks's work [47 established rigorous benchmark for testing image classifier robustness by introducing diverse types of corruptions and perturbations, including
3

---

noise, blur, weather effects, and digital distortions. For benchmarks based on video editing, Grover et al. 48] established an occluded dataset and further developed the benchmark explore the impact of occlusion action recognition models. Schiappa et al.'s work 49] proposed a robustness analysis by introducing 90 perturbations that reflected different real-world distribution shifts in their bench- mark, offering insights into robust video action recognition. However, most of these works were counterfactual or only considered the effects of camera disturbances from the camera's 's viewpoint, neglecting factors such scene characteristics (e.g., shooting distance and direction) that are likely to be encountered real filming scenarios. Inspired by the aforementioned studies, Animal- -eechhh employs video editing techniques create animal videos under varying weather conditions and shooting parameters, presenting new demands and challenges for model robustness.
Methods
In this section, we will introduce the details of our Animal-Bench In the first part, describe how our Animal-Bench is designed in terms of task definition and the automated pipeline of data processing. The second part details our approach editing videos in our benchmark, aiming to simulate realistic scenarios.
Common Tasks
Count
Object
Action
(A
Object
Ia Eane dis video? (Armcia
(233
(B)
How amadora leepad
attck?
Wha but appuaas
Ofject
Reoognition
Object
LAV
(C' usal
(lI) sume spider
(O)
leeger
Remoning
Why jueping
Action
LA) getinggggg
Whne is artose performed by the aniea
Abductive
playing
Action Recegaition
Driug
begging for fool
(Di Scoring
Special Tasks
Action Sequenee
(AX Lyinggg insile
eaaned eatay?
Prodatur-
uaeud this vidao
(B0 Spiting Vaxce
Pray
Bchavior Monitering
Wise will oeceiiel neneett 5ide Tit
the villo loviing dgeing (Rims
Social
Action Prediction
(PC wakkin
(C) Claziia
(C)yo
Becedleg Moaiturint
Time
During winch par: of does flying
(C:00
antmal this vidao
Actiun Loculization
Strre and Paln Detection
FIE7
(BI) the raidil the vidoo
(C1 Theragaon the entre vidao
(D) boginning video
Ticarco Infurder Ludinod uodiiiied Infecie
Figure 2: Example demonstrations of each task in Animal-Bench
3.1
Animal Bench: Animal-Centric Evaluation
3.1.1
Animal-Centric Tasks System
To evaluate the perceptual and cognitive abilities of multimodal large models on data where animals serve as the main agents, our Animal-Bench Bench redefines tasks that were previously overlooked in human-centric benchmarks. First, we consider the common tasks from human-centric evaluations to assess the models' abilities "object", "action". "time", "count". and "reasoning" on videos featuring animals. Additionally, from the perspective of application value, we identify specific tasks related to animals that are of greater interest to zoologists. For instance, we focus on models' ability to detect predator-pin behaviors (50), social interactions (51), breeding behaviors [52), and stress and

---

thereby promoting better research and protection of animals. As shown in Figure 2 we designed the following tasks:
pain [53],
Common Task Perception: Object. (1) Object Existence(OE): Judge whether a certain item exists during a particular video; (2) Object Recognition Determine the specific class of the object that appears in the video. Action. (3) Action Recognition(AR); Recognize the action performed by the animal based on a piece of video; (4) Action Sequence (AS): Infer the action of an animal before or after a certain action in chronological order; (5) Action Prediction(AP): Given a specific action and its starting and finishing time in the video, predict the subsequent action performed by the animal; (6) Action Localization(AL); Assess the start and end time of a specific action performed by the animals in the video. Count. (7) Action Count( AC): Calculate how many times an action has been performed in the video; (8) Object Count(OC): Calculate how many times an object appears in the video. Cognition: (9) Reasoning(RS); Logically infer why an event or a certain scenario occurred in the video.
Specific Task (10) Predator-Prey Behavior Monitoring(PMM) Detecting the interactions between predators and their prey that influence survival strategies, such as hunting techniques and evasion lactics; (11) Social Interaction Analysis(SA)) Analyzing behaviors occurring between animals that affect social structure, communication, and cooperation: (12) Breeding Behavior Monitoring(BMI: Monitoring activities related to reproduction, including mating and caregiving for offspring; (13) Stress and Pain Det) Detecting physiological and psychological responses to harmful stimuli or adverse conditions that impact the welfare and behavior of animals.
3.1.2 Animal-Centric Data Processing Pipeline
Various Data
Data Fiftering
Question-Answer Pair Generation
Data Diverdty:: Asimal comtie data
Antomated Question Georration
from TGII-QA, MSRVTI-QA, Nia/--QA ManmmaNeet, Animal Kinglem and LaTT- aninal
ChargPt gauntt typos Eqqectionn and modemly secign
Evaluation Results
Temporal Sunsitivity
Various Tasks (Commun Spocial)
rime ansotations:
the Animal Kingdom daznet
Task- Besed Option Design:
Random selected
Vidzo dusaiinn limised secomde for
Tipe simple difficut
timeechaaal
Figure 3: The diagram of our unimal-centric data processing pipeline: Firstly, choose dataset and identify tasks, then establish rules to filter data, and finally automatically generate QA pairs.
We develop an automated pipeline for processing animal-centric data. Initially, we conduct data filtering on the existing dataset according to predefined task definitions. Subsequently, we formulate rules to automatically generate questions and options for all data, culminating in a dataset structured in the multiple-choice question answering (QA) format. The rationale for adopting the multiple-choice
QA format [54] 55 [6) is twofold. Firstly, regarding difficulty, this format alleviates the challenge of delineating the scope of options inherent in open-ended QA formats, while still presenting a substantial challenge to capabilities of models. Secondly, from the point of evaluation faimesss the multiple-choice QA format facilitates the calculation of final evaluation accuracy and enables effective comparison of different models' performance.
Data Filtering We first select data that aligns with the defined evaluation tasks. The principles for data selection are as follows. (1) Data diversity: Diverse data enables our evaluation to encompass a variety of complex situations, avoiding biases caused by single species or single dataset. (2) Temporal sensitivity: The temporal sensitivity present in the data allows to assess the model's temporal modeling capabilities, as appropriate video lengths facilitate accurate decision recision-making for multi-modal video models.
To ensure the diversity of data, our Animal mnimal-Bench dataset is sourced from 6 datasets, encompassing 7 major animal categories, including mammals, insects, reptiles, amphibians, birds, fishes, and sea animals, totaling 819 different animal species. Specifically, for tasks related to "action", "object" and "time", we primarily get the annotated data for animals, actions, and grounding from MammalNet [38 and Animal Kingdom (37]. For the "count" and "reasoning" tasks, our evaluation data is sourced
from open datasets such TGIF-QA [57] MSRVTT-QA [42), and NExT-QA [40), from which we
5

---

extract substantial amount of data featuring animals as agents. For special tasks, due to the lack of annotated data, we select data with amnotations that match the requirements of the tasks from Animal Kingdom, LoTF-animal [58], and MammalNet (3
To ensure the temporal sensitivity of the data, first select the Animal Kingdom dataset, which contains annotations relevant to ridco-grounding tasks. Specifically, it marks the time intervals of actions, facilitating our evaluation of tasks such as action prediction, action sequence, and action localization. Additionally, for tasks other than "object," we believe that correct answers cannot be obtained solely from spatial information but require temporal information. For these tasks, we constrain the video duration to between 3 to 35 seconds. This ensures that answering questions requires relevant temporal information without causing confusion for the model due excessively long video lengths. Ultimately, the average duration of our benchmark videos is 14.61 seconds.
QA Pair Generation We have meticulously designed the generation rules for both the questions and the options. Here is the detailed process of our QA pair generation.
Automated Question Generation: For each task's s description, we use ChatGPT [59 to generate 3
types of questions and randomly assign one of the generated questions to each piece of data.
Option Design: (1) Directly adopt from existing QA annotation: For count and reasoning tasks, the options are directly chosen from the annotated multi-choice QA dataset. (2) Automatic option design: For other data without QA annotations, we automatically convert the original annota- tions into multi-choice QA format. For object existence tasks and special tasks, our options were set as "yes", "no", and "not sure". While for other tasks, options besides the right answer should be neither simple nor difficult in order to reflect the real perceptual ability of evaluated MLLMs, thereby our options should not be randomly chosen from the dataset. Specifically, for action-related tasks, considering the long-tailed distribution in the dataset, our four options consist of the correct answer, two options from the top 50% of most frequent answers, and one option from the least frequent 50% answers. For object recognition task, besides the right answer, two options are sampled from different major animal categories and one option is sampled from the same major animal category. This design ensures a balance in options. Once the options are set, they are randomly shuffled to ensure robustness of our evaluation. For a discussion the option design rules, please refer to Appendix B
3.2 Realistic simulation based on video editing
First, we select different aspects to simulate, as follows:
Weather conditions. Weather changes common during outdoor filming. In this work, choose snowy and frosty weather as the simulated conditions. Snow is a form of precipitation that visually obstructs. Frost forms when ice crystals coat lenses or windows.
Shooting parameters. Due to the camera's 's placement in the natural habitat of animals, the shooting parameters may vary due to animal movements. In this work, we select shooting distance (affected by the movements of animals resulting in changes in proximity to the camera) and shooting direction (affected by animal collisions resulting in camera tilt) as the simulated shooting parameters.
Simulating variations in outdoor weather conditions We take simulating snowy weather as an example. First, we assume that the snow layer S is an image with random noise, which follows a normal distribution S ~~ ((,,, Next, we resize the image to make the density and distribution of snow more uniform. Since snowflakes do not accumulate in dark areas but only in well-lit areas when they fall on an object's 's surface, we set a threshold and remove parts below it. Specifically, we can express it as:
(s00((S.f) if zoom 2 t. s' (0
otherwise
Finally, apply blur effect the snow layer to soften its edges. The blur operation can be expressed as:
Suturrea(I, 2T
20
. 5'(x (x--inh -,
(2)

---

where, Sotured (2. y) denotes the pixel value after blurring, y denotes the blur radius. o represents the standard deviation of the Gaussian kernel, i and represent the indices of the convolution kernel.
Simulating variations shooting parameters We aim to simulate different shooting distances from the camera to animals, as well as shooting directions. To simulate proximity, we achieve this through central cropping of each frame. However, when simulating remoteness or different shooting directions, face the challenge of lacking broader context outside the current frame. To address this issue and enhance the realism of our video evaluations, we leverage the capabilities of the Diffusion model [60] perform outpainting for regions beyond the original scene. Furthermore, have developed an automated video editing pipeline to streamline this process.
Difflsiion Model
Guided Frame Selection
Frame Bllndding
M
Encoder
Decoder
Figure 4: The diagram of simulation process for shooting parameters, Firstly, the transformed images along with their masks are encoded, and then passed through the diffusion model for denoising. After decoding, the final simulated video is obtained through the guided frame selection module and frame blending module.
For a video frame E X we perform scaling or rotational transformations on using an orthogonal matrix. Scaling and rotation matrices can be represented as follows,
scale O
cos
-sin 0
Tss
TR
scale
sin A cos
Then we create a new blank canvas I' E R3xHx which has the same center position as the original image. Then the part of the original video frame that remains on the new canvas after transformations is:
((''./) {f (T (x' if' is within the bounds of f 10.
(4)
otherwise.
We encode f and the mask of f using variational autoencoder [61] to derive their respective image embeddings y within the latent space. Subsequently, these embeddings, accompanied by initially sampled noise 27 ~ NO0 drawn from standard normal distribution, are jointly fed into denoising process. The reverse diffusion process formulated as:
Br
(5) where aa and are predefined diffusion coefficients. c0 L, y) is the predicted noise. a, is the noise standard deviation at each step. After obtaining the final latent space representation Zo, the image is generated through the decoder, Xo D(zo).
Z
Z,
_ 2, + O,Z, z ~1,,
VI
Due to the varying quality of outpainting generated by diffusion for each vidco frame, with some frames showing significantly better results than others, observe that frames with smoother pixel transitions at the edges of the original image tend to have better outpainting quality. We hypothesize that the highest values in the image spectrum originate from the edges of the original outpainted image. Therefore, we select the frame with the smallest highest spectrum value as the guide image.
f arg min max(F(fa)
for ft E
(6)
If directly using the outpainting part in the f would cause slight misalignment the edges of other frames. Inspired by (34), we blend images at different noise levels along the diffusion process using diffusion models. Starting from the outpainting part f at each stage, we perform a guided diffusion step with latent variable J to obtain f while simultaneously obtaining noisy version ft-1 1 of the original frame f. The fi-1 generated at this stage a blend of two latent variables using a mask M. represented follows.
f- M * f-- (1 M) f
7

---

Experiments
In this section, we will sequentially introduce the details of our experimental implementation, the effectiveness of existing multimodal video models on our proposed evaluation benchmark, and their robustness on our editing-based realistic scenario simulation data. Finally, we will discuss the experimental results, hoping to provide guidance for model optimization.
4.1 Implementation details
We conducted tests for multimodal video models on an NVIDIA RTX 4090 with 24GB of VRAM. To ensure fair comparisons, standardize the 7B LLM backend versions used across all multi-modal video models tested during inference, thereby minimizing discrepancies in language proficiency due to differences in model sizes. Following the methodology outlined in [1, establish uniform system prompt and adopt the prompt-based model output matching strategy. All generated outputs successfully match the corresponding options. For each video, sample 16 frames and resize them to (224, 224). During video editing, we utilize StabbeDiffusion-nnaaannigg to expand the scenes beyond the captured footage and subsequently apply StableDiiffsion-vi.. for noise addition in frame blending.
4.2
Evaluation results
Multimodul Video Model
Random
mPLUG
Video
Vidco
Video
Chat
Video
Video
Task
Valley
64
Avg
95% crffbae manval
-Owl [4
Chai 6
ChatGr 63
-LLaMA 8
-UaiVi 65
-LLaVA +
Char2 B
OF
33.32 + 0.46
42.20
49.40
44.65
49.20
41.70
44.65
45.90
50.00
45.96
OR
24.96 1 0.19
33.62
51.61
24.31
60.23
25.06
43.25
40.55
86.75
45.67
25.26 0.17
24.28
35.34
24.56
32.98
31.71
22.41
27.16
33.19
25,86
AP
24.16 1.31
25.48
27.88
24.52
29.81
28.37
27.88
28.37
50.00
30.29
AL
0.349
24.49
23.25
21.22
24.67
25.45
24.14
24.32
21.22
23 60
OC
25.17
1.18
24.14
27.59
24.71
26.44
25.29
31.61
31,03
64.94
31.97
AC
25.06
0.37
24.43
25.51
22.92
24.99
23.78
21.34
22.19
29.16
24.70
RS
19.46 0.71
22.38
27.07
25.69
35.08
22.65
36.46
21.27
68.23
32.35
Special Task
49.70
43.35
SA
33.22
47.23
42.96
48.18
44.16
46.80
PD
33.15 1.55
40.56
47.55
46.85
50.35
38.16
44.06
45.80
54.20
45.98
Overall Performance
Avg
27.89 2.90
31.83
37.87
32.08
39.54
31.58
37.64
34,54
$3.66
37.34
Table The evaluation results multimodal video models on our Animal-Bench (the first place for each task is marked in red, and the second place is marked in blue, and those below random accuracy are marked in gray).
Effectiveness evaluation results The table presents the evaluation results of eight existing multimodal video models on our Animal-Benchs As models evolve, their performance also improves. The recently released model VideoChat2 3 surpasses previous methods in most tasks. Notably, VideoChat2 achieves an accuracy of 86.75% in object recognition tasks and 68 68.23% in reasoning tasks. However, also observe shortcomings existing models on certain tasks. For instance, in action localization and action counting tasks, the answers provided by existing models are nearly equivalent to random guesses. These tasks typically require strong temporal understanding capabilities, which cannot be inferred solely from spatial scene comprehension. This indicates need for enhancement the temporal modeling abilities of existing models. Additionally, the object existence task, the models tend to respond with "yes", indicating severe hallucination problem.

---

Robustness evaluation results We test the robustness of our models and their sensitivity to different types of variations on four types simulated real-world data. We select the top four models in terms of effectiveness evaluation for robustness testing, as models with lower accuracy tend to provide responses close to random guessing, rendering discussions about robustness less meaningful. As shown in Table 2, VideoChat2 demonstrates relatively good robustness, with an overall decrease in accuracy of 3.70%. However, Video-LLaMA (8] shows sensitivity to the four simulated variations, with overall decrease in accuracy of 8. 8.72%. As depicted in Figure 5, we calculated the average accuracy decrease of the models for the four types of variations, revealing that models are more sensitive to shooting parameters than to changes in weather changes.
Valeo E. UMA
Viled
Weather condition
Shooting parameters
Models
Overall
Snow
Frosi
Distance
Direction
VideoChar2
1.49
2.17
4.76
6.39
3.70
Video-ILaMA
541
7.46
10.82
11.19
8.72
VidenChat
743
5.22
Chat-UniVi
504
181
383
7.86
4.64
Table Sensitivity of multimodal video models to different variations(relative accuracy dr).
Figure 5: Average decrease in model accuracy( (%) across four types of variations.
4.3 Further discussion
6 we demonstrate the effects of our video editing pipeline used for simulating shooting parameters. It can be observed that the guided frames obtained through spectrum filtering have initially achieved good results. However, there still exist unnatural transitions at the edges of the original image. After undergoing the diffusion noise addition process again, the edges of the original image can transition better into the newly generated parts of the image. Since we employed a frozen stable diffusion model, the generation effect relies on the performance of this model.
Video editing samples As shown in Figure
Original
Guided
Final
Original
Guided
Final
frame
frame
Figure 6: The visualization results of simulated changes in shooting parameters. Zoom in to view details.
Animal category bias We aim to investigate whether the models exhibit biases towards different animal categories. We analyze the performance of multimodal video models in two tasks: object recognition and action recognition. We choose these two tasks for analysis due to their involvement with the most diverse range of animal species. As shown in Figure in the object recognition task, models demonstrate higher recognition accuracy for the "mammal" and "bird" categories, while accuracy is generally lower for categories like "amphibian" and "reptile". Similarly, in the action recognition task, models exhibit higher accuracy in recognizing "fish" and "mammal" categories.

---

D VideoChat2 uChat-UniVi Video- LLaMA Video-LLLaV VideoChat CaPPLUG-Oww Valley @ Video-ChatGiP Avvg
Amphibian
Amphibian
Sea animal
Bird
animal
0,30
1,.39
0.34
0,35
0.43
0.28
0.40
Reptile
Fish
Reptilc
Fish
0.49
0.30
Mammal
Insect
Mammal
Insect
Figure 7:
The radar map illustrates the accuracy on "object recognition" and "action recognition" tasks across animal categories.
This could possibly be attributed to the greater prevalence of mammalian and fish species in the pretraining video datasets. Conversely, other categories suffer from lower recognition performance due to larger domain disparities.
Model structure We also analyze on the impact of model architecture on accuracy, we find from the data in Figure 8 that using the frozen CLIP ViT/L-14 (66] as the video encoder resulted in overall performance inferior to models employing larger or fine-tuned video encoders. This suggests that employing more powerful video encoders aids in more comprehensive exploration of video features, which is of significant importance for the development of multi-modal video models. addition, we observe that introducing temporal modeling module into the model architecture is not particularly effective. According to Figure 8] models with an additional temporal modeling module outperform some earlier models in terms of answer accuracy but do not reach the level of some recently released models. This finding suggests that in the model design, the impact of the temporal modeling module may not be as significant expected.
CLIP VIDL-14 Larger fine-tuned video
temporal modeling
temporal modeling
43
VideoCTuat2
Averont Accunry
VirboChat2
Video- LLAVA
VidasChal
LLaMA
Video- LLAVA
Uaat-UniVi
Clar-
ChaGGP
ThdcoChat
LLaMA
mPLIIG-Dul
Valley
Video ChatGPT
@PLUGGOO
Villey
Video-LLMs
Figure 8: The line graph depicting the impact of video encoders and temporal modeling on the results.
Conclusion
In this work, we introduced Animal-Bench an animal-centric benchmark to evaluate multimodal video models in real-world contexts. It includes 13 tasks across major animal categories and 819 animal species. We proposed an automated pipeline of data processing. And we used video editing approaches to simulate realistic scenarios. We evaluated current models on our Animal-Bench, and found significant room for improvement. We analyzed the bias of models towards animal categories and examined the influence of model architecture on experimental results. Our work aims to provide valuable insights and foster new research directions in multimodal video models.
10

---

Acknowledgements
This work supported in part by National Key R&D Program of China (20222D00 16300), and in part by National Natural Science Foundation of China (NSFC) No. 62106022, 62225601, U23B2052, and in part by the Youth Innovative Research Team of BUPT No. 2023YQTDO2, and in part by the Beijing Natural Science Foundation Project No. L242025, and in part by BUPT Excellent Ph.D. Students Foundation No. CX20241086, and in part by scholarships from China Scholarship Council (CSC) under Grant CSC No. 202406470082 and sponsored by Beijing Nova Program.
References
[1] Bo Li, Yuanhan Zhang, Liangyu Chen, Jinghao Wang, Jingkang Yang, and Ziwei Liu. Otter: multi-modal model with in-context instruction tuning.
[2] Feilong Chen, Minglun Han, Haozhi Zhao, Qingyang Zhang, Jing Shi, Shuang Xu, and Bo Xu. X-IIm: Bootstrapping advanced large language models by treating multi-modalities as foreign languages. arXiv preprint arXiv:2305.cm 00.55.0666... 06000, 2023.
[3] Kunchang Li, Yali Wang. Yinan He, Yizhuo Li, Yi Wang. Yi Liu, Zun Wang. Jilan Xu, Guo Chen, Ping Luo, et al. Mvbench: A comprehensive multi-modal video understanding benchmark. arXiv preprint arXiv:23 2..i::311.1120 17005, 2023.
[4] Bin Lin, Bin Zhu, Yang Ye, Munan Ning, Peng Jin, and Li Yuan. Video-Ilava: Learning united visual representation by alignment before projection. arXiv preprint arXiv:2311..01111 2023.
[5] Yixuan Su, Tian Lan, Huayang Li, Jialu Xu, Yan Wang, and Deng Cai. Pandagpt: One model to caatucttionfollow them all. arXiv preprint arXiv:2305. 2023.
[6] Junke Wang, Dongdong Chen, Chong Luo, Xiyang Dai, Lu Yuan, Zuxuan Wu, and Yu-Gang Jiang. Chatvideo: A tracklet-centric multimodal and versatile video understanding system. arXiv preprini arXiv: 14407, 2023.
[7] Junke Wang, Dongdong Chen, Zuxuan Wu, Chong Luo, Luowei Zhou, Yucheng Zhao, Yujia Xie, Ce Liu, Yu-Gang Jiang, and Lu Yuan. Omnivl: One foundation model for image-language and video-language tasks. Advances in neural information processing systems, 55700,, 2022.
[8] Hang Zhang, Xin Li, and Lidong Bing. Video-Ilama: An instruction-tum tuned audio-visual language model for video understanding. arXiv preprint arXiv:2306.0283 2023.
[9] Bohao Li, Rui Wang, Guangzhi Wang, Yuying Ge, Yixiao Ge, and Ying Shan. Seed- bench: Benchmarking multimodal lims with generative comprehension. arXiv preprint arXiv:2307.16125, 2023.
[10] Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al. Mmbench: Is your multi-modal model an all-around player? arXiv preprint arXiv:2307.0022
2023.
[11] Munan Ning, Bin Zhu, Yujia Xie, Bin Lin, Jiaxi Cui, Lu Yuan, Dongdong Chen, and Li Yuan. Video-bench: A comprehensive benchmark and toolkit for evaluating video-based large language models. arXiv preprint ar arXiv:2311.16103 2023.
[12] Peng Xu, Wenqi Shao, Kaipeng Zhang, Peng Gao, Shuo Liu, Meng Lei, Fanqing Meng, Siyuan Huang, Yu Qiao, and Ping Luo. LvIm-ehub: A comprehensive evaluation benchmark for large vision-language models. arXiv preprint 2rXiv:23366.09263 2023.
[13] Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang. and Lijuan Wang. Mm-vet: Evaluating large multimodal models for integrated capabilities. arXiv preprint arXiv:2308.0244: 2023.
[14] Norman Christensen, Ann M Bartuska, James Brown, Stephen Carpenter, Carla d' Antonio, Rober Francis, Jerry F Franklin, James A MacMahon, Reed F Noss, David J Parsons, et al. The report of the ecological society of america committee on the scientific basis for ecosystem management. Ecological applications, 6(3):665-691, 1996.

---

[15] José M Gómez, Jordi Bosch, Francisco Perfectti, Juande Fernández, and Mohamed Abdelaziz. Pollinator diversity affects plant reproduction and recruitment: the tradeoffs of generalization. Occologia, 153:597-605, 2007.
[16] Rafal Zwolak and Andrew Sih. Animal personalities and seed dispersal: a conceptual review. Functional Ecology, 34(7):1294-133) 1310, 2020.
[17] Robert T Paine. Food web complexity and species diversity. The American Naturalist, 100(9 10)6555) 1966.
[18) Yinuo Jing, Chunyu Wang, Ruxu Zhang, Kongming Liang, and Zhanyu Ma. Category-specificc prompts for animal action recognition with pretrained vision-language models. In Proceedings of the s1 ACM International Conference on Multimedia, pages 5716-5724, 2023.
[19] David J Anderson and Pietro Perona. Toward a science of computational ethology. Neuron, 84(1):18 31, 2014.
[20] Liqi Feng, Yaqin Zhao, Yichao Sun, Wenxuan Zhao, and Jiaxi Tang. Action recognition using spatial-temporal network for wild felines. Animals, 11(2))485, 2021.
[21] Jacob M Graving, Daniel Chae, Hemal Naik, Liang Li, Benjamin Koger, Blair Costelloe, and lain D Couzin. Deepposekit, software toolkit for fast and robust animal pose estimation using deep learning. Elife, 8:c47994, 2019.
[22] Hung Nguyen, Sarah J Maclagan, Tu Dinh Nguyen, Thin Nguyen, Paul Flemons, Kylie Andrews, Euan G Ritchie, and Dinh Phung. Animal recognition and identification with deep convolutional neural networks for automated wildlife monitoring. In 2017 IEEE Ini'l Conf. on Data Science and Advanced Analytics (DSAA), pages 40 2017.
[23] Abhineet Singh, Marcin Pietrasik, Gabriell Natha, Nehla Ghouaiel, Ken Brizel, and Nilanjan Ray. Animal detection in man-made environments. In Proceedings of the IEEE Winter Conf. on Applications of Computer Vision, pages 1438-1449, 2020.
[24] Lukas von Ziegler, Oliver Sturman, and Johannes Bohacek. Big behavior: challenges and opportunities in a new era of deep behavior profiling. Neunpsychhopharaccolyy 46(1 46(1):33 44, 2021.
[25] Hüseyin Gökhan Akçay, Bekir Kabasakal, Duygugül Aksu, Nusret Demir, Melih Öz, and Ali Erdogan. Automated bird counting with deep learning for regional bird distribution mapping. Animals, 10(7):1207, 2020.
[26] Dominique Chabot and Charles M Francis. Computer-automated bird detection and counts in high-resolution aerial images: review. Journal of Field Ornithology, 87(4):343-359 2016.
[27] Arturo Gomez Chavez, Jorge Fontes, Pedro Afonso, and Max Plingsthorn. Automated species counting using a hierarchical classification approach. In IEEE Oceans.
[28] Marcelo Feighelstein, Yamit Ehrlich, Li Naftaly, Miriam Alpin, Shenhav Nadir, Ilan Shimshoni, Renata H Pinho, Stelio PI. Luna, and Anna Zamansky. Deep learning for video-based automated pain recognition in rabbits. Scientific Reports, 13(1): 14679,, 2023.
[29] Marcelo Feighelstein, Ilan Shimshoni, Lauren R Finka, Stelio PL Luna, Daniel S Mills, and Anna Zamansky. Automated recognition of pain in cats. Scientific Reports, 12(1):9575, 2022.
[30] Krista McLennan and Marwa Mahmoud. Development of an automated pain facial expression detection system for sheep (ovis aries). Animals, 9(4::196, 2019.
[31] Yamin Han, Jie Wu, Hongming Zhang, Mingyu Cai, Yang Sun, Bin Li, Xilong Feng, Jinye Hao, and Hanchen Wang. Beef cattle abnormal behaviour recognition based on dual-brancl frequency channel temporal excitation and aggregation. Biosystems Engineering, 241:28-42, 2024.
[32] Xin Li, Yinfeng Hao, Muhammad Akhter, Daoliang Li, et al. A novel automatic detection method for abnormal behavior of single fish using image fusion. Computers and Electronics in Agriculture, 203:107435, 2022.
12

---

[33] Melinda A Novak and Jerrold S Meyer. Abnormal behavior of animals in research settings, In Behavioral Biology of Laboratory Animals, pages 27-50. CRC Press, 2021.
[34] Xiaodan Li, Yuefeng Chen, Yao Zhu, Shuhui Wang, Rong Zhang, and Hui Xue. Imagenet-e: Benchmarking neural network robustness via attribute editing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 20371 1-2081,, 2023.
[35] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684-10695, 2022.
[36] Chaoyou Fu, Yuhan Dai, Yondong Luo. Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang, et al. Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal Ilms in video analysis. arXiv preprint arXiv:2405.21075. 2024.
[37] Xun Long Ng. Kian Eng Ong, Qichen Zheng, Yun Ni, Si Yong Yeo, and Jun Liu. Animal kingdom: A large and diverse dataset for animal behavior understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19023-19034, 2022.
[38] Jun Chen, Ming Hu, Darren J Coker, Michael L Berumen, Blair Costelloe, Sara Beery, Anna Rohrbach, and Mohamed Elhoseiny. Mammalnet: A large-scale video benchmark for mammal recognition and behavior understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13052-13061, 2023.
[39] Will Kay, Joao Carreira, Karen Simonyan, Brian Zhang. Chloe Hillier, Sudheendra Vijaya- narasimhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, et al. The kinetics human action video dataset. arXiv preprint arXiv: tet::000500000 2017.
[40] Junbin Xiao, Xindi Shang, Angela Yao, and Tat-Seng Chua. Next-qa: Next phase of question- answering to explaining temporal actions. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 9777-9786, 2021.
[41] Raghay Goyal, Samira Ebrahimi Kahou, Vincent Michalski, Joanna Materzynska, Susanne Westphal, Heuna Kim, Valentin Haenel, Ingo Fruend, Peter Yianilos, Moritz Mueller-Freitag, et al. The" something something" video database for learning and evaluating visual common sense. In Proceedings of the IEEE international conference on computer vision, pages 5842- 5850, 2017.
[42] Dejing Xu, Zhou Zhao, Jun Xiao, Fei Wu, Hanwang Zhang, Xiangnan He, and Yueting Zhuang. Video question answering via gradually refined attention over appearance and motion. In Proceedings of the 25th ACM international conference on Multimedia, pages 1645-1653, 2017.
[43] Jun Xu, Tao Mei, Ting Yao, and Yong Rui. Msr-vtt: A large video description dataset for bridging video and language. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5288-5288 5296, 2016.
[44] Qinghao Ye, Haiyang Xu, Guohai Xu, Jiabo Ye, Ming Yan. Yiyang Zhou, Junyang Wang, Anwen Hu, Pengcheng Shi, Yaya Shi, et al. mplug-owl: Modularization empowers large language models with multimodality. arXiv preprint arXiv:2304. /4178, 2023.
[45] Viorica Patraucean, Lucas Smaira, Ankush Gupta, Adria Recasens, Larisa Markeeva, Dylan Banarse, Skanda Koppula, Mateusz Malinowski, Yi Yang, Carl Doersch, et al. Perception test: A diagnostic benchmark for multimodal video models. Advances in Neural Information Processing Systems, 36, 2024.
[46] Viraj Prabhu, Sriram Yenamandra, Prithvijit Chattopadhyay, and Judy Hoffman. Lance: Stress- testing visual models by generating language-guidee guided counterfactual images. Advances in Neural Information Processing Systems, 36, 2024.
[47] Dan Hendryeks and Thomas Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. arXiv preprint
2019.
13

---

[48] Shresth Grover, Vibhav Vineet, and Yogesh Rawat. Revealing the unseen: Benchmarking video action recognition under occlusion. Advances in Neural Information Processing Systems, 36, 2024.
[49] Madeline Chantry Schiappa, Naman Biyani, Prudvi Kamtam, Shruti Vyas, Hamid Palangi, Vibhav Vineet, and Yogesh S Rawat. A large-scale robustness analysis of video action recog- nition models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14698-14708, 2023.
[50] John A Endler. Natural selection in the wild. Number 21. Princeton University Press, 1986.
[51] Edward Wilson, Sociobiology: The new synthesis. Harvard University Press, 2000.
[52] Robert L Trivers. Parental investment and sexual selection. In Sexual selection and the descent of man, pages 136-179, Routledge, 2017.
[53] Robert M Liptrap. Stress and reproduction in domestic animals. Annals of the New York Academy of Sciences, 697(1):275522)) 2244, 1993.
[54] Jie Lei, Licheng Yu, Mohit Bansal, and Tamara L Berg. Tvqa: Localized, compositional video question answering. arXiv preprint arXiv:1809.01696 2018.
[55] Makarand Tapaswi, Yukun Zhu, Rainer Stiefelhagen, Antonio Torralba, Raquel Urtasun, and Sanja Fidler. Movieqa: Understanding stories in movies through question- answering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4631- 4640, 2016.
[56] Kexin Yi, Chuang Gan, Yunzhu Li, Pushmeet Kohli, Jiajun Wu, Antonio Torralba, and Joshua B Tenenbaum. Clevrer: Collision events for video representation and reasoning, arXiv preprint 1rXiv:1910.01442, 2019.
[57] Yunseok Jang, Yale Song, Youngjae Yu, Youngjin Kim, and Gunhee Kim. Tgif-qa: Toward spatio-temporal reasoning in visual question answering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2758-2766, 2017
(58) Dan Liu, Jin Hou, Shaoli Huang, Jing Liu, Yuxin He, Bochuan Zheng, Jifeng Ning, and Jingdong Zhang. Lote-animal: A long time-span dataset for endangered animal behavior understanding. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 20064-20075, 2023.
[59] OpenAl. Chatgpt. https://openi..com//oo/////
2023.
[60] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-6851, 2020.
[61] Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint 1r12v:1312.6114, 2013.
[62] KunChang Li, Yinan He, Yi Wang. Yizhuo Li, Wenhai Wang, Ping Luo, Yali Wang, Limin Wang. and Yu Qiao. Videochat: Chat-centric video understanding. arXiv preprint arXiv:2305.06335, 2023.
[63] Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fahad Shahbaz Khan. Video-chatgpt: Towards detailed video understanding via large vision and language models. arXiv preprint wwXiv:2306.05424 2023.
[64] Ruipu Luo, Ziwang Zhao, Min Yang, Junwei Dong. Da Li, Pengcheng Lu, Tao Wang, Linmei Hu, Minghui Qiu, and Zhongyu Wei. Valley: Video assistant with large language model enhanced ability. arXiv preprint 2rXiv:2306.07207, 2023.
[65] Peng Jin, Ryuichi Takanobu, Caiwan Zhang, Xiaochun Cao, and Li Yuan. Chat-univi: Unified visual representation empowers large language models with image and video understanding. arXiv preprint 2tt:::2111.88044 2023.
14

---

[66] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 9748-8763 PMIR, 2021.
[67] Del Thiessen and Maureen Rice. Mammalian scent gland marking and social behavior. Psycho- logical bulletin, 83(4):505, 1976.
15

---

Appendix
A
More Details on QA Generation
In 5 we provide detailed breakdown of task divisions from coarse- -ransee-raieddd fine- une-grained levels, specifying the data used for each task. We have established specific rules for generating QA pairs based on the data requirements of different tasks. For tasks without existing QA pairs, use ChatGPT (59 to automatically generate three questions and randomly select one to mínimize model bias towards the questions. We also present the amount of data included for each task. Except for few tasks, most tasks have sufficient data.
B Option Difficulty
Quantifying the difficulty of options inherently challenging. In this study, we employ qualitative analysis approach to achieve moderate level of difficulty for the options. For the action recogni- tion task, examined the frequency of various actions and found that they adhere distribution. We categorize common actions, or "head actions," such as "running" and "eating," " as simple options that can be identified without specialized knowledge. actions," such as "molting" in birds, require specialized knowledge to identify and are thus classified as difficult options. Our approach involves incorporating the correct answer along with two simple options and one difficult option, thereby ensuring that the difficulty of the options is balanced and reflective of the natural frequency distribution actions. For the object recognition task, we test four situations:
long- -ail
contrast, rare actions, or "tail
Random selection: Besides the correct answer, the other three options are randomly selected from all the animal species involved.
Different major categories: Besides the correct answer, the other three options are randomly selected from different major animal categories than the correct answer. This setting makes the question-answar pairs easier because is a coarse- garseegrained judgment. The difference between the other three options and the correct answer is large, and if the model can identify correctly at the coarse-grained level, can answer correctly.
Same major category: Besides the correct answer, the other three options are randomly selected from the same major animal category as the correct answer. This setting makes the question-answer pairs more difficult because it is fine-grained judgment. The difference between the other three options and the correct answer is small.
Rules designed in this paper: Besides the correct answer, two options come from a different major animal category than the correct answer, and option come from the same major animal category as the correct answer. This design makes the question-answer pairs neither too difficult nor too easy.
The comparative experimental results regarding the four rules shown in Table The results indicate that the selection of options affects the experiment results, which also supports our theoretical analysis above. Our design can moderate the difficulty of the question-answer pairs, making the evaluation of the model more aligned with real-world scenarios.
Rules
Acc (%)
Different major categories
97.31
Random selection
91.96
Same major category
76.34
Rules designed in the paper
86.75
Table 4: The table of accuracy rates of VideoChat2's 2 responses for the four different option generation rules.
16

---

Ability
Coarse
Fine-grained
Datavets
Rules
0: there (X) in this video?
Existence
Object
Ne
50% probubility chosen from the GT lahels
25% prolubiliit chosen from the other animals with the mian 2000
25% prohubility chosen from other classes compared with GT
Object
0: "yes", "no". "mol sare"(sheffle)
Q: Randomly 0:
sentence patterns generated by
NemmalNet
Object
Aniuaal
One option from the label option
Recognition
Kingdom
Two options from other main classes compared with
Data: (1) Choose data that with single action label aumolation
(2) Duration 33
MammalNet
Q: Randomly choose
of three ChatGP T-generated questions
Recognition
Kingdom
O:
One uption from the GT label
Two options from the 50%
frequent frequent actions
One option
Data: (1) Choose data which include only one kind of animual with
different actions
(2) DDuration
Action
Animal
Q: Sentence paltern generated by CharGPT
Action
Sequence
Kingdom(VG)
232
One option
Two options from the top 50% of most frequent actions
One option from the least frequent 50% actions
Perception
Data: Choose data is bich include unly one kind amiimal with
more than two different actions
Common
Durattion 35
Prediction
Action
Kingdom (VG) O:
Animal
Q: Sentence pallern generated by ChaiGPT
208
One option from the label
Two options from the top 50% most frequent actions
One uption from the least frequent 50% actions
Choose
do
repeated
(2) Split the video into three time zones (beginning midddee'
ead/throghoutt, Nest, re re-annotate the data into those lnbels by longest.
checking in which time
grounding occurs the very beginning or end of the video and lasts
longer than 50% of the total video length, It should be removed.
Time
Action
Animal Kingdom
3- Duration
Q: Randomll choose one out of three ShanGGTT-generatee questions
O: "In the middle of the video."
"Throughout entire videa."
"Ai the beginning of the video." (haiflee
Data: DDrattio <:3
Object Count
MISVTTTQAA
O:
Utiibze CharGPT anmotations MSRVTT-QA
those animal-centric videos and
174
Count
Data: 3 Duration <:35
TGIF-QA
Utilize ChatGPT to adopt those animal- centric videos and anmotations in TGIF-QA
Cognition
Reasoning
Abductive Reasoming
NEXT-QA
Q.
Utilize CharGPT to adept those animol-centric videos and anmotations NExT-QA
362
Data: (1) Choose data that include action of "hunts other animals". "aammall""ccmmataiii "camooulaggigg" etc.
Predator- Prey Detection
MammalNet
Kinglom
Q: Sentence pattern generated by CharGPT
C: "yes". "ao". "not sure" (shuffle)
NammalNet
Data: (1) Cheose data that include action "hugging" "circummnal
Social Interaction
Animal
gland signing" "arine signing" and "aggregation" etc.
Kingdom
Duration
2337
Analysis
LaTE-
Q: Sentence pattern generated by
Special Task
Special Behavior
Annimal
C: "yes". "av". "not sure" (shuffle)
Perception
data that include with other animals". "gives birth to baby" and "nurses or breastfeeds its baby" etc.
"parental"
Breeding
Animal
Monitoring
Behavior
Kingdom
Duration
346
Ansimal
Q: Sentence pattern generated by ChatGPT
C: "yes". "ae". "not sure" (shuffle)
MammalNet Animal
"dying"
Detection
Kingdom
Q: Sentemce pattern generated by CharGPT
"yes" "not (shuffle)
(2) Duration
286
Table 5: More details about how Animal-Bena divide tasks, filter data and process data into QA format.

---

C More Experiment Details
As shown in Table 6 we standardized the parameters of all language models used to ensure the fairness of the experiments. Following [3), we designed uniform system prompt and answer prompt. By setting the answer prompt, our model outputs correspond to one of the options. In robustness testing, we selected 100 data samples from each task in the original Animal-Bench. Our selection process adhered to two principles: randomness, to minimize bias introduced by human intervention; and diversity, ensuring that the selected data represents the species in the original dataset. We believe that these data samples are sufficient to represent our original dataset and can reduce computational burden. We set different levels of severity for each variation. For weather changes, we set five variation parameters and randomly selected one for each piece of data. For size and angle changes, to simulate distant scenes, set the ratio of the simulated video height to the original video height between 0.1 (3 - 7); to simulate close scenes, we set the scale between 0.1 * (1.3 - 1.7). To simulate direction changes, we set the angle between (30% 90%) and (270 330%) mimic realistic conditions.
Parameters
Value
Carefully watch the video and pay attention to the cause and sequence of events, the detail and movement of objects, and the action and pose of animals.
System Prompt
Based on your observations, select the best option that accurately addresses the question. Only give the best option.
Question Prompt
Answer Prompt
Best option:
Crop Size
(224, 224)
Number of Frames
Max Tokens
200
Temperature
1.0
Scale
0.1 * (3 7), * (13 17)
Angle
(30 90"), (270' 330")
Table 6: Detailed Experimental Parameters.
D Ablation Study on Data Preprocessing
We conduct an ablation study on the number of video frames input to the model and the frame cropping method. We test with both 16 frames and 8 frames, designing two different video frame cropping settings.
Setting 1: The specific video preprocessing process is as follows: if W, the frame is scaled to (224, 224 x H/W). W > H. the frame is scaled to (224 x W/H 224). After scaling, the video frames are center-cropped to obtain center region of (224, 224).
Setting We also experiment with padding non-square videos along the shorter side to make them square before scaling them to (224, 224).
The experimental results are shown in Table 7 Reducing the number of frames leads to a decrease in the model's accuracy on most time-related tasks, which demonstrates that increasing the input video frames benefits the model in extracting temporal information and improving accuracy. However, in the object counting task, reducing the number of frames results in a significant performance improvement. We believe this is because the MSRVTT-QA dataset used for object counting contains video segments from different perspectives within single video. These different perspectives may capture various states of instances, and increasing the number of frames could confuse the model when processing redundant information from different perspectives, making it difficult to distinguish then-cropping video frame preprocessing
between instances. We also find that using the method leads to slight decrease in the model's accuracy.
18

---

Mithhols
A8
num-16
50,00
86.75
66:27
34.31
50,00
21.22
64.94
29.16
68.23
52,40
47.69
52.42
54.20
53.66
Setting
Sating
50.00
34.74pa0
46.15
21.28
52.40
name 16
50.00
85.15,
22.17
62 12.64,123
66.57,
$4.20
53.04,
Sattnng
Table 7: Ablation study results table for data preprocessing.
E
Results on MSRVTT-QA and TGIF-QA
also analyze the models' performance on all data from MSRVTT-QA and TGIF-QA, as well as the animal subset data from Hhimal-Bench, as shown in Table The model performs better on the overall data but worse on the animal subsets. This indicates, to some extent, that the model exhibits agent bias, demonstrating stronger understanding of videos with humans and objects agents compared to those with animals as agents.
MSRVTT-QA
TGIF-QA
Method
Unimal-Bende (Object Count)
Animal-Benc
Original All Data
Original All Data
(Action Count)
VideoChat
45.0
27.6
34.4
25.5
Video-LLaMA
29.6
26.4
25.0
Video-ChatGPT
49.3
24.7
51.4
22.9
Video-LLaVA
59.2
31.0
70.0
22.5
Table Experimental results comparison on MSRVTT-QA, TGIF-QA, and Animal-Bench (object count and action count).
F Multimodal Video Model Parameters
In Table we present a comparison of the parameters of the multimodal video models used in experiments. We provide detailed description of the model parameters from aspects such as video encoder, temporal module, large language model, training datasets, and the size of the training datasets. By analyzing these alongside the experimental results, we obtain several discussions on the model structure in the main part of the paper. In addition to the discussions on the video encoder and temporal module mentioned in the main paper, we also find that the amount of training data does not significantly affect the experimental results; a larger training dataset does not necessarily lead to better model performance, and vice versa. We hope these findings on the impact of model structure and training process experimental results can inspire future development of multimodal video models.
G Data Statistics on Different Agents
In Fig. g we present statistics on the agents involved in each task of MVBench. It can be observed that most tasks only include human agents objects interacting with humans, with very few tasks involving animal agents. This further illustrates that existing evaluation benchmarks are biased towards humans, and the evaluation of multimodal video models in the domain of animals is lacking.
We have calculated occurrence frequency each animal species in our Animal- dataset, as depicted in Figure 10 It can be observed that the distribution of animals is not uniform, following long-tail distribution, which aligns with the general pattern where some species are more abundant while others are less in the natural world.
19

---

Training
MLLMM
Video Encoder
Temporal Module
L.M.
Training Dataset
stage LAION 400, MCOYO 700M.
stige 1105M
MPLUGGOMM OWW
CLIP W17L-14
LILaMA-78
Concepinal Captions, MSC000
stage
2: ALpaca, Vicuna, Hazze LLaVA
Video-ChatGPT
VILL-14
Vicuna-72 788
10UK
Mage
WebNidZM
Valley
Temporal unemiion,
Stable
stage video, image instrutionn
stage 1297K
Tempural token
234K
(combineed. LLaVA, Video CharGPT)
stage
WebVidd 10M, COCO, Caption, Visual Genome,
VidenClat
VIT-V144
Stalle Vicuns-BB
SRU Captions, CC3M, CC12M
stage
Aggregstor
stage
MiniGPT LLaVA
stage imagosciption pairs (COCO.
stage 595K stage
CaiV
CLIP VIT-L/14
Dynamic visual token
Vicuna -78
stage MIMIC-TI, Video
Video-LLaMA
VTT-/14
Vicuma -BB
stage Webvid CCS95
stage 2M
stage MiniGPT-4, LLaVA, Video-ChutCPP
stage 2: 649K
stige 1: CCSM, CCT2M, WebVid-IUM
stage I: 15M
VideoChar2
UMTE
Vicuta-78
COCO, Visual Genome, SBU, InterViid
2: 12M
stage catogorios
stage
Video LLaVA
Vicuna 7B
stage LLAON-OC-SBB, WelVid
stage
stage LIaVA, VideoClauiiF
stage 765K
Table 9: Comparison of parameters of different multimodal video models, where "f" indicates that no trainable temporal modules are introduced.
250
200
200
200
200
200200198
200
200
200
200
200
200
200
168
150
123
100
50
26
20
aciion antenyym
action count
action localization
action prediction
action sequencee
counterfactual infercnce
character order
egncontrc nev gggaaimmm
quisadic rensoning
fine grained action
moving attribute
moving count
moving direction
object existonee
objest interaction
object shuffl
scone transition
state change
umespested action
human
animal
object
Figure 9: The data quantity for different agents in each task of MVBench.
490
380
380
190
Whit Word decker
Dady perpaee Syetr
Cos oi Workin
thaik Crves Coewr
Whet hovell Codl
Grean Macken thegg
The
Doe
ceecral that theee
Set taadee 20r9
carpholl beeko
Cardel
GrarCryy
Centaa
Guiiaa thas
Oviss
Figure 10: Statistical chart of data quantities for different animal species.
20

---

H Potential Negative Impacts
Although the technology itself is intended to protect and study wildlife, if it falls into the hands of malicious actors, it could be used for illegal hunting, poaching, and animal exploitation. These actions could cause severe damage to wildlife populations and make conservation efforts more challenging. Also, excessive reliance on technology for animal monitoring and protection may lead to neglect of manual patrols and traditional conservation methods. If the technology fails or is compromised, it could result in serious security vulnerabilities. Given these potential negative impacts, recommended implement strict security measures in technology application and data management, and to establish appropriate laws and regulations to ensure the technology is used correctly and safely.
I Limitations and Future Directions
Our work has some limitations: First, the responses of multimodal video models may be influenced by the input parameters of the models, and different parameter settings could affect the accuracy of the responses. We did not test the sensitivity of the model to these parameters and instead used uniform parameters for the models. Second, since our editing is based on stable diffusion, the editing effect is subject the capabilities and limitations of stable diffusion. Third, video editing may introduce some counterfactuals or additional animals that could change the answers to the questions. We have made efforts to minimize this occurrence through manual selection. In futurre work, we will continue to study the impact of model parameters on model performance and conduct deeper research on diffusion models achieve better editing effects. We also plan to reduce the appearance of new animals in the future by adding negative prompts and conducting further exploration.
J Assets and Licenses
In our paper. we use data from six datasets: Hhimal-Kingdom 37, MammalNet [38, LoTE- Animal [58], MSRVTT-QA [42). NExT-QA [40), and TGIF-QA of the aforementioned works, all which have been cited in the main article. Specifically:
QAA (57]. We appreciate the contributions
MammalNet is licensed under the CC BY license.
LoTi--nimal is licensed under the Creative Commons Attribation-ShareAANi 4.0 Interna- tional License.
MSRVTT-QA is licensed under the MIT license.
NExT-QA is licensed under the MIT license.
For the Hhimal-Kingdon dataset, we have contacted the authors by filling out a questionnaire regarding the dataset's use and have obtained an official download link. Additionally, we have emailed the authors about our use of the dataset in our paper.
The TGIF-Q4 QA dataset is explicitly stated on its GitHub page "to be free to use for academic purposes only."
We believe that our work fully respects the original authors' copyrights, and all assets have been used appropriately.
21

---

Original
Distance variation
Direction
Original
Snow
Frost
Figure 11: Visualization of simulated changes mimicking real shooting scenarios
22

---

Objeet Recognition
Chuz-UniVi metlapckcatlus mecuicamss jumping pii viper
mPLUG-ONe Owl: (A) medaplooathss mexicanes jumping pi viperiala he vodeo shows metlaplioaatns
meticanas jumping viper. which also known Central American rattlestale.
can video?
mecapicopus
jumping pit viper
(A) (A)metlapiccoatus mexicanus jumping pit viper (Bymonkey young
metapikooztus mexicanus jumping viper.
with black markings that is visible when jumps up the and disappears into the vegetation. This stake is identifeed the mellapilceechs mesicanus, species of pit viper that is known for its jumping abiliny. It found in tropical and subinopiesl regions of Ameriens and
(Cstoor
(D)bamnophii eyrtopsis snake
VideoChat2: thamnophis cynopsis
Right anewer: (A (A)metlapilcoatus mexicanus [umping siper
Mibe-ChanPP:: (A) mettpillcztuus mexicanus jamping viper, The video shows a stake jumping and dithering the tres norr troe stump. The scene shows the mnke moving quickly away from the person looking crosing move into different position and then hide under the tice.
Vadeo-LLMMA; thamnophis cystopii stake.
Vida--IaaVA
Object Existence
Chat-UniVi: (C)
Is there canis in this video? (AJyes
mPLUG-Owl: (A) yes. Yes, there is canis the video. The brown dog is seen eating its dogg
bowl.
Valley: (A) Yos, there is a canis the video.
(B)no
ViaeoChat: Yes,
dog video.
(C)not sure
VideoC hat:: (A) yes
Video-ChatGPP: (A)
canis the video.
Right answer: (A) yes
Vidzo-LLaMA: (A)
Vidgo-LLaVA: (A)
Action Recugnition
Chat-UniVi: (D) Swimming. The video shows group of small fish swimming together school
What activity does the video depiet?
within the open sea, emphasizing the communal nspect of their behavior.
(A)Flying (B)Spitting Venom Unddrrgggng Chrysalis (D)Swimming
mPEII--OW: Flying
Valley: (A) Flying.
VideoChat: (A) Flying
VideoChat2: (D) Swimming
(D) Swimming, as video focuses swimming in water.
Right (D)Swimming
Video-LLaMA:
Swimming
Video-LLaVA: (C)
Action Prediction
What animal de after
Chus-UniVi: Sunfaeingg
mPLUG-Dwl. Hanging.
animal was chirping?
Valley: Surfacing,
most appeopeiate response given cues video.
(A]Hanging
squinel indied surfing on its back.
(B)Drinking
VideoChut (C) Surtacing
(C)Surfacing
VhdeoChat2: Hanging
(D)Kesping still
CharGPT: Keeping
keeping tre, possibly
steady and recording of bird.
Right answer:
Ke)peggn still
Video LLaVA: (D)
Video- LLaMA: Keeping still
Figure 12: The illustrative sample of the generated responses from different multimodal video models for "object recognition", "object existence", "action recogntion" and "action prediction" tasks. Green indicates correct answers, red indicates incorrect answers, and orange indicates that although the chosen option is correct, there are some errors in the response.
23

---

Action Localization
Can you ideatity when the action "keeping siill
happens the
mPI UI-OUt: (C) end video,
still the plant starts moving
(AIA bepinning
Valley: (D) thott the entire videa
(Dii middle the video.
VideoChat (D) Throughout the cutire valeo.
(C)AI the and the video.
VideoChat2: (i)
middle the
(D/hhrouhhou entire video.
ChasGP (C)AI
Video-LLaMA: (8) the auddle of the video.
Right answers (C)A the end the video.
Video-LLaVA LLaVA: (A)
Reasoning
why did the black dog ran after the white the video?
Chat- UniVi: (C) play with ball.
mPLUG-Owl: (D) Playing with each other. Roth dogs playing with ball, which could
have bocn the reason the white dog ran towards the grass and the black dog ran after it. Dogs among Valley: (A) the black dog was chased by the white dog running towards the grass.
was chased (B]brown dog took its tay (Clllay
play,
type playful interaction
Videot brown dog tool toy.
(Diplaying with each other
VideoChar2: (A) it was chased
(E)attracted by
Midro-ChatGPT: (B) black dog chased after the white dog who had taken toy and was
running towards
Right answers (D)playing with each other
Video-LLaMM:: (D) playing with each other
Video-LLaVA:
Object Count
Chat-UniVV: four
how many wallinng
mP..UG-OW:: throiieleThe video
walking
(A)Three
Valley (C) Rour
FRvv
VideoChat (A) three
(C)
VideoChat2: (D) two
Video- ChaokC
dinactions
(D)
actions. Allhooug observing mulliple walking indicative pride, possible to determine the exeett number from a single image.
Right answer: (D)Two
Viden-l LaMA; (C) four Tions walking the fonist.
Video-LLaMA.
Social Interaction Analysis
hugging, mPt U.-.UG--Wh:: (C) Noe sune. The animal the video not hugging.
Is the animal this video bugeing? (Alto
Valley (C) not surcnPPlesse watch the vidlon again and make sume you have observed the snimal elearly before
providing anower.
(Byye
ViduoChat:
(C'not sure
Videa-ChhutiP. (B) You The video shows aninal lugging as object whilee moving scross the seroen.
Right answer: (B)yes
Since the object green, is possible that is stalk wath toliage plant.
Video LLaMA: (BB
Figure 13: The illustrative sample of the generated responses from different multimodal video models for "action localization". "reasoning", "object count" and "social interation analysis" tasks. Green indicates correct answers, indicates incorrect answers, and orange indicates that although the chosen option is correct, there are some errors in the response.
24

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract and introduction sections of the paper clearly and accurately state the contributions and scope of the paper.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We analyzed the limitations of the paper in the "Limitations and Future Directions" section of the appendix.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-speciiiatiinn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g. if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
25

---

Answer:
[NA]
Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof's can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: Section 4.1 and the "More Experiment Results" part in appendix of the paper specifies the experimental details.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, descríbing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e-g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
5. Open access to data and code
26

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We have included anonymous link to GitHub repository in abstract of the paper, and we will soon make our data and code publicly available.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://niis.cc
aublic/guudees/Coeeetuii.ss casssiionoolyyyy for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ //nips. wwtps/cccppcccguuuaieeaeeecc witeeee//ooSSSeSese..ee.e www.ssssiicggoooooo for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Section 4. and the "More Experiment Results" part appendix of the paper provides detailed description of the experimental details.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We present the 95% confidence intervals for the random results in Table 2 demonstrate significantly lower results compared random selection.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
27

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative еπτог rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: In the Section 4.1, we present the computing resources we used.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https://neuripp: wccppbblic/Bhhccsuuedeellle
Answer: [Yes]
Justification: The authors have reviewed the NeurIPS Code of Ethics and have strictly adhered to it in the paper.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: The introduction and and the "Limitations and Future Directions" section of appendix outline the paper's potential societal impacts.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
28

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), faimess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best. faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: In the "Assets and Licenses" section of the appendix, provide details about the assets and licenses involved our paper.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
29

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com/dataseta
latasets.papersstt
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
We provide certain explanations in Section 3, 4.1 and the "More Experiment Details" section in the Appendix. We have included the relevant links the abstract and will release our data and code soon.
Guidelines:
The answer NA means that the paper does release assets.
Researchers should communicate the details the dataset/code/mooell as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsoureing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
30

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
31


Reviews:
[{"review_id": "LGJVTUFTuL", "reviewer": "Reviewer_xc4z", "summary": "This work introduces Animal-Bench, a novel benchmark for evaluating multimodal video models in animal-centric video understanding. The benchmark covers 13 tasks spanning 7 major animal categories and 822 species. It proposes an automated pipeline for data filtering and question-answer pair generation, reducing human effort and potential biases. To simulate real-world shooting conditions, it employs video editing methods based on diffusion models to evaluate model robustness under various scenarios. This work evaluates 8 popular multimodal video models on Animal-Bench, identifying considerable room for improvement on animal-centric tasks.", "questions": "See [Weaknesses]", "limitations": "The uniform set of parameters used for all models in the evaluation, as mentioned in Table 4, may not align with each model's recommended settings, such as temperature. It could potentially prevent from fully leveraging the capabilities of individual models.", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1) This work introduces a comprehensive animal-centric benchmark covering a diverse range of tasks, including several that have been previously under-explored in the field.\n2) The authors claim to open source the code and data, which could be beneficial for the research community.\n3) By evaluating multiple recent multimodal video models on Animal-Bench, the work provides insights into current model capabilities and limitations, and highlights potential directions for future research and development.", "weaknesses": "1. The answer accuracy of the QA pairs. For example, in the \"Reasoning\" task illustrated in Figure 2, the correct answer appears to be \"to fight with dog\" rather than \"cat\".\n\n2. The question quality needs further improvement. 1) Ambiguity: in the \"Time\" task shown in Figure 2, the presence of multiple objects in the video frames renders the subject of the action ambiguous. 2) Inconsistency between video frames and question description: in the \"Object Count\" task in Figure 13, the setting appears to be a \"grassland\" rather than a \"forest\".\n\n3. The simulated changes intended to mimic real-world shooting scenarios exhibit noticeable artifacts and unrealistic situations. For example, Figure 11 shows visible boundaries from outpainting and implausible weather conditions (e.g., snow added to scenes with green grassland). To address this, the authors could consider implementing an aesthetic score-based filter or a specially trained discriminator to get rid of data with severe artifacts.\n\n4. Section 4.1 mentions resizing input videos to 224. For non-square videos (particularly those with highly disproportionate size), it's unclear whether additional operations (such as padding or cropping) were employed to accommodate the inputs. If such operations were used, an analysis of their potential impact on model performance across various tasks would be beneficial.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable comments and thoughtful suggestions, and hope our response will address your concerns.\n\n**W1, W2: Question and answer quality**\n\nIn fact, although the examples in the “Time” task in Table 2 show multiple objects, only one object is performing an action. The other objects remain stationary or their actions are different from the action in the question, thus avoiding action ambiguity. \nSince the question-answer pairs for the “Reasoning” and “Object Count” tasks are directly sourced from existing data annotations (NExT-QA, MSRVTT-QA), the quality of these question-answer pairs is influenced by the existing data annotations to some extent. In reality, we have manually filtered them to make as few errors as possible. Despite the minor errors, these errors do not pertain to key information that can affect the ability of humans and video language pre-training models to choose the correct answer.\n\n**W3: Artifacts and unrealistic situations**\n\nThank you for your valuable suggestion. In fact, our goal is to mimic shooting parameters and weather changes in real-world scenarios to evaluate the practical applicability of the model. This is very different from the previous method of generating counterfactual and unnatural videos through video editing (such as the crane appearing tilted on the grassland or butterflies appearing in the water, as shown in the lower left corner of Figure 1) for model evaluation. As shown in Figure 11, our method can address the issue of unnatural scenarios. Even though using editing methods inevitably results in some artifacts, this does not affect the basic adherence to natural conditions. For example, our editing results would not show scenarios like butterflies flying in the water. In the future, we will pay more attention to improve the editing effects based on your suggestion.\n\n**W4: Video preprocessing**\n\n**(Setting1:)** The specific video preprocessing process is as follows: if H > W, the frame is scaled to (224, 224 * H / W). If W > H, the frame is scaled to (224 * W / H, 224). After scaling, the video frames are center-cropped to obtain a center region of (224, 224). \n\n**(Setting2:)** We also experimented with padding non-square videos along the shorter side to make them square before scaling them to (224, 224).\n\nThe results are as follows:\n\n| | OE  |  OR  |  OC  |  AR  | RS | AC | AS | AP | AL |\n|------|------|------|------|------|------|------|------|------|------|\n|Setting1|53.63|83.90|65.69|57.57|40.71|28.38|47.26|40.41|24.44|\n|Setting2|53.64|79.15|64.71|56.51|40.02|28.29|45.89|38.70|24.24|\n\nIt can be seen that the video preprocessing method has a certain impact on the test results. Using padding causes a slight decrease in test accuracy, but the decrease is not significant.\n\n**L1: The parameter set**\n\nPlease kindly refer to \"Author Rebuttal\" Q4.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "A9t86VHNGh", "reviewer": "Reviewer_yXxA", "summary": "1. This paper introduces Animal-Bench, an animal-centric video understanding benchmark. The benchmark includes 13 tasks, spanning 7 major animal categories and 822 species.\n2. The authors collect data primarily from 6 open datasets, such as TGIF-QA and Animal Kingdom, and apply data filtering based on diversity and temporal sensitivity. They then generate task-specific QA pairs.\n3. To evaluate the robustness of video understanding models, the authors use realistic simulations based on video editing, including variations in weather conditions and shooting parameters.\n4. Eight video understanding models based on 7B LLM are tested on Animal-Bench.", "questions": "1. What are the resolution and duration of the dataset?\n2. How is data filtering specifically performed (e.g., manual filtering or automated filtering using other models)? Additionally, for each source dataset, what proportion of data is filtered?\n3. For the option-setting method described in L176-183, has the rationality and differentiation of this method been experimentally verified? For example, if a different random seed in setting options is used, how much would it affect the final results? For some questions, is the difference between the correct and incorrect options too large, making it easy to choose the correct one?", "limitations": "The web page for accessing the data and code is still unavailable.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. This work focuses on animal-centric visual understanding, which not only aids the community in better evaluating video understanding models but also holds significant social value.\n2. It provides a comprehensive classification of animal video understanding tasks, summarizing and categorizing existing open datasets.\n3. The study uses Animal-bench to evaluate existing video models, conducting detailed performance and robustness analyses.", "weaknesses": "1. According to Table 3, the dataset suffers from severe class imbalance.\n2. If my understanding is correct, all videos and annotations in the dataset are derived from existing open datasets. Therefore, this benchmark primarily serves to summarize and convert data formats to a conversational format, without adding new manual annotations. Also, this dataset-building strategy poses a potential risk of data leakage.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable comments and thoughtful suggestions, and hope our response will address your concerns.\n\n**W1: Class imbalance**\n\nFirstly, the size shown in Table 3 represents the number of data points for each task, rather than the number of animal class or behavior class. Our evaluations for each task are independent, so the number of test samples for different tasks does not affect each other. \n\nSecondly, the primary design goal of our benchmark is to evaluate the understanding capabilities of current video-language models for animal-centric videos. We believe that each task contains sufficient data for evaluation (refer to MVbench (CVPR24), where each task only has 200 videos).\n\nAdditionally, for each task, we have collected as much data as possible to ensure the richness of our benchmark. For a few tasks, due to the limited availability of related videos, the number of videos included in the benchmark is relatively small. However, as mentioned earlier, the data volume for these tasks is still sufficient to validate model capabilities. We also plan to collect more data for these tasks to further enhance the richness of the dataset.\n\n**W2: No new annotations and potential data leakage**\n\nFirstly, we chose to use open datasets because they have been widely utilized and validated by the community, ensuring better data quality and label reliability. Additionally, collecting animal videos, especially wildlife footage, is challenging, and it is difficult for ordinary people to annotate such data. Therefore, using open datasets significantly saves manpower.\n\nSecondly, we selected data for the designed tasks instead of simply stitching together existing datasets. We examined multiple datasets and chose those capable of accomplishing specific tasks while being as diverse as possible. Many of the videos in these datasets primarily feature humans rather than animals, requiring us to filter the data accordingly. We used a combination of manual and automated tools for data filtering. For details on the filtering process, please see Q4. \n\nThirdly, converting the original annotations into question-answer pairs is not that easy. It requires careful consideration of how to design questions and options to be as fair and reasonable as possible. In the \"Roles\" column of Table 3, we present the detailed rules for generating our question-answer pairs. Additionally, in \"Author Rebuttal\" Q2, we elaborate on our thought process and experiments to make the generation of question-answer pairs more reasonable when no directly available pairs were present.\n\nIn Table 5, we present the pre-training datasets of the models we tested, excluding the dataset used by our Animal-Bench. Additionally, the datasets we used, such as Animal Kingdom, MammalNet, and LoTE-Animal, were proposed after 2022, whereas the datasets used for model pre-training are primarily from before 2022, thus thus preventing any data leakage issues.\n\n**Q1: The resolution and duration**\n\nPlease kindly refer to \"Author Rebuttal\" Q1.\n\n**Q2: Data filtering details**\n\nWe adopt a combination of manual and automated filtering methods. First, after designing and determining specific tasks, we manually select datasets from a large number of animal category datasets, animal behavior datasets, and general QA datasets that can achieve specific tasks. Specifically, for tasks related to \"Action,\" \"Object,\" and \"Time,\" we mainly obtain annotated data for animals, actions, and locations from Animal Kingdom, LoTE-animal, and MammalNet. For \"Counting\" and \"Reasoning\" tasks, our evaluation data comes from TGIF-QA, MSRVTT-QA, and NExT-QA. For special tasks, we select data from Animal Kingdom, LoTE-animal, and MammalNet.\n\nSecondly, we filter the data to ensure that Animal-Bench only contains animal data. For datasets like TGIF-QA, MSRVTT-QA, and NExT-QA, where most videos are human-centric and a few are animal-centric, we use GPT-3.5 to filter animal videos based on question-answer pairs and annotations. The system prompt is: \"Determine if the data is about animals, not humans, based on the question and answer.\" We then perform further manual filtering on the automatically filtered results to ensure that the data is solely about animals.\n\nAfter that, we filter each dataset according to the designed rules (as shown in the last column of Figure 3). We believe these rules can ensure the fairness and moderate difficulty of the evaluation as much as possible. We have written data filtering code to perform automated filtering.\n\nThe overall data volume and filtered data volume for each dataset are as follows:\n|Dataset|overall volume|filtered volume|proportion|\n|---------|---------|---------|---------|\n|Animal Kingdom(AR)|30100|13577|45.11%|\n|Animal Kingdom(VG)|18744|2718|14.50%|\n|MammalNet|18395|1626|8.84%|\n|LoTE-Animal|9991|602|6.03%|\n|MSRVTT-QA|244337|102|0.04%|\n|TGIF-QA|165165|3394|2.05%|\n|NExT-QA|52021|1007|1.94%|\n\n**Q3: Option-setting methods**\n\nPlease kindly refer to \"Author Rebuttal\" Q2.\n\n**L1: Code and data**\n\nTo present our data and code more clearly and comprehensively, we are currently working on organizing them and will make them publicly available soon.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4pYtDmNB06", "reviewer": "Reviewer_xfS8", "summary": "Propose of an automated pipeline for animal-centric large-scale multimodal video benchmark, Animal-Bench, that simulates real-world conditions such as snowing via diffusion-based video editing approach. This data is sourced from 6 dataset and multiple filtering have been applied to ensure diversity and lack of bias. ChatGPT is utilized for creating 3 types of questions for each task and one is randomly assigned to each data. Furthermore, authors simulate closeness, distance, and different angles by changing the shooting parameters. Animal-Bench covers 13 tasks across 8 animal categories (reptile, fish, insect, amphibian, mammal, sea animal, and bird), and 822 animal species. They have tested 8 current baselines in multimodal video and have analyzed the results.", "questions": "1- How did you determine the appropriate level of difficulty for options in multiple-option questions, especially for tasks that don’t have existing QA data?\n\n2- Did you notice any limitations or biases that were introduced by the outpainting method (line 209)?\n\n3- Which LLM are you using? “7B LLM backend versions” line 242\n\n4- Regarding the hallucination problem in line 260, have you considered including negative examples or adjusting the loss function to mitigate the problem?\n\n5- Which version of ChatGPT did you use? How much was the cost of using ChatGPT per data point and also overall?\n\n\n\n\nSuggested references\n\n1- Van Horn, Grant, Oisin Mac Aodha, Yang Song, Yin Cui, Chen Sun, Alex Shepard, Hartwig Adam, Pietro Perona, and Serge Belongie. \"The inaturalist species classification and detection dataset.\" In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 8769-8778. 2018.\n\n2- Parashar, Shubham, Zhiqiu Lin, Yanan Li, and Shu Kong. \"Prompting scientific names for zero-shot species recognition.\" arXiv preprint arXiv:2310.09929 (2023).\n\n3- Parashar, Shubham, Zhiqiu Lin, Tian Liu, Xiangjue Dong, Yanan Li, Deva Ramanan, James Caverlee, and Shu Kong. \"The Neglected Tails in Vision-Language Models.\" In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12988-12997. 2024.\n\n\n\n\n\n\n\nSuggestions:\n\n1- Please use “Schiappa et. al” instead of “ Madeline Chantry Schiappa’s work” line 99\n\n2- For each video, we sample 16 frames and resize them to 224 (line 246) – Please change to 224x224\n\n3- line 246 stablediffusion-inpainting  → StableDiffusion-inpainting \n\n4- line 248 stablediffusion-v1.5 → StableDiffusion-v1.5\n\n5- Please provide how much VRAM your GPU has (24GB?), how much RAM and which CPUs. How much space does your dataset take. Line 486 → Please note this should be in the main paper not appendix.\n\n6- Please have a look at https://forum.inaturalist.org/t/published-papers-that-use-inaturalist-data-wiki-3-2022-and-2023/34753 which is a list of published papers that made use of iNaturalist data.", "limitations": "- Lack of exploration of input language parameter sensitivity\n\n- Relying on StableDiffusion which has its own set of limitations (training data bias, manipulated content, lack of diversity)\n\n- Limited dataset (while they have combined multiple data points, still the variety of animals out there makes this quite a small dataset). Further, I am curious how many animals may only have a small sample of videos while animals like cats and cows may have way more samples presents. \n\n- The potential hallucination that happens should warn the users that may use this dataset to train their model for tasks such as animal conservation. \n\n- While 13 tasks is a great starter on its own (given there’s not much parallel research going on), there’s a lot more that could be explored and expanded.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Data and code will be publicly available, and accelerate animal research\n\n- To the best of my knowledge, this is the first large-scale multi-categories multi-species multimodal dataset with a focus on animals across 13 tasks. This paper will open lots of avenues in the field of animal research (e.g. animal surveillance) as well as introducing new multidisciplinary collaborations.\n\n- Experiments are well-described and reproducible upon release of code.", "weaknesses": "- Missing some important implementation details (still quite reproducible) \n\n- The main paper is 9 pages \n\n- Great research into prior work\n\n- Authors have not mentioned any potential negative impact of the paper. However, potential of misuse of technology by bad actors for hunting, poaching, and animal exploitation are some examples of how it could be leveraged incorrectly.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable comments and thoughtful suggestions, and hope our response will address your concerns.\n\n**W1: Potential negative impacts**\n\nPlease kindly refer to \"Author Rebuttal\" Q3.\n\n**Q1: Option-setting methods**\n\nPlease kindly refer to \"Author Rebuttal\" Q2.\n\n**Q2: Limitations of the outpainting method**\n\nWe use the StableDiffusion model for outpainting, which means the effectiveness of the outpainting depends on the capabilities of the StableDiffusion model. Additionally, employing video editing models for outpainting inevitably introduces some artifacts, such as unnatural transitions at the boundaries or the potential introduction of new animals. However, we manually reviewed the edited videos and found that these issues do not alter the answers to the questions.\n\n**Q3: Types of LLMs**\n\nIn Table 15, we present the types of large language models (LLMs) we used. Except for mPLUG-Owl, which uses LLaMA-7B, all others use Vicuna-7B.\n\n**Q4: Hallucination problem**\n\nThank you for your suggestions. In fact, our ultimate goal is indeed to fine-tune a video-language model suitable for animal video understanding and wildlife protection. Recently, research on hallucinations has been emerging continuously, and we will further consider how to mitigate this issue in the future.\n\n**Q5: Regarding the GPT used**\n\nAfter manually pre-filtering the data according to the task, we used the gpt-3.5-turbo-0125 model on the remaining 48,043 data points. Each data point cost approximately 0.000035 dollars, with a total expenditure of around 1.7 dollars.\n\n**L1: Parameter sensitivity**\n\nPlease kindly refer to \"Author Rebuttal\" Q4.\n\n**L2: The limitation of StableDiffusion model**\n\nAs you mentioned, the editing model does have some limitations. Therefore, we conducted manual reviews of the results from StableDiffusion to filter out a certain amount of substandard outputs. Moving forward, we will further address this issue by developing automated tools and combining them with manual reviews to ensure the quality of the edited data.\n\n**L3: The limitation in the number of animal species**\n\nAlthough the 822 animal species included in our benchmark are still insufficient compared to the vast diversity in nature, they cover the characteristics of most animals in the seven main animal categories (and similar species often exhibit similar behaviors). Moving forward, we plan to further expand the dataset to encompass a broader range of animals. The distribution of the number of different animal species follows a long-tail distribution, which aligns with the general pattern where some species are more abundant while others are less so in the natural world.\n\n**L4: The potential hallucination**\n\nThank you for your suggestion. We will emphasize the potential hallucination issues in future versions to ensure that users pay close attention to and consider these issues when using our dataset.\n\n**L5: More tasks**\n\nThank you for your suggestion. The selection of these 13 tasks helps us understand the core capabilities of the model from a coarse to a fine-grained perspective. We hope our work will provide insights for the community and stimulate further exploration.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "iE0tp4q1zD", "reviewer": "Reviewer_Mynx", "summary": "This paper introduces Animal-Bench, a video question answering benchmark focused on animals, which is usually overlooked in previous video benchmarks. Animal-Bench is sourced from six datasets and includes 13 tasks. Eight video-language models are evaluated on the benchmark and the results reveal shortcomings in the models. Moreover, the paper evaluates the robustness of the model by simulating weather and shooting parameter changes, which are challenging in real-world application.", "questions": "1. Is the accuracy drop in Table 2 relative drop or absolute drop?", "limitations": "Limitations are discussed in Appendix F. No potential negative societal impacts are mentioned.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. While the current video question answering benchmarks focus on human activities, Animal-Bench is the first animal-centric benchmark. It can not only boost the AI application in animal studies, but also help the development of video-language models.\n2. Animal-Bench aligns the benchmark with real-world applications by 1) including domain-specific tasks, e.g., breeding monitoring; 2) simulating realistic scenarios - weather and shooting parameter changes - by video editing.\n3. The paper thoroughly evaluates recent video-language models.", "weaknesses": "### Missing dataset statistics:\n\nThe paper provides many detailed dataset statistics about the number of videos and the long-tail distribution of animal categories. But some important statistics are still missing:\n\n1. The number of questions on each video. If there can be multiple questions generated from one video, both the number of videos and number of questions should be listed in Section 3.1 and Table 3.\n2. The distribution of the video durations.\n\n### Answer correctness after video outpainting: \n\nIn the robustness evaluation, outpainting may introduce new animals to the video, e.g., row 2 column 2 in Figure 11. It may change the answer of the question on tasks like object existence and object count. As a result, the robustness evaluation could be inaccurate. It would be better to manually check a subset of the outpainted videos to see how well outpainting can preserve the correct answer.\n\n### Minor weaknesses:\n\nWhen a pre-trained model is applied to a specific downstream domain, it is natural to improve the performance by fine-tuning it on the downstream domain. However, this benchmark only provides a test set without training and validation sets. It would have greater impact if a training set is included and the fine-tuned model performance is evaluated.\n\n### Writing:\n\n1. It is better to explain the task abbreviations in Table 1 (either in the text or in the table caption).\n\n2. Showing the accuracy drop in Table 2 is a great way to demonstrate the model robustness. But it would be better to also list the absolute model accuracy.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable comments and thoughtful suggestions, and hope our response will address your concerns.\n\n**W1: Missing dataset statistics**\n\nFor each task, the included videos correspond to only one question. For example, for the object recognition task, we only selected videos containing a single species to avoid potential ambiguities. For different tasks, the same video may correspond to two or more tasks. Some videos can be used to evaluate both the object recognition task and the action recognition task. This data reuse can, to some extent, alleviate the scarcity of animal videos. We will supplement this in subsequent versions.\n\nPlease kindly refer to \"Author Rebuttal\" Q1 for statistics on video duration.\n\n**W2: Answer correctness after video outpainting**\n\nIt is indeed possible for new objects to appear in outpainting. To address this, we conducted manual filtering to avoid potential impacts from outpainting of new species on the answers. In practice, such changes to the answers are virtually nonexistent. For tasks like object existence and object count, the questions include specific species, such as \"Is there a monkey?\" This means that the example in the second column of Figure 11 does not change. Due to the diversity of species in nature, the likelihood that the species mentioned in the question coincidentally matches the new species that appear in outpainting is very low, reducing the risk of changing the original answer. Of course, outpainting does carry this risk, and we plan to reduce the appearance of new species in the future by adding negative prompts and conducting further exploration.\n\n**W3: Minor weaknesses**\n\nAlthough the amount of data included in our Animal-Bench is sufficient to support model evaluation, it may not be entirely adequate for model training. This is related to the difficulty of collecting animal videos, especially those of wild animals. Currently, we have only taken the first step. In the future, we will add more data to Animal-Bench and conduct model fine-tuning to ultimately achieve the goal of applying the pre-trained model in the field of wildlife protection.\n\n**W4: Writing**\n\nThank you for your valuable suggestions. We will make the necessary improvement in the subsequent revisions.\n\n**Q1: Is the accuracy drop in Table 2 relative drop or absolute drop?**\n\nIt is the relative drop that provides a clearer demonstration of the model's performance changes and differences under various conditions.\n\n**L1: Potential negative impacts**\n\nPlease kindly refer to \"Author Rebuttal\" Q3.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank all reviewers for their valuable comments.

Due to the word limit for responses to each reviewer, we respond to common questions that were mentioned more than once here, and respond to other questions in the individual reviewer responses. In the numbering, "W" indicates a response to the "Weaknesses" part of the review, "Q" indicates a response to the "Questions" part, and "L" indicates a response to the "Limitations" part. We hope our replies will resolve your concerns.

**Q1: The resolution and duration**

Since our data is selected from multiple datasets, the resolutions of the original videos are not uniform. Most of the data have a resolution greater than or equal to 640\*360, with the highest resolution being 1920\*1440. A small portion of the data has a resolution less than 640\*360, with the lowest resolution being 176\*132.

The duration of all our videos ranges from 0.13 seconds to 9.52 minutes, with an average duration of 9.07 seconds. Due to different tasks requiring varying amounts of temporal information, the average video duration differs across tasks. For instance, the average duration for Object tasks is 4.13 seconds, while for Time tasks, it is 35.50 seconds.

**Q2: Option-setting methods**

Quantifying the difficulty of options is inherently challenging. In this study, we employ a qualitative analysis approach to achieve a moderate level of difficulty for the options.

For the action recognition task, we examined the frequency of various actions and found that they adhere to a long-tail distribution. We categorize common actions, or "head actions," such as "running" and "eating," as simple options that can be identified without specialized knowledge. In contrast, rare actions, or "tail actions," such as "molting" in birds, require specialized knowledge to identify and are thus classified as difficult options. Our approach involves incorporating the correct answer along with two simple options and one difficult option, thereby ensuring that the difficulty of the options is balanced and reflective of the natural frequency distribution of actions.

For the object recognition task, we tested four situations:

1. Random selection: Besides the correct answer, the other three options are randomly selected from all the animal species involved.

2. Different major categories: Besides the correct answer, the other three options are randomly selected from different major animal categories than the correct answer. This setting makes the question-answer pairs easier because it is a coarse-grained judgment. The difference between the other three options and the correct answer is large, and if the model can identify correctly at the coarse-grained level, it can answer correctly.

3. Same major category: Besides the correct answer, the other three options are randomly selected from the same major animal category as the correct answer. This setting makes the question-answer pairs more difficult because it is a fine-grained judgment. The difference between the other three options and the correct answer is small.

4. Rules designed in this paper: Besides the correct answer, two options come from a different major animal category than the correct answer, and one option comes from the same major animal category as the correct answer. This design makes the question-answer pairs neither too difficult nor too easy.

The following are the accuracy rates of videochat2's responses:

|Method|acc(%)|
|---------|---------|
|Random selection|91.97|
|Different major categories|97.60|
|Same major category|73.83|
|Rules designed in the paper|83.90|

The results indicate that the selection of options affects the experiment results, which also supports our theoretical analysis above. Our design can moderate the difficulty of the question-answer pairs, making the evaluation of the model more aligned with real-world scenarios.

**Q3: Potential negative impacts**

Although the technology itself is intended to protect and study wildlife, if it falls into the hands of malicious actors, it could be used for illegal hunting and animal exploitation. These actions could cause damage to wildlife populations. Also, excessive reliance on technology for animal monitoring and protection may lead to neglect of manual patrols and traditional conservation methods. Considering these potential negative impacts, we hope that the community adheres strictly to laws and regulations when applying relevant data and technology, to ensure the correct and safe use of technology.


**Q4: Parameter sensitivity**

While using a uniform set of parameters may not perfectly align with each model's best settings, this approach ensures fairness and standardization in evaluating all models. This helps in identifying relative performance differences due to model architecture rather than parameter tuning. Moreover, in real-world applications, optimal settings are not always known or achievable. Therefore, using uniform parameters also potentially validates the robustness of models to parameter variations. In the future, we will also conduct further research to test each model with other settings, thereby gaining a more comprehensive understanding of each model's potential.

Meta-review: This article presents a animal-centric dataset for the the assessment of multimodal video models. The works is timely and certainly will be of interest, thus it should be accepted. 

Firstly, the paper certainly needs careful grammar checking. Secondly, the reviewers provided a number of suggestions that the authors should consider for the final version. I have also identified a number of additional issues that I will mention below. 

Comments on the manuscript:

The authors argue that previous benchmarks have little focus on animals. However, this quantification is based on MVBench (Fig 1, top left), which seems anecdotal. Please support this statement also based on other data. 

A table comparing different benchmarks is missing (incl. the various sources). 

Examples in Figure 2 seem to be full of errors:
- Formatting, space at wrong spot in: “(A )Shaking head”
- It appears to me that the insect in the “action prediction” task is not the same and that it shows 2 different insects arriving (rather than one exciting)? [Temporal order unclear]
- For the answer to “Time”, how did the animal arrive in frame 2 vs. frame 1, if there was no locomotion at the beginning of the video? [“at the end” is marked as a correct example for this example]
- I’m honestly unsure about this one, but why would “circumanal gland signing behavior” be a social interaction?
- Typos in the reasoning example: “why does the cat jumps [..]”, the answer is also grammatically and factually incorrect (its a dog, not a cat).
- Typos in the counting examples “how many antelope does a leopard attack?”, “How many times does the panda shake its body back and forth (extra space) ?” — Also how can one know the correct answer (4) from 3 still images?
- The question of reviewer xc4z along those lines was answer with: "In reality, we have manually filtered them to make as few errors as possible. Despite the minor errors, these errors do not pertain to key information that can affect the ability of humans and video language pre-training models to choose the correct answer.”

If these typos are intended, I think the rationale for the typos should be explained and also indicated in the caption.

The figure captions are very sparse and do not explain the relevant context.
- Typos in lines 137, 140, 246 (pixel missing)
- Figure 7 misses a x-axis.
- Figure 5: clarify if change is in percent vs. in accuracy points.
- Apart from [25], references in 4.1 missing (for models).
- Table 1, can you calculate a confidence interval for the “random”model, and then indicate which models are actually significantly better than the random baseline?
- Formatting: Main table 1 -- the task acronyms seem to be nowhere defined (at least not in the main text)?

I'd find it great if the authors also report performance for the questions from TGIF-QA, MSRVTT-QA, and NExT-QA individually and compare the performance to SOTA models from those datasets. This would be interesting to 1) assess leakage (question from reviewer yXxA), as the datasets are older than the MLMs 2) show more baselines (and compare with the prior literature for those widely used datasets). Is it true that models are worse on the animal subset of those benchmarks?

Ceiling performance:

One of the arguments in the paper (also abstract) is that there is much room for improvement. The authors then focus on 7B models (for comparability). Simply evaluating bigger models would thus also be interesting to assess this claim. Given that the authors do not train, but just evaluate the models that seems feasible as well.
Decision: Accept (poster)