# Zero-shot animal behaviour classification with vision-language foundation models

Gaspard Dussert, Vincent Miele, Colin van Reeth, Anne Delestrade, Stéphane Dray, Simon Chamaillé-jammes

► To cite this version:

Gaspard Dussert, Vincent Miele, Colin van Reeth, Anne Delestrade, Stéphane Dray, et al.. Zero-shot animal behaviour classification with vision-language foundation models. Methods in Ecology and Evolution, 2025, 16 (7), pp.1460-1472. 10.1111/2041-210x.70059. hal-05093147

HAL Id: hal-05093147

https://hal.science/hal-05093147v1

Submitted on 2 Jun 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
<figure><img src="image_2.png" /></figure>

# Zero-shot animal behaviour classification with vision-language foundation models

Gaspard Dussert<fnref n="1" /> ID | Vincent Miele<fnref n="1" />,<fnref n="2" /> ID | Colin Van Reeth<fnref n="3" /> ID | Anne Delestrade<fnref n="3" /> ID |
Stéphane Dray<fnref n="1" /> ID | Simon Chamaillé-Jammes<fnref n="4" /> ID

## Abstract

1. Understanding the behaviour of animals in their natural habitats is critical to ecology and conservation. Camera traps are a powerful tool to collect such data with minimal disturbance. They however produce a very large quantity of images, which can make human-based annotation cumbersome or even impossible. While automated species identification with artificial intelligence has made impressive progress, the automatic classification of animal behaviours in camera trap images remains a developing field.

2. Here, we explore the potential of foundation models, specifically vision-language models (VLMs), to perform this task without the need to first train a model, which would require some level of human-based annotation. Using two datasets, of alpine and African fauna, we investigate the zero-shot capabilities of different kinds of recent VLMs to predict behaviours and estimate behaviour-specific diel activity patterns in three ungulate species.

3. By comparing our predictions to behaviours annotated by participatory science, our results show that using these automatic methods, it is possible to achieve F1-score as high as 86.39% in behaviour classification and produce activity patterns that closely align with those derived from participatory science data (overlap indexes between 84% and 90%).

4. These findings demonstrate the potential of foundation models and vision-language models in ecological research. Ecologists are encouraged to adopt these new methods and leverage their full capabilities to facilitate ecological studies.

## KEYWORDS

activity pattern, behaviour classification, camera trap, downstream task, foundation model, machine learning, vision-language model, zero-shot

<figure><img src="image_3.png" /></figure>

<footnote n="1">¹Université Lyon 1, CNRS, UMR 5558, Laboratoire de Biométrie et Biologie Evolutive (LBBE), Villeurbanne, France</footnote>

<footnote n="2">²Laboratoire d'Ecologie Alpine (LECA), Université Grenoble Alpes, CNRS, Université Savoie Mont Blanc, Chambéry, France</footnote>

<footnote n="3">³Centre de Recherches sur les Ecosystèmes d'Altitude (CREA Mont-Blanc), Observatoire du Mont-Blanc, Chamonix, France</footnote>

<footnote n="4">⁴CEFE, Univ Montpellier, CNRS, EPHE, IRD, Montpellier, France</footnote>

<footnote>**Correspondence**
Simon Chamaillé-Jammes
Email: simon.chamaille@cefe.cnrs.fr</footnote>

<footnote>**Handling Editor:** Lorna Hernandez-Santin</footnote>
<figure><img src="image_4.png" /></figure>

non-exclusive classes: *eating*, *resting*, *moving*, *standing* and *interacting*. As the proposed method does not require a training set, both datasets can be seen as test sets.

Figure 1 shows sample images of both datasets. The distribution of species and behaviours per dataset is available in Tables S1 and S2.

## 2.1.1 | CREA Mont-Blanc camera trap dataset

This dataset was collected by CREA Mont-Blanc (Centre de Recherches sur les Ecosystèmes d'Altitude) in the valley of Chamonix in the French Alps. Camera trapping does not involve any direct contact with the animals and thus does not require animal ethics approval under French law. The dataset is made of 131,537 images of 20 species, collected between 2017 and 2020 during 24,580 events, events being defined as 1-min periods that can include one or several bursts of images. Each event had been viewed by at least 30 people at the end of the Zooniverse annotation process.

Since the participants could only annotate the behaviours for the entire event, rather than for each individual or image, the annotations can sometimes be ambiguous: if the votes are spread across multiple behaviours, it is not possible to determine which specific images or individuals exhibited those behaviours. To address this, we created a subset of the dataset by keeping only the events where the most voted behaviour had at least 90% of the total votes. This resulted in a *high-agreement* dataset of 5566 events, which we will use to evaluate the classification performance of our models. However, for a realistic real-world application where votes are not available, we will estimate the activity patterns using the full dataset, rather than just the *high-agreement* subset.

We have focused our work on three behaviours: *eating*, *moving* and *resting*. *Standing* was excluded due to inconsistent interpretation

by participants and its frequent combination with other behaviours, while *interacting* has only 41 events in the entire dataset.

## 2.1.2 | Snapshot Serengeti

This large dataset was collected in the Serengeti national park (Tanzania) from 2010 to 2016 and has more than 7 million images (Swanson et al., 2015). We use the data preprocessed by WildCLIP (Gabeff et al., 2023), which uses capture seasons 1 to 6 (2010–2013) and evaluate all the models using their test set, on the same behaviours (eating, moving and resting) and 38 species. To evaluate the classification performance, we also created a *high-agreement* dataset for Snapshot Serengeti, which contained 11,757 events.

## 2.2 | Zero-shot vision-language models

Vision-language models (VLMs) are multimodal deep-learning models that can process both image and text simultaneously. These models, often trained on very large datasets, demonstrate very good zero-shot performances in downstream tasks. Here, we investigate the zero-shot capabilities of two categories of VLMs: contrastive vision-language models and multimodal large language models (LLMs). A brief overview of these two categories is given below, for a comprehensive review, see Bordes et al. (2024).

### 2.2.1 | Contrastive vision-language models

These models are trained to predict the similarity between a text and an image. Formally, a vision encoder and a text encoder are used to extract embeddings of images and texts, and the similarity

<figure><img src="image_5.png" /><figcaption>FIGURE 1 Sample images of the three behaviours of interest. The Zooniverse label and proportion of vote is shown at the top in green, and the prediction of the best zero-shot model (CogVLM) with its confidence score is shown at the bottom in white. The first four columns show images from the CREA Mont-Blanc dataset and the next four from the Snapshot Serengeti dataset.</figcaption></figure>
of an image-text pair is obtained through the cosine similarity of the normalized embeddings (see Figure 2a). During the training, a contrastive image-text objective is used to bring closer the embeddings of positive image-text pairs and push apart those of negative pairs. In our study, we decided to use CLIP (ViT-B/16-224) (Radford et al., 2021) for our baseline as the first contrastive vision-language model of large scale, and SigLIP (ViT-SO400M-384) (Zhai et al., 2023) which improved on CLIP with a new contrastive objective. We also include WildCLIP (ViT-B/16-224) (Gabeff et al., 2023), a fine-tuned version of CLIP trained exclusively on camera trap images of African fauna. WildCLIP uses the large Snapshot Serengeti dataset (Swanson et al., 2015), with image captions built from attributes labelled through participatory science, including animal behaviour.

## 2.2.2 | Multimodal large language models

These models process both image and text tokens at the input of a large language model (LLM). The image tokens are obtained using a vision encoder, typically a pretrained contrastive model such as CLIP. The integration of a language model allows these models to perform more complex reasoning compared to contrastive models (Wang, Chen, et al., 2024). We examine three multimodal LLMs of varying sizes: CogVLM (Wang, Lv, et al., 2024), which has a very large vision encoder (EVA2-CLIP-E); MobileVLM V2 (Chu et al., 2024), a lightweight VLM which uses CLIP ViT-L/14; and the more recent PaliGemma (Beyer et al., 2024), which utilizes SigLIP-SO.

## 2.3 | Zero-shot inference

### 2.3.1 | General methodology

For all methods, images are first cropped using MegaDetector v5 (Beery et al., 2019) to ensure that the image encoder of each model

extracts information related to the animal and not to the rest of the scene.

Contrastive vision-language models such as CLIP are able to perform zero-shot classification (Radford et al., 2021) using their ability to predict the similarity score between an image and a text. Here, we follow this methodology and match the images with the following captions: 'a camera trap picture of an animal {behavior}', which were used by Gabeff et al. (2023) for WildCLIP's training. This process is illustrated on Figure 2a. We calculate similarity scores for each behaviour, normalize them using the softmax function and select the behaviour with the highest score as the final prediction. We use the normalized similarity score of the chosen behaviour as the confidence score.

For the multimodal LLMs, we perform a Visual Question Answering task (Agrawal et al., 2016). As shown on Figure 2b, we prompt the VLM with the following question: 'Is the animal in the image eating, moving or resting? Answer the question using a single word.' This prompt template is used to clearly specify the expected output format to the model (Liu et al., 2024). For each word prediction in the answer, the model considers all possible words in the vocabulary, assigns a score to each one and predicts the one with the highest score. In our case, since the model predicts directly the behaviour with the first word, we can use it to retrieve the score of each behaviour. Following Jiang et al. (2021), we normalize the predicted behaviour's score by the sum of the scores of the candidate's words only (the three behaviours words).

### 2.3.2 | Providing additional context

We also explore if two additional contexts could help to improve model performances. In the first one, we inform the model whether the image was taken during daytime or nighttime. For CLIP-based methods, we change the caption to 'a camera trap picture of an animal {behavior}, captured during {context}.' with context being either 'nighttime' or 'daytime'. For LLMs, we add 'This image is captured

<figure><img src="image_6.png" /><figcaption>**FIGURE 2** Overview of the zero-shot inference process for the two model families we tested. Contrastive vision-language model (a) predicts the similarity between the image and the three captions, one for each behaviour. The predicted behaviour is the one with the highest similarity score (light blue). Multimodal large language models (LLMs) (b) are language models that can process visual tokens in addition to the prompt and predict the behaviour as the next word in the sequence.</figcaption></figure>
during {context}.' before the question. In the second one, we provide
the species name by replacing 'animal' with the actual species name
in both prompts.

2.3.3 | Prompt formulation

To evaluate the influence of prompt formulation on the models'
performance, we tested variations of the prompts used for behav-
iour classification. We compared the baseline caption with alter-
native phrasings, including shortened descriptions (e.g. 'an animal
{behavior}'), reformulations with additional context (e.g. 'an image
of an animal in the wild that is {behavior}') and synonym substitu-
tions (e.g. 'feeding' instead of *eating*). Prompt variations were tested
using the SigLIP model on the high-agreement subset of the CREA
Mont-Blanc dataset. We selected SigLIP for this analysis as it was
the best-performing contrastive model, offering a good balance be-
tween accuracy and computational efficiency. We did not evaluate
multimodal LLMs due to their significantly greater inference time,
which would make testing multiple prompt variations computation-
ally expensive and thus would have substantially increased the com-
putational carbon footprint of this study.

2.3.4 | Metrics

The classification performance of the model is evaluated with
the recall and precision of each behaviour. The calibration of the
scores is measured with the Expected Calibration Error (ECE) (Guo
et al., 2017). For these metrics, we evaluate the model only on the
high-agreement dataset. This subset may include more easy samples
than the full dataset, but the lower amount of noise in the labels
should allow us to more accurately evaluate and discriminate be-
tween models. To mitigate class imbalance, we also report macro-
averaged recall, precision, and F1-scores, averaging the metric
equally across behaviours to prevent dominance by a single class. For
all classification metrics, tables in the Result section report the mean
value of 1000 bootstrap replicates. Confidence intervals are avail-
able in the Supporting Information. Finally, we aggregate individual
images predictions to event predictions to improve the accuracy of
the models: for each sequence of cropped images, we compute the
sequence-level confidence scores of the three behaviours by aver-
aging the confidence scores of all images in that sequence (Dussert,
Chamaillé-Jammes, et al., 2024).

2.4 | Behaviour-specific activity patterns

Changes in time at which animals decide to conduct their activi-
ties, be it foraging, move or rest, is often revealing of the con-
straints they are exposed to. Because of this, diel activity patterns
are often used in comparative studies looking at describing the re-
sponse of animals to, for instance, predation risk, human activities

or anthropization in general (Gilbert et al., 2023). Here, we com-
puted diel activity patterns either on the Zooniverse annotations
or on the model predictions using the time at which the camera
trap events were triggered. As proposed by Vazquez et al. (2019),
times are first converted to radian and standardized using two an-
chor time (the average sunrise time and the average sunset time)
to account for the variation of the length of the day during the
year. A circular kernel density (of the von Mises distribution) is fit-
ted to the radian times to obtain the activity pattern. The overlap
index Δ is used to compare two activity patterns by computing
the integral of the minimum between the two-density function
(Ridout & Linkie, 2009). An index of 1 is obtained for a perfect
overlap and 0 for no overlap at all. In practice, it is computed with
the estimator Δ̂₄ of Ridout and Linkie (2009). These operations
were done with the functions *solartime*, *fitact* and *compareCkern*
of the activity package in R (Rowcliffe, 2023), with 200 bootstrap
replicates for the estimation of Δ̂₄.

Activity patterns are computed using the complete dataset for
the three behaviours of interest. For Zooniverse annotations, we
keep only the events for which a behaviour has at least 50% of the
votes. For model prediction, we used the same threshold of 0.75 on
the confidence scores of all the models to reduce false positives. We
did not apply any filter to ensure independence between event (e.g.
30min interval between two events) as it has been shown to bias the
activity patterns (Peral et al., 2022).

3 | RESULTS

3.1 | Behaviour classification

The performance of the six vision-language models (VLMs) on
the CREA Mont-Blanc high-agreement dataset is summarized in
Table 1. Among the contrastive models, SigLIP achieved the high-
est macro F1-score (85.67%), significantly outperforming the CLIP
baseline (63.07%). The specialized WildCLIP model performed
comparably to SigLIP, achieving a slightly lower F1-score but dem-
onstrating better score calibration. Of all the multimodal LLMs,
CogVLM exhibited the best performance, reaching a macro F1-
score of 86.39% with high recall across all behaviours (99.39% for
eating, 89.61% for moving and 99.05% for resting). The two other
LLMs (MobileVLM V2 and PaliGemma) underperformed, mainly
due to lower precision across behaviours. Performance on the
Snapshot Serengeti dataset (Table 2) was generally similar, except
for WildCLIP, which outperformed all other models with an F1-
score of 88.61%.

Providing day-night context improved the macro recall of all
contrastive models (Table 3), but decreased it for multimodal LLMs.
Additionally, precision decreased across almost all models, often re-
sulting in a lower macro F1-score compared to prompts without this
context. Specifying species names in the prompt led to even worse
F1-scores. In more detail, Table 3 shows that for most models, adding
context enhanced recall for eating behaviour while simultaneously
<table><caption>TABLE 1 Behavioural classification performances of six studied models, on the CREA Mont-Blanc high-agreement dataset.</caption><thead><tr><th rowspan="2">Model</th><th colspan="2">Eating</th><th colspan="2">Moving</th><th colspan="2">Resting</th><th colspan="2">Macro-averaged</th><th rowspan="2">F1-score</th><th rowspan="2">ECE (%)</th><th rowspan="2">No. params</th></tr><tr><th>Recall</th><th>Precision</th><th>Recall</th><th>Precision</th><th>Recall</th><th>Precision</th><th>Recall</th><th>Precision</th></tr></thead><tbody><tr><td colspan="13"><strong>Contrastive models</strong></td></tr><tr><td>CLIP (ViT-B/16)</td><td>31.31</td><td>69.77</td><td>88.29</td><td>91.70</td><td>91.09</td><td>40.50</td><td>70.23</td><td>67.32</td><td>63.07</td><td>15.88</td><td>150M</td></tr><tr><td>SigLIP (ViT-SO400M)</td><td>84.87</td><td><strong>75.31</strong></td><td><strong>92.87</strong></td><td>97.51</td><td>94.60</td><td>72.50</td><td>90.78</td><td><strong>81.77</strong></td><td>85.67</td><td>10.72</td><td>878M</td></tr><tr><td>WildCLIP (ViT-B/16)</td><td>89.47</td><td>74.12</td><td>92.26</td><td>98.19</td><td>92.72</td><td>70.46</td><td>91.48</td><td>80.92</td><td>85.42</td><td>7.66</td><td>150M</td></tr><tr><td colspan="13"><strong>Multimodal LLMs</strong></td></tr><tr><td>CogVLM</td><td><strong>99.39</strong></td><td>67.10</td><td>89.61</td><td><strong>99.93</strong></td><td>99.05</td><td><strong>73.80</strong></td><td><strong>96.01</strong></td><td>80.28</td><td><strong>86.39</strong></td><td>12.24</td><td>18B</td></tr><tr><td>MobileVLM V2</td><td>95.90</td><td>49.16</td><td>75.13</td><td>99.67</td><td><strong>99.07</strong></td><td>46.64</td><td>90.03</td><td>65.16</td><td>71.36</td><td><strong>6.02</strong></td><td>7B</td></tr><tr><td>PaliGemma</td><td>90.93</td><td>72.02</td><td>86.45</td><td>98.74</td><td>96.49</td><td>50.66</td><td>91.29</td><td>73.80</td><td>79.66</td><td>7.13</td><td>3B</td></tr></tbody></table>

Note: The table reports per-class recall and precision for the behaviours (eating, moving, resting) and macro-averaged recall, precision and F1-score, along with the expected calibration error (ECE). Bold indicates the best value per metric.

<table><caption>TABLE 2 Behavioural classification performances of six studied models, on the Snapshot Serengeti high-agreement dataset.</caption><thead><tr><th rowspan="2">Model</th><th colspan="2">Eating</th><th colspan="2">Moving</th><th colspan="2">Resting</th><th colspan="2">Macro-averaged</th><th rowspan="2">F1-score</th><th rowspan="2">ECE (%)</th></tr><tr><th>Recall</th><th>Precision</th><th>Recall</th><th>Precision</th><th>Recall</th><th>Precision</th><th>Recall</th><th>Precision</th></tr></thead><tbody><tr><td colspan="13"><strong>Contrastive models</strong></td></tr><tr><td>CLIP (ViT-B/16)</td><td>9.72</td><td>73.07</td><td>85.54</td><td>79.57</td><td>68.61</td><td>18.21</td><td>54.62</td><td>56.95</td><td>42.79</td><td>4.30</td></tr><tr><td>SigLIP (ViT-SO400M)</td><td>79.02</td><td>82.85</td><td>91.37</td><td>92.30</td><td>77.93</td><td>58.54</td><td>82.77</td><td>77.90</td><td>79.86</td><td>10.67</td></tr><tr><td>WildCLIP<sup>a</sup> (ViT-B/16)</td><td>94.20</td><td><strong>88.97</strong></td><td><strong>94.45</strong></td><td>97.66</td><td>85.20</td><td><strong>72.41</strong></td><td>91.29</td><td><strong>86.35</strong></td><td><strong>88.61</strong></td><td>8.69</td></tr><tr><td colspan="13"><strong>Multimodal LLMs</strong></td></tr><tr><td>CogVLM</td><td><strong>99.05</strong></td><td>70.32</td><td>83.18</td><td><strong>99.83</strong></td><td><strong>97.68</strong></td><td>68.81</td><td><strong>93.31</strong></td><td>79.65</td><td>84.58</td><td>5.07</td></tr><tr><td>MobileVLM V2</td><td>91.56</td><td>57.18</td><td>63.87</td><td>99.61</td><td>97.15</td><td>31.28</td><td>84.19</td><td>62.69</td><td>65.18</td><td><strong>2.96</strong></td></tr><tr><td>PaliGemma</td><td>93.50</td><td>73.15</td><td>84.74</td><td>97.63</td><td>90.48</td><td>59.16</td><td>89.57</td><td>76.65</td><td>81.45</td><td>3.24</td></tr></tbody></table>

Note: Bold indicates the best value per metric.

reducing its precision, whereas the opposite trend was observed for moving behaviour.

None of the models demonstrated strong score calibration, with the best-calibrated model (MobileVLM V2) achieving an Expected Calibration Error (ECE) of 6.02%. Moreover, adding context slightly deteriorated calibration in almost all models.

As shown in Table 4, prompt formulation had a significant impact on SigLIP predictions. Shorter captions drastically reduced performance, with macro F1-scores dropping to 10.57% when behaviour names were presented alone. In contrast, reformulated captions performed comparably to the baseline, with F1-scores between 83.35% and 85.13%. The use of behaviour synonyms also affected model performance: using *feeding*, *walking* or *lying down* slightly reduced performance (F1-scores between 82.12% and 84.18%), whereas using *foraging*, *wandering* or *sleeping* led to a more substantial performance drop (F1-scores between 76.61% and 78.68%).

A visual analysis of the classification errors with the best-performing model (CogVLM) reveals that some errors can be

attributed to a poor scene understanding: an animal walking in the foreground might be misclassified as *eating* if its head aligns with background vegetation, or misclassified as *resting* if its legs are hidden by deep snow or vegetation (Figure 3).

## 3.2 | Behaviour-specific activity patterns

We examined the behaviour-specific activity patterns of the three most abundant species in the CREA Mont-Blanc dataset: chamois, roe deer and red deer (for other species of this dataset and for species found in the Snapshot Serengeti dataset, see Tables S8–S13).

Quantitative measures of the overlap in activity patterns between the one derived from Zooniverse annotations and those derived from the different models are shown in Table 5. Among contrastive models, SigLIP and WildCLIP performed best, with average overlap indices of 84.00% and 87.98%, respectively. However, the baseline CLIP model performed significantly worse, with an overlap

<footnote><sup>a</sup>The model was trained on Snapshot Serengeti</footnote>
<table><caption>TABLE 3 Comparison of model performance between the base prompt (Table 1) and the prompt with additional context, across the six studied models on the CREA Mont-Blanc high agreement dataset.</caption><thead><tr><th rowspan="2">Model</th><th colspan="2">Context</th><th colspan="2">Eating</th><th colspan="2">Moving</th><th colspan="2">Resting</th><th colspan="2">Macro-averaged</th><th rowspan="2">ECE (%)</th></tr><tr><th>Day/night</th><th>Species</th><th>Recall</th><th>Precision</th><th>Recall</th><th>Precision</th><th>Recall</th><th>Precision</th><th>Recall</th><th>Precision</th><th>F1-score</th></tr></thead><tbody><tr><td colspan="13"><strong>Contrastive models</strong></td></tr><tr><td>CLIP (ViT-B/16)</td><td>✓</td><td>×</td><td>16.81</td><td>-11.35</td><td>-4.17</td><td>0.97</td><td>2.41</td><td>1.12</td><td>5.02</td><td>-3.08</td><td>3.1</td><td>1.91</td></tr><tr><td></td><td>×</td><td>✓</td><td>0.15</td><td>-0.02</td><td>-0.02</td><td>0.03</td><td>0.06</td><td>0.05</td><td>0.06</td><td>0.02</td><td>0.07</td><td>0.03</td></tr><tr><td>SigLIP (ViT-SO400M)</td><td>✓</td><td>×</td><td>9.95</td><td>-10.73</td><td>-5.93</td><td>1.51</td><td>0.24</td><td>-11.83</td><td>1.42</td><td>-7.02</td><td>-4.54</td><td>3.11</td></tr><tr><td></td><td>×</td><td>✓</td><td>-0.03</td><td>-0.12</td><td>0.0</td><td>0.0</td><td>-0.06</td><td>0.02</td><td>-0.03</td><td>-0.03</td><td>-0.03</td><td>0.0</td></tr><tr><td>WildCLIP (ViT-B/16)</td><td>✓</td><td>×</td><td>0.88</td><td>-1.41</td><td>-0.75</td><td>0.25</td><td>1.88</td><td>-1.17</td><td>0.68</td><td>-0.77</td><td>-0.29</td><td>0.68</td></tr><tr><td></td><td>×</td><td>✓</td><td>0.06</td><td>-0.02</td><td>-0.02</td><td>0.0</td><td>-0.01</td><td>0.06</td><td>0.01</td><td>0.02</td><td>0.01</td><td>0.0</td></tr><tr><td colspan="13"><strong>Multimodal LLMs</strong></td></tr><tr><td>CogVLM</td><td>✓</td><td>×</td><td>0.16</td><td>-19.47</td><td>-8.79</td><td>-0.01</td><td>-0.23</td><td>0.75</td><td>-2.95</td><td>-6.25</td><td>-6.8</td><td>-2.23</td></tr><tr><td></td><td>×</td><td>✓</td><td>0.16</td><td>-5.17</td><td>-3.27</td><td>0.02</td><td>0.0</td><td>-7.52</td><td>-1.03</td><td>-4.23</td><td>-3.59</td><td>-0.88</td></tr><tr><td>MobileVLM V2</td><td>✓</td><td>×</td><td>0.31</td><td>-4.34</td><td>-3.49</td><td>0.05</td><td>0.0</td><td>-1.55</td><td>-1.06</td><td>-1.95</td><td>-2.54</td><td>0.24</td></tr><tr><td></td><td>×</td><td>✓</td><td>1.53</td><td>-8.95</td><td>-6.31</td><td>0.17</td><td>-0.22</td><td>1.11</td><td>-1.66</td><td>-2.56</td><td>-3.77</td><td>0.06</td></tr><tr><td>PaliGemma</td><td>✓</td><td>×</td><td>1.64</td><td>-8.14</td><td>-0.11</td><td>0.06</td><td>-1.43</td><td>7.6</td><td>0.03</td><td>-0.15</td><td>0.33</td><td>2.24</td></tr><tr><td></td><td>×</td><td>✓</td><td>-0.36</td><td>-8.11</td><td>-1.09</td><td>-0.06</td><td>0.47</td><td>3.99</td><td>-0.33</td><td>-1.39</td><td>-0.88</td><td>-0.85</td></tr></tbody></table>

Note: Performance differences between context-enriched and base prompts are indicated in green (improved performance) and red (decreased performance).
<figure><img src="image_7.png" /></figure>

<table><caption>TABLE 4 Performance of SigLIP on macro-averaged metrics for different prompts on the CREA Mont-Blanc high-agreement dataset.</caption><thead><tr><th rowspan="2">Prompt</th><th colspan="3">Macro-averaged</th></tr><tr><th>Recall</th><th>Precision</th><th>F1-score</th></tr></thead><tbody><tr><td>Original caption</td><td></td><td></td><td></td></tr><tr><td>a camera trap picture of an animal (eating / moving / resting)</td><td>90.78</td><td>81.77</td><td>85.67</td></tr><tr><td>Shorter caption</td><td></td><td></td><td></td></tr><tr><td>a picture of an animal (eating / moving / resting)</td><td>89.46</td><td>78.77</td><td>83.08</td></tr><tr><td>an animal (eating / moving / resting)</td><td>87.90</td><td>63.13</td><td>68.63</td></tr><tr><td>(eating / moving / resting)</td><td>36.62</td><td>26.96</td><td>10.57</td></tr><tr><td>Caption reformulation</td><td></td><td></td><td></td></tr><tr><td>an image of an animal in the wild that is (eating / moving / resting)</td><td>81.97</td><td>89.58</td><td>85.13</td></tr><tr><td>a camera trap image capturing an animal (eating / moving / resting)</td><td>89.96</td><td>80.11</td><td>83.71</td></tr><tr><td>an image from a camera trap featuring an animal (eating / moving / resting)</td><td>88.54</td><td>79.94</td><td>83.35</td></tr><tr><td>Behaviour synonyms</td><td></td><td></td><td></td></tr><tr><td>a camera trap picture of an animal (feeding / moving / resting)</td><td>90.67</td><td>77.25</td><td>82.41</td></tr><tr><td>a camera trap picture of an animal (foraging / moving / resting)</td><td>89.54</td><td>73.71</td><td>78.68</td></tr><tr><td>a camera trap picture of an animal (eating / walking / resting)</td><td>89.04</td><td>78.01</td><td>82.12</td></tr><tr><td>a camera trap picture of an animal (eating / wandering / resting)</td><td>89.78</td><td>70.41</td><td>76.61</td></tr><tr><td>a camera trap picture of an animal (eating / moving / lyingdown)</td><td>89.71</td><td>80.10</td><td>84.18</td></tr><tr><td>a camera trap picture of an animal (eating / moving / sleeping)</td><td>83.29</td><td>73.25</td><td>77.18</td></tr></tbody></table>

Note: Bold indicates the best value per metric.

index of only 64.03%. For the multimodal LLMs, CogVLM achieved the highest overlap index (90.04%), followed by PaliGemma (87.15%) and MobileVLM V2 (88.16%). The quality of activity pattern estimations also varied by behaviour: models generally achieved higher overlap indices for resting and moving behaviours, while predictions for eating behaviour showed greater variation across models. For instance, the overlap index for eating behaviour was 91.67% for CogVLM but only 36.60% for CLIP.

Providing additional context about species or day-night time affected activity pattern estimation in a manner similar to behaviour classification: adding species information decreases the overall overlap index for all models except WildCLIP, while adding day/night context benefits only the contrastive models (Table 6).

A visual comparison of the activity patterns generated by different models is shown in Figure 4. CogVLM produced activity curves that closely matched those derived from participatory science annotations. Even minor variations in activity levels throughout the day were well captured. In contrast, CLIP exhibited a strong bias towards predicting moving at nighttime, leading to unreliable activity patterns. To a lesser extent, SigLIP also showed a nighttime bias, particularly in its estimations of red deer activity patterns.

## 4 | DISCUSSION

For the first time, we show that a foundation model can be used without fine-tuning or re-training, without complicated implementation or high programming skills, to extract, using text only (prompts), an ecologically relevant information from an image. We propose to use a zero-shot approach with general-purpose vision-language models (i.e. non-specialized models) to address ecological issues, using text to formulate the question about animal behaviour. We show that these models can have impressive performance, with a macro F1-score reaching 86.39% for CogVLM. Furthermore, the diel activity patterns generated by these models closely align with those obtained through participatory science, exhibiting overlap indexes between 84% and 90% (excluding the baseline CLIP) for three mountain ungulates.

Using an automated and flexible method, such as the one presented in this study to classify animal behaviours, can save considerable time compared to image annotations by citizen scientists or experts. For instance, the SigLIP model can process tens of thousands of pictures in an hour (see below for further discussion). Therefore, this approach could enable large-scale annotation of camera trap datasets and facilitate or automate the study of behaviours, which could, for example, be particularly useful for studying the effects of climate change or human activities on the behaviour of wild animals using camera traps (Bison et al., 2024; Blount et al., 2024). Furthermore, even though this study illustrates the effectiveness of foundation vision-language models for a specific task, the flexibility of the method makes it easily adaptable to other ecological tasks. For example, it could be possible to predict human behaviours or landscape attributes by changing the prompt (or the caption) given to the model.

While our study highlights the good performances of foundation models on two Zooniverse-based datasets, some existing datasets such as (Chen et al., 2023) and Animal Kingdom (Ng et al., 2022) are also available with a higher diversity of behaviours. However, we decided to not consider these databases as they consist of very high-quality video footage. Good model performance on these datasets
<figure><img src="image_8.png" /><figcaption>FIGURE 3 Classification errors with CogVLM: (a) images wrongly classified as *eating*, (b) images wrongly classified as *moving* and (c) images wrongly classified as *resting*.</figcaption></figure>

<table><caption>TABLE 5 Overlap index averaged over the three species of interest on the complete dataset.</caption><thead><tr><th rowspan="2">Model</th><th colspan="4">Overlap index (%)</th></tr><tr><th>Eating</th><th>Moving</th><th>Resting</th><th>All</th></tr></thead><tbody><tr><td colspan="5">Contrastive models</td></tr><tr><td>CLIP (ViT-B/16)</td><td>36.60</td><td>66.97</td><td>88.53</td><td>64.03</td></tr><tr><td>SigLIP (ViT-SO400M)</td><td>75.27</td><td>86.27</td><td>90.47</td><td>84.00</td></tr><tr><td>WildCLIP (ViT-B/16)</td><td>91.03</td><td>80.63</td><td>92.27</td><td>87.98</td></tr><tr><td colspan="5">Multimodal LLMs</td></tr><tr><td>CogVLM</td><td>91.67</td><td>87.17</td><td><strong>91.30</strong></td><td><strong>90.04</strong></td></tr><tr><td>MobileVLM V2</td><td><strong>93.43</strong></td><td>83.37</td><td>87.67</td><td>88.16</td></tr><tr><td>PaliGemma</td><td>88.17</td><td><strong>90.47</strong></td><td>82.83</td><td>87.15</td></tr></tbody></table>

Note: Bold indicates the best value per metric.

may not transfer to camera trap datasets given the distribution gap between the two. Additionally, we decided not to focus on video processing, as many camera trap studies use single shot or burst mode, requiring the development of methods to obtain the most accurate behaviour possible from one or a few images (Rovero & Zimmermann, 2016).

A visual analysis of discrepancies between model predictions and participatory science annotations reveals a degree of inherent ambiguity and that both may be correct in certain instances. Some sequences contain an animal that displays multiple behaviours within the same sequence, with eating and moving being the most commonly co-occurring behaviours. While not common,

these ambiguities may explain the slight deviation in the diel activity curves obtained from CogVLM, despite its near-perfect recall on the high-agreement dataset. This suggests that exploring a zero-shot multilabel classification approach might be useful to predict multiple behaviours in a single event. Finally, by working with cropped images, our approach is limited to predicting the behaviour of single individuals and cannot predict interactions between two or more individuals in the same image. These problems could potentially be eliminated using VLMs capable of simultaneously processing multiple images from the same sequence, rather than independently.

Given that several of the models achieve very good performance, it is interesting to study the trade-off between model size and performance. When considering multimodal LLMs, CogVLM appears to be the best model if inference time is not a concern while MobileVLM V2 is clearly not interesting, as its performance is worse than the one of WildCLIP, which is 50 times smaller. The difference in performance is highly likely due to the differing sizes of the vision encoders between CogVLM (10B parameters) and MobileVLM V2 (428M parameters). For contrastive models, WildCLIP seems to be the most efficient model. It performs considerably better than CLIP ViT-Base, despite having the same number of parameters. It is worth noting that WildCLIP has been trained on images of African fauna only, but still manages to predict well on European fauna. The release of our data can facilitate the development of specialized models more robust to the changes in species, habitats and behaviours. Finally, SigLIP seems to be the most efficient non-specialized foundation model and has a reasonable inference time (1h for 40,000 images on one NVIDIA V100 GPU), which could also make it useful for analyses
<figure><img src="image_9.png" /></figure>

<table><caption>TABLE 6 Comparison of overlap index between the base the prompt (Table 1) and the prompt with additional context, across the six studied models on the complete dataset.</caption><thead><tr><th rowspan="2">Model</th><th colspan="2">Context</th><th colspan="4">Average overlap index (%)</th></tr><tr><th>Day/night</th><th>Species</th><th>Eating</th><th>Moving</th><th>Resting</th><th>All</th></tr></thead><tbody><tr><td colspan="7"><strong>Contrastive models</strong></td></tr><tr><td>CLIP (ViT-B/16)</td><td>✓</td><td>×</td><td>3.13</td><td>8.83</td><td>2.74</td><td>4.90</td></tr><tr><td></td><td>×</td><td>✓</td><td>-3.03</td><td>14.93</td><td>-29.43</td><td>-5.84</td></tr><tr><td>SigLIP (ViT-SO400M)</td><td>✓</td><td>×</td><td>14.70</td><td>-1.14</td><td>3.26</td><td>5.61</td></tr><tr><td></td><td>×</td><td>✓</td><td>2.16</td><td>1.06</td><td>-6.30</td><td>-1.02</td></tr><tr><td>WildCLIP (ViT-B/16)</td><td>✓</td><td>×</td><td>0.17</td><td>-0.40</td><td>-1.27</td><td>-0.50</td></tr><tr><td></td><td>×</td><td>✓</td><td>-0.50</td><td>12.04</td><td>-8.70</td><td>0.94</td></tr><tr><td colspan="7"><strong>Multimodal LLMs</strong></td></tr><tr><td>CogVLM</td><td>✓</td><td>×</td><td>1.33</td><td>0.20</td><td>-6.03</td><td>-1.50</td></tr><tr><td></td><td>×</td><td>✓</td><td>0.56</td><td>0.26</td><td>-3.97</td><td>-1.04</td></tr><tr><td>MobileVLM V2</td><td>✓</td><td>×</td><td>-0.33</td><td>3.49</td><td>-4.04</td><td>-0.30</td></tr><tr><td></td><td>×</td><td>✓</td><td>-0.36</td><td>0.26</td><td>-3.14</td><td>-1.08</td></tr><tr><td>PaliGemma</td><td>✓</td><td>×</td><td>0.83</td><td>-1.34</td><td>0.47</td><td>-0.01</td></tr><tr><td></td><td>×</td><td>✓</td><td>-0.70</td><td>-0.07</td><td>-0.60</td><td>-0.45</td></tr></tbody></table>

Note: Performance differences between context-enriched and base prompts are indicated in green (improved performance) and red (decreased performance).

<figure><img src="image_10.png" /><figcaption>FIGURE 4 Comparison of the activity patterns using either the participatory science labels from Zooniverse (black) or predictions of three models (colours). Each row corresponds to a different species of the CREA Mont-Blanc dataset (top to bottom: chamois, roe deer, red deer). Overlap values are indicated on the right of the curves.</figcaption></figure>
conducted on standard desktop CPU-only computers (1 h for 2400 images on one Intel i7-10700 CPU).

Providing additional context, such as the time of capture or the species name, yielded mixed results for both behaviour classification and activity pattern estimation, often leading to reduced performance. On the other hand, shorter captions with less contextual information clearly degraded the predictive performance. Thus, identifying the appropriate balance is crucial to leverage the full potential of these models. Although testing multiple prompt variations with contrastive models is almost computationally free, doing so with multimodal LLMs is considerably more demanding, making extensive prompt experimentation impractical for large datasets. In addition to prompt formulation, careful selection of class labels is also essential, as synonym variations significantly influence classification performances. To address these issues, one approach could be to first experiment with multiple prompts on smaller subsets of data to identify effective wording before applying them at larger scales. Prompt ensembling could also be a potential strategy to reduce the sensitivity to prompt wording and improve overall performance (Radford et al., 2021). However, it is important to note that these experiments were conducted on only a few models, and in some cases exclusively with SigLIP, other VLMs may behave differently.

## 5 | CONCLUSIONS

Overall, our study raises questions regarding the necessity of training new models for ecological applications, especially considering the rapidly accelerating rate of publication of foundation models. Whereas training models to achieve high performance requires a strong expertise in deep learning, the manual annotation of thousands of images, and access to great computational power, foundation models offer a promising alternative to obtain good predictive performance without training. Moreover, the capabilities of AI models are evolving quickly, as shown by models like GPT-4 Vision (OpenAI, 2023) and OpenFlamingo (Awadalla et al., 2023), which can now analyse sequences of images, a capability particularly relevant for camera trap image processing. One of the fundamental elements of our study is that the approach is accessible to a large number of ecologists who are not experts in machine learning or in scientific computing. Ecologists should embrace these new methods, evaluate their performance and learn how to exploit them to their full potential.

### AUTHOR CONTRIBUTIONS

Gaspard Dussert, Simon Chamaillé-Jammes, Stéphane Dray and Vincent Miele conceived the ideas and designed the methodology. Anne Delestrade and Colin Van Reeth gathered, annotated and processed the CREA Mont-Blanc dataset. Gaspard Dussert coded and performed the analysis. Gaspard Dussert wrote the first version of the manuscript; Simon Chamaillé-Jammes, Stéphane Dray, Anne Delestrade, Colin Van Reeth and Vincent Miele contributed critically to the drafts and gave final approval for publication.

### ACKNOWLEDGEMENTS

This work was granted access to the HPC resources of IDRIS under the allocation 2022-AD010113729 made by GENCI. We warmly thank the Zooniverse network of volunteers who labelled the large quantity of images from the Wild Mont-Blanc program.

### FUNDING INFORMATION

None.

### CONFLICT OF INTEREST STATEMENT

None of the authors has a conflict of interest.

### PEER REVIEW

The peer review history for this article is available at https://www.webofscience.com/api/gateway/wos/peer-review/10.1111/2041-210X.70059.

### DATA AVAILABILITY STATEMENT

The code and dataset are available on Zenodo: https://doi.org/10.5281/zenodo.10925926 (Dussert, Miele, et al., 2024).

### ORCID

Gaspard Dussert https://orcid.org/0000-0003-3833-8093

Vincent Miele https://orcid.org/0000-0001-7584-0088

Colin Van Reeth https://orcid.org/0000-0002-7070-9838

Anne Delestrade https://orcid.org/0000-0003-0249-2494

Stéphane Dray https://orcid.org/0000-0003-0153-1105

Simon Chamaillé-Jammes https://orcid.org/0000-0003-0505-6620

### REFERENCES

Agrawal, A., Lu, J., Antol, S., Mitchell, M., Zitnick, C. L., Batra, D., & Parikh, D. (2016). VQA: Visual question answering. arXiv:1505.00468 [cs]. http://arxiv.org/abs/1505.00468

Awadalla, A., Gao, I., Gardner, J., Hessel, J., Hanafy, Y., Zhu, W., Marathe, K., Bitton, Y., Gadre, S., Sagawa, S., Jitsev, J., Kornblith, S., Koh, P. W., Ilharco, G., Wortsman, M., & Schmidt, L. (2023). OpenFlamingo: An open-source framework for training large autoregressive vision-language models. arXiv:2308.01390. http://arxiv.org/abs/2308.01390

Beery, S., Morris, D., & Yang, S. (2019). Efficient pipeline for camera trap image review. arXiv:1907.06772 [cs]. https://doi.org/10.48550/arXiv.1907.06772

Beyer, L., Steiner, A., Pinto, A. S., Kolesnikov, A., Wang, X., Salz, D., Neumann, M., Alabdulmohsin, I., Tschannen, M., Bugliarello, E., Unterthiner, T., Keysers, D., Koppula, S., Liu, F., Grycner, A., Gritsenko, A., Houlsby, N., Kumar, M., Rong, K., ... Zhai, X. (2024). PaliGemma: A versatile 3B VLM for transfer. arXiv:2407.07726. https://doi.org/10.48550/arXiv.2407.07726

Bison, M., Yoccoz, N., Carlson, B., Bayle, A., & Delestrade, A. (2024). Camera traps reveal seasonal variation in activity and occupancy of the alpine mountain hare Lepus Timidus varronis. Wildlife Biology, 2024(4), e01186. https://doi.org/10.1002/wlb3.01186

Blount, J. D., Green, A. M., Chynoweth, M., Kittelberger, K. D., Hipólito, D., Bojarska, K., Çoban, E., Kusak, J., & Şekercioğlu, Ç. H. (2024). Seasonal activity patterns and home range sizes of wolves in the human-dominated landscape of northeast Türkiye. Wildlife Biology, 2024(6), e01257. https://doi.org/10.1002/wlb3.01257
<figure><img src="image_11.png" /></figure>

Bordes, F., Pang, R. Y., Ajay, A., Li, A. C., Bardes, A., Petryk, S., Mañas, O., Lin, Z., Mahmoud, A., Jayaraman, B., Ibrahim, M., Hall, M., Xiong, Y., Lebensold, J., Ross, C., Jayakumar, S., Guo, C., Bouchacour, D., Al-Tahan, H., ... Chandra, V. (2024). An introduction to vision-language modeling. arXiv: 2405.17247. https://doi.org/10.48550/arXiv.2405.17247

Chen, J., Hu, M., Coker, D. J., Berumen, M. L., Costelloe, B., Beery, S., Rohrbach, A., & Elhoseiny, M. (2023). MammalNet: A large-scale video benchmark for mammal recognition and behavior understanding. arXiv:2306.00576 [cs]. https://doi.org/10.48550/arXiv.2306.00576

Chu, X., Qiao, L., Zhang, X., Xu, S., Wei, F., Yang, Y., Sun, X., Hu, Y., Lin, X., Zhang, B., & Shen, C. (2024). MobileVLM V2: Faster and stronger baseline for vision language model. arXiv:2402.03766 [cs]. http://arxiv.org/abs/2402.03766

Dussert, G., Chamaillé-Jammes, S., Dray, S., & Miele, V. (2024). Being confident in confidence scores: Calibration in deep learning models for camera trap image sequences. https://doi.org/10.1101/2023.11.10.566512

Dussert, G., Miele, V., Van Reeth, C., Delestrade, A., Dray, S., & Chamaillé-Jammes, S. (2024). Data release of Zero-shot animal behavior classification with vision-language foundation models. Zenodo, https://doi.org/10.5281/zenodo.10925926

Fabian, Z., Miao, Z., Li, C., Zhang, Y., Liu, Z., Hernández, A., Montes-Rojas, A., Escucha, R., Siabatto, L., Link, A., Arbeláez, P., Dodhia, R., & Ferres, J. L. (2023). Multimodal foundation models for zero-shot animal species recognition in camera trap images. arXiv:2311.01064 [cs]. https://doi.org/10.48550/arXiv.2311.01064

Gabeff, V., Rußwurm, M., Tuia, D., & Mathis, A. (2023). WildCLIP: Scene and animal attribute retrieval from camera trap data with domain-adapted vision-language models. https://doi.org/10.1101/2023.12.22.572990

Gerber, B. D., Devarajan, K., Farris, Z. J., & Fidino, M. (2024). A model-based hypothesis framework to define and estimate the diel niche via the 'Diel.Niche' R package. Journal of Animal Ecology, 93(2), 132-146. https://doi.org/10.1111/1365-2656.14035

Gilbert, N. A., McGinn, K. A., Nunes, L. A., Shipley, A. A., Bernath-Plaisted, J., Clare, J. D. J., Murphy, P. W., Keyser, S. R., Thompson, K. L., Maresh Nelson, S. B., Cohen, J. M., Widick, I. V., Bartel, S. L., Orrock, J. L., & Zuckerberg, B. (2023). Daily activity timing in the anthropocene. Trends in Ecology & Evolution, 38(4), 324-336. https://doi.org/10.1016/j.tree.2022.10.008

Guo, C., Pleiss, G., Sun, Y., & Weinberger, K. Q. (2017). On calibration of modern neural networks. arXiv:1706.04599 [cs]. http://arxiv.org/abs/1706.04599

Jiang, Z., Araki, J., Ding, H., & Neubig, G. (2021). How can we know when language models know? On the calibration of language models for question answering. arXiv:2012.00955 [cs]. http://arxiv.org/abs/2012.00955

Kirillov, N. A., Mintun, E., Ravi, N., Mao, H., Rolland, C., Gustafson, L., Xiao, T., Whitehead, S., Berg, A. C., Lo, W.-Y., Dollár, P., & Girshick, R. (2023). Segment anything. arXiv:2304.02643 [cs]. http://arxiv.org/abs/2304.02643

Liu, D., Hou, J., Huang, S., Liu, J., He, Y., Zheng, B., Ning, J., & Zhang, J. (2023). LoTE-animal: A long time-span dataset for endangered animal behavior understanding. In 2023 IEEE/CVF international conference on computer vision (ICCV) (pp. 20007-20018). IEEE. https://doi.org/10.1109/ICCV51070.2023.01836

Liu, H., Li, C., Li, Y., & Lee, Y. J. (2024). Improved baselines with visual instruction tuning. arXiv:2310.03744. https://doi.org/10.48550/arXiv.2310.03744

Mitterwallner, V., Peters, A., Edelhoff, H., Mathes, G., Nguyen, H., Peters, W., Heurich, M., & Steinbauer, M. J. (2023). Automated visitor and wildlife monitoring with camera traps and machine learning. Remote Sensing in Ecology and Conservation, 10(2), 236-247. https://doi.org/10.1002/rse2.367

Ng, X. L., Ong, K. E., Zheng, Q., Ni, Y., Yeo, S. Y., & Liu, J. (2022). Animal kingdom: A large and diverse dataset for animal behavior understanding. In 2022 IEEE/CVF conference on computer vision and pattern recognition (CVPR) (pp. 19001-19012). IEEE. https://doi.org/10.1109/CVPR52688.2022.01844

Norouzzadeh, M. S., Nguyen, A., Kosmala, M., Swanson, A., Palmer, M. S., Packer, C., & Clune, J. (2018). Automatically identifying, counting, and describing wild animals in camera-trap images with deep learning. *Proceedings of the National Academy of Sciences of the United States of America*, 115(25), E5716-E5725. https://doi.org/10.1073/pnas.1719367115

OpenAI. (2023). GPT-4 vision. https://cdn.openai.com/papers/GPTV_System_Card.pdf

Peral, C., Landman, M., & Kerley, G. I. H. (2022). The inappropriate use of time-to-independence biases estimates of activity patterns of free-ranging mammals derived from camera traps. *Ecology and Evolution*, 12(10), e9408. https://doi.org/10.1002/ece3.9408

Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., Krueger, G., & Sutskever, I. (2021). Learning transferable visual models from natural language supervision. arXiv:2103.00020 [cs]. https://doi.org/10.48550/arXiv.2103.00020

Ridout, M. S., & Linkie, M. (2009). Estimating overlap of daily activity patterns from camera trap data. *Journal of Agricultural, Biological, and Environmental Statistics*, 14(3), 322-337 1537-2693. https://doi.org/10.1198/jabes.2009.08038

Rigoudy, N., Dussert, G., Benyoub, A., Besnard, A., Birck, C., Boyer, J., Bollet, Y., Bunz, Y., Caussimont, G., Chetouane, E., Carriburu, J. C., Cornette, P., Delestrade, A., De Backer, N., Dispan, L., Le Barh, M., Duhayer, J., Elder, J.-F., Fanjul, J.-B., ... Chamaillé-Jammes, S. (2023). The DeepFaune initiative: A collaborative effort towards the automatic identification of European fauna in camera trap images. *European Journal of Wildlife Research*, 69(6), 113. https://doi.org/10.1007/s10344-023-01742-7

Rovero, F., & Zimmermann, F. (2016). Camera trapping for wildlife research. Pelagic Publishing.

Rowcliffe, J. M. (2023). activity: Animal activity statistics. R package version 1.3.4. https://CRAN.R-project.org/package=activity

Rowcliffe, J. M., Kays, R., Kranstauber, B., Carbone, C., & Jansen, P. A. (2014). Quantifying levels of animal activity using camera trap data. *Methods in Ecology and Evolution*, 5(11), 1170-1179. https://doi.org/10.1111/2041-210X.12278

Schindler, F., & Steinhage, V. (2021). Identification of animals and recognition of their actions in wildlife videos using deep learning techniques. *Ecological Informatics*, 61, 101215. https://doi.org/10.1016/j.ecoinf.2021.101215

Schindler, F., Steinhage, V., van Beeck Calkoen, S. T. S., & Heurich, M. (2024). Action detection for wildlife monitoring with camera traps based on segmentation with filtering of Tracklets (SWIFT) and mask-guided action recognition (MAROON). *Applied Sciences*, 14(2), 514. https://doi.org/10.3390/app14020514

Swanson, A., Kosmala, M., Lintott, C., Simpson, R., Smith, A., & Packer, C. (2015). Snapshot Serengeti, high-frequency annotated camera trap images of 40 mammalian species in an African savanna. *Scientific Data*, 2(1), 150026. https://doi.org/10.1038/sdata.2015.26

Vazquez, C., Rowcliffe, J. M., Spoelstra, K., & Jansen, P. A. (2019). Comparing diel activity patterns of wildlife across latitudes and seasons: Time transformations using day length. *Methods in Ecology and Evolution*, 10(12), 2057-2066. https://doi.org/10.1111/2041-210X.13290

Wang, W., Lv, Q., Yu, W., Hong, W., Qi, J., Wang, Y., Ji, J., Yang, Z., Zhao, L., Song, X., Xu, J., Xu, B., Li, J., Dong, Y., Ding, M., & Tang, J. (2024). CogVLM: Visual expert for pretrained language models. arXiv:2311.03079 [cs]. http://arxiv.org/abs/2311.03079

Wang, Y., Chen, W., Han, X., Lin, X., Zhao, H., Liu, Y., Zhai, B., Yuan, J., You, Q., & Yang, H. (2024). Exploring the reasoning abilities of
multimodal large language models (MLLMs): A comprehensive survey on emerging trends in multimodal reasoning. *arXiv:2401.06805*. http://arxiv.org/abs/2401.06805

Whytock, R. C., Świeżewski, J., Zwerts, J. A., Bara-Stupski, T., Pambo, A. F. K., Rogala, M., Bahaa-el-din, L., Boekee, K., Brittain, S., Cardoso, A. W., Henschel, P., Lehmann, D., Momboua, B., Opepa, C. K., Orbell, C., Pitman, R. T., Robinson, H. S., & Abernethy, K. A. (2021). Robust ecological analysis of camera trap data labelled by a machine learning model. *Methods in Ecology and Evolution, 12*(6), 1080–1092. https://doi.org/10.1111/2041-210X.13576

Willi, M., Pitman, R. T., Cardoso, A. W., Locke, C., Swanson, A., Boyer, A., Veldthuis, M., & Fortson, L. (2019). Identifying animal species in camera trap images using deep learning and citizen science. *Methods in Ecology and Evolution, 10*(1), 80–91. https://doi.org/10.1111/2041-210X.13099

Zhai, X., Mustafa, B., Kolesnikov, A., & Beyer, L. (2023). Sigmoid loss for language image pre-training. arXiv:2303.15343 [cs]. http://arxiv.org/abs/2303.15343

## SUPPORTING INFORMATION

Additional supporting information can be found online in the Supporting Information section at the end of this article.

**Table S1:** Species and behaviour distribution of the CREA Mont-Blanc camera trap dataset for: (a) all events, (b) events for which the most voted behaviour has at least 90% of the votes.

**Table S2:** Species and behaviour distribution of the Snapshot Serengeti camera trap dataset for: (a) all events, (b) events for which the most voted behaviour has at least 90% of the votes.

**Table S3:** Confidence intervals for the CREA Mont-Blanc dataset.

**Table S4:** Confidence intervals for the Snapshot Serengeti dataset.

**Table S5:** Confidence intervals for the CREA Mont-Blanc dataset, with the day/night context.

**Table S6:** Confidence intervals for the CREA Mont-Blanc dataset, with the species context.

**Table S7:** Confidence intervals of SigLIP on macro-averaged metrics for different prompts on the CREA Mont-Blanc high-agreement dataset.

**Table S8:** Overlap index and standard errors of the CLIP model, for each species and behaviour.

**Table S9:** Overlap index and standard errors of the WildCLIP model, for each species and behaviour.

**Table S10:** Overlap index and standard errors of the SigLIP model, for each species and behaviour.

**Table S11:** Overlap index and standard errors of the CogVLM model, for each species and behaviour.

**Table S12:** Overlap index and standard errors of the MobileVLM V2 model, for each species and behaviour.

**Table S13:** Overlap index and standard errors of the PaliGemma model, for each species and behaviour.

**Table S14:** Behavioural classification recall, precision and expected calibration error (ECE) of the six studied models, using all events for which the most voted behaviour has at least 50% of the votes.

**Figure S1:** Activity patterns of CLIP, with and without day-night context, using the complete dataset.

**Figure S2:** Activity patterns of SigLIP, with and without day-night context, using the complete dataset.

**Figure S3:** Activity patterns of WildCLIP, with and without day-night context, using the complete dataset.

**Figure S4:** Activity patterns of CogVLM, with and without day-night context, using the complete dataset.

**Figure S5:** Activity patterns of MobileVLM V2 with and without day-night context, using the complete dataset.

**Figure S6:** Activity patterns of PaliGemma, with and without day-night context, using the complete dataset.

**Figure S7:** Average overlap index as a function of the threshold on the confidence score.

**How to cite this article:** Dussert, G., Miele, V., Van Reeth, C., Delestrade, A., Dray, S., & Chamaillé-Jammes, S. (2025). Zero-shot animal behaviour classification with vision-language foundation models. *Methods in Ecology and Evolution, 00*, 1–13. https://doi.org/10.1111/2041-210X.70059