# Background-invariant re-identification of dogs from camera-trap videos in non-controlled environments

Eugenio Dias Ribeiro Neto, Cyril Barrelet, Marc Chaumont, Gérard Subsol, Muhammad Nur Faiz Mahfudz, Muhammd Najib Arung Petana Raja Bone, Barandi Sapta Widartono, Hery Wijayanto, Dyah Ayu Widiasih, Mia Nur Farida, et al.

<figure><img src="image_1.png" /></figure>

## ► To cite this version:

Eugenio Dias Ribeiro Neto, Cyril Barrelet, Marc Chaumont, Gérard Subsol, Muhammad Nur Faiz Mahfudz, et al. Background-invariant re-identification of dogs from camera-trap videos in non-controlled environments. Ecological Informatics, 2025, 93, pp.103547. 10.1016/j.ecoinf.2025.103547. hal-05492220

HAL Id: hal-05492220

https://hal.science/hal-05492220v1

Submitted on 4 Feb 2026

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_2.png" /></figure>
<figure><img src="image_3.png" /></figure>

# Background-invariant re-identification of dogs from camera-trap videos in non-controlled environments

Eugênio Dias Ribeiro Neto a,b,D,*, Cyril Barrelet a,D, Marc Chaumont a,c,D,*, Gérard Subsol a,D,*, Muhammad Nur Faiz Mahfudz d,D, Muhammad Najib Arung Petana Raja Bone d, Barandi Sapta Widartono d,D, Hery Wijayanto d,D, Dyah Ayu Widiasih d,D, Mia Nur Farida d,D, Wayan Tunas Artama d,D, Thibaut Langlois b,D, Hélène Guis b,e,D, Etienne Loire b,D, Michel de Garine-Wichatitsky b,D

## ARTICLE INFO

Dataset link: https://www.lirmm.fr/YT-BB-Dog_Sibetan/, https://github.com/eugeniodias5/BI_FOR/

**Keywords:**

Generalizable re-identification

Stray dogs

BIFOR

YT-BB-Dog dataset

Sibetan dataset

## ABSTRACT

This paper addresses the general problem of re-identification in natural conditions with multiple camera traps, poor video quality and small datasets. We focus on generalizable re-identification of dogs in cross-camera setups, adapting from short-term to long-term scenarios. Long-term re-identification across multiple cameras presents challenges due to variations in background, camera angles, and lighting conditions. While realistic, few animal re-identification methods are tested under such settings, mainly due to the lack of datasets and high complexity of annotation. Short-term datasets are often used to train re-identification networks, since they can be simply generated through web scraping algorithms. We introduce two publicly available datasets: the YT-BB-Dog, a short-term dataset with 2723 dogs from YouTube videos, and the Sibetan, a long-term dataset featuring 59 dogs recorded over 5 days and 12 cameras placed in Sibetan, Bali, Indonesia. Our goal is to use the YT-BB-Dog to train a feature extractor robust to covariate shifts, enabling better generalization in unknown domains. Our experiments revealed that state-of-the-art (SOTA) methods trained on the YT-BB-Dog are heavily influenced by background variations and perform poorly on complex scenarios like Sibetan. To address this, we propose Background Invariant Feature extractOR (BIFOR), a three-step method that leverages a novel mini-batch sampling technique with triplet loss and online hard mining. BIFOR achieves SOTA performance on Sibetan, improving rank-1 accuracy of the baseline by more than 9%. We also present a complete pipeline combining detection, tracking, and re-identification based on BIFOR.

## 1. Introduction

Dogs represent a potential relay host for emerging pathogens from wildlife or vector-borne diseases (Otranto et al., 2017). In rural and natural areas of Southeast Asia, widespread free-roaming domestic or wild dogs are a common occurrence. It is then important to monitor their displacements and assess their contact patterns with the other dogs in inhabited areas (de Garine-Wichatitsky et al., 2024). For this purpose, geolocalized and time-stamped camera traps equipped with movement sensors are dispatched over a study area (Tanwar et al., 2021; Harley and Eyre, 2024). They are equipped with a movement

sensor and automatically acquire, over a period of several days to several weeks, short video clips when an object passes in front of them. Notice that the object may be a car, a motorbike, a person or any domestic or wild animal. Looking at these videos, it becomes possible to reconstitute the trajectory of the individuals dogs and then, their contacts.

These studies require extensive analysis of thousands of videos to track the displacement and interactions of dogs. This cannot be made manually and efficient image processing techniques must be proposed (Simões et al., 2023).

<footnote>ª ICAR, LIRMM, Univ Montpellier, CNRS, Montpellier, France</footnote>

<footnote>ᵇ UMR-ASTRE, CIRAD, INRAE, Montpellier, France</footnote>

<footnote>ᶜ Université de Nîmes, Nîmes, France</footnote>

<footnote>ᵈ One Health Collaborating Center, Faculty of Geography, Faculty of Veterinary Medicine, Graduate School, Universitas Gadjah Mada, Yogyakarta, Indonesia</footnote>

<footnote>ᶜ Epidemiology and Public Health Unit, Institut Pasteur du Cambodge, Phnom Penh, Cambodia</footnote>

<footnote>* Corresponding author at: ICAR, LIRMM, Univ Montpellier, CNRS, Montpellier, France.
E-mail addresses: ediasribeiro@lirmm.fr (E.D.R. Neto), marc.chaumont@lirmm.fr (M. Chaumont), gerard.subsol@lirmm.fr (G. Subsol).</footnote>
<figure><img src="image_4.png" /><figcaption>Fig. 1. A Sequence of Thumbnails of the Same Dog (STSD) extracted from a camera trap video clip.</figcaption></figure>

<figure><img src="image_5.png" /><figcaption>Fig. 2. Cross-camera detection of a dog with a 5 days gap.</figcaption></figure>

First, for each video, all the dogs have to be detected, a thumbnail around each of them is extracted and is tracked all along the video, resulting in a *Sequence of Thumbnails of the Same Dog* (STSD) - we refer STSDs as the set of frames of a dog individual extracted in a same video by a tracking algorithm. Fig. 1 shows an example of extraction of STSD from a video. This first step can be considered as a standard problem: there have been many proposals to localize specific objects (for example using You Only Look Once (YOLO) (Redmon et al., 2016)), and then to track these objects (see a review on Multiple Object Tracking in Ciaparrone et al. (2020)).

Secondly, each dog in a STSD is searched in all the other videos. This process is called “re-identification” (ReID) and has been intensively studied with Deep Learning methods (Wu et al., 2019; Ye et al., 2022).

Notice that, in our case of study, it is performed in a non-controlled environment with a cross-camera setting, which makes the problem very complex. Images may be of poor quality, we can observe significant variations of lighting, points of views, the dogs may be in very different poses (walking, sitting, lying) and backgrounds are heterogeneous between cameras as we can see in Fig. 2. It is also what is called a long-term problem (Xu and Zhu, 2023), which means that the dog appearance can change between two acquisitions: it can wear a collar or nor, be clean, dusty, muddy, wet, or even injured over a period of several days.

To apply Deep Learning algorithms to our study conditions, we must train them on an external database, as we lack prior knowledge about the dogs to be identified (e.g., breed, color, size, number) and we want our model to be independent of domain-specific characteristics

(e.g., background, lighting conditions, number of cameras). We will then use a general database and cannot ensure that the results will generalize well to our practical case. This corresponds to the problem of *generalizable ReID* (Yi et al., 2014), in which we train a model on a source domain and we expect it to be adaptable to unseen domains without needing any domain adaptation or transfer learning.

Due to the scarcity and complexity of annotating large scale long-term ReID datasets, we constructed the *YT-BB-Dog*, a large short-term ReID dataset of videos extracted from Youtube, providing diverse environments and variations in appearances to learn the essential fine-grained features for dog ReID. The main challenge lies in adapting models from short-term domains to unfamiliar long-term domains due to background bias (Lynch et al., 2023; Xiao et al., 2020). To address this, we propose BIFOR (Background Invariant Feature extractOR), a feature extractor trained using triplet loss and a novel mini-batch sampling technique. This approach is designed to mitigate the influence of the background on training phase. We evaluate BIFOR on *Sibetan*, a cross-camera long-term dataset proposed by us, and demonstrate that BIFOR achieves SOTA performances when trained on *YT-BB-Dog* and tested on *Sibetan*.

Our contributions are thus:

* In Section 3.2, the building of the *YT-BB-Dog* dataset, which is derived from the *YouTube-BoundingBoxes* dataset (Real et al., 2017), and contains 2723 dogs. To the best of our knowledge, this is the biggest public dataset for dog ReID.
* In Section 3.3, the creation of the *Siberan* dataset containing 59 dogs. To the best of our knowledge, this is the largest cross-camera dataset of long-term re-identification of dogs in a non-controlled environment.

* In Section 4, the introduction of BIFOR (Background Invariant Feature extractOR), designed to address the domain generalization challenge in ReID by reducing the influence of the background on training phase.

* In Section 5, the analysis of a complete benchmark comparing BIFOR with numerous SOTA algorithms for person and animal re-identification. Our approach achieves the best performances on the *Siberan* dataset.

## 2. Related work

This paper deals with detecting dogs in video clips and then their re-identification (ReID). By ReID, we mean that given a thumbnail or a small sequence of thumbnails of an object or an individual, which is named a probe or a query, an algorithm should retrieve the k most similar thumbnails or sequences of thumbnails which are ranked by decreasing similarity. This ranking is used to calculate the Cumulative Matching Characteristic (CMC) curve (Gray et al., 2007), that is the probability that the query identity is ranked in the top-k retrieved images. Since only the first match counts for the CMC calculation, when there are multiple images of the same object or individual in the ranking, Zheng et al. (2015) recommend the use of the mean Average Precision (mAP) for a better evaluation.

The feature extractor is the most fundamental algorithm block for ReID as it gives a "visual hash" for a thumbnail. Building a discriminative and robust feature extractor is a problem that has been widely studied. Most approaches rely on a specific Deep Learning model and metric learning algorithms like the triplet loss (Zhai et al., 2019), which automatically untangles the features specific to each individual. There are many efficient solutions currently available, based on the triplet loss which have been enriched by different methods as an attention mechanism (DenseIL) (He et al., 2021b), a clustering approach (CTL) (Wieczorek et al., 2021), an unsupervised pre-training approach (Fu et al., 2021), the augmentation (RGT) (Gong and Zeng, 2021), or sub-sections of the image (st-ReID) (Wang et al., 2018).

Person ReID has been intensively studied (Wu et al., 2019; Ye et al., 2022), but these approaches are not directly adaptable to animal ReID. This can be explained by the fact that it requires collecting and annotating a dataset involving many individuals with several images for each, which is difficult when dealing with wild or stray animals (most available datasets contain only 50 to 100 individuals) but also because "the shape of the animal varies significantly when compared to that of humans (...) as well as the angle of capture (...) (some identifying traits may not appear in certain angles)" (Ravoor and Tsb, 2020) or "wildlife data have a wide range of pose variations due to unrestricted four-limbed movement, complex natural backgrounds, and unconstrained lighting conditions" (Li et al., 2020).

Some studies explore the development of general ReID methods that can handle multiple wildlife species (Ravoor and Tsb, 2020; Nepovinnykh et al., 2024). Čermák et al. (2024) present WildlifeDatasets, a toolkit containing various public wildlife re-identification datasets. They introduce MegaDescriptor, a Swin Transformer trained using ArcFace. Jiao et al. (2023) developed UniReID (Universal ReID), a model capable of identifying unseen wildlife animals. They introduce the Wildlife-71 dataset, a dataset of 71 animal categories, each category containing an average of 29 individuals. A problem with such general methods is that they can underperform when compared to specialized approaches, since each species of animal has specific fine-grained features that are important for ReID.

In the field of specialized animal ReID, Li et al. (2020) proposed a full-body tiger re-identification dataset, the ATRW (Amur-Tiger Re-identification in the Wild), that contains a total of 92 tigers. Many

solutions have been proposed to the re-identification of tigers (Liu et al., 2019a,b). Yu et al. (2019) focused on enhancing the performances by computing the shortest path between corresponding local parts. However, some of the most effective solutions require additional metadata during the training phase, such as the pose and orientation of the tigers, which are provided by the ATRW dataset. The cross-camera query evaluation split of the ATRW contains only 28 entities (left and right sides of the same tiger are counted as distinct entities) and 20 tigers.

More specifically, few studies of dog ReID have been carried out. Moreira et al. (2017) worked with high-resolution thumbnails of dog heads (image size were 200 × 200 and 250 × 250) with a front view and an eye alignment, and with a very small dataset of only 21 individuals for the Flickrdog dataset, and 18 individuals for the Snoopybook dataset. A paper devoted to DogFaceNet, Mougeot et al. (2019) also focused on the dog head (face), with only 48 individuals and good quality thumbnails with a front view. Lai et al. (2019) also used faces with an evaluation on the Flickrdog dataset (Moreira et al., 2017). Lai et al. (2019) also tested their proposition on a more coarse-grained problem by evaluating the breed ReID, which is easier than individual ReID. They used the Stanford Dogs dataset (Khosla et al., 2011) (120 different breeds), and Columbia Dogs dataset (Liu et al., 2012) (133 different breeds). Borwarnginn et al. (2021) and Zou et al. (2020) also simplified the problem by considering breeds. Finally, Jang et al. (2020) only focused on the nose for dog ReID. These studies related to dog ReID only used small datasets. They did not address the general dog ReID problem in natural/diverse conditions, sometimes with poor-quality videos. Unlike the majority of existing approaches, we address the full-body ReID in non-controlled environments.

In a recent work, He et al. (2023) propose an approach in the domain of dog re-identification in non-controlled environments. They address the pose variation challenge by introducing the Multipose Feature Fusion Network (MPFNet). The poses are classified into three categories: standing, sitting and lying. MPFNet is a three module network, each dedicated to one pose category, extracting global and local features from images. The outputs of these modules are weighted by the pose discrimination module, a pose classifier trained separately. It was evaluated on MPDD-192, a relatively large dataset of 1657 images and 192 identities. Although its diversity of poses and scenarios, MPDD-192 appears to be a short-term re-identification dataset, as there are not significant variations between two images of the same dog.

To address the Domain Generalization (DG) problem, where a model trained on a source domain should be capable of extending its performances to unknown target domains, we have to go further than standard ReID methods which are very impacted when applied to unseen domains. To tackle this problem, Liao and Shao (2019) treat the ReID as image matching rather than the traditional representation learning, introducing the Query-Adaptive Convolution (QAConv). To exploit the best performances of the Transformers in common image recognition tasks, Liao and Shao (2021) propose TransMatcher, employing hard attention for more efficient image matching. They also propose Graph Sampling (GS) (Liao and Shao, 2022), a mini-batch sampling strategy that exploits the relationships between neighboring classes during each training epoch. Zhang et al. (2023) decompose the feature space into identity and domain-specific features, training them separately to enhance generalization. Ni et al. (2023) propose the PAT (Part-Aware Transformer), aiming to learn representations of different identities that share similar parts.

The discussed DG ReID methods are developed for person ReID. As previously stated, they are generally not directly adaptable to animal ReID. Moreover, none of these approaches address the short to long-term adaptation, being typically trained and tested on short-term domains with a limited number of cameras, not considering scenarios involving clothes change.
<figure><img src="image_6.png" /><figcaption>Fig. 3. Detection of two dogs with YoloV5 in a frame of the Tibetan dataset.</figcaption></figure>

## 3. Datasets

### 3.1. Extracting STSDs from a video clip

The first step to compose a STSD involved localizing the dogs in each frame, and then tracking them. For object detection, we used YOLOv5¹ (Redmon et al., 2016) pre-trained on the COCO dataset (Lin et al., 2014). YOLO is a family of architectures and models for object detection, and COCO is a generalist dataset that contains many types of objects, including the dog class. Fig. 3 illustrates the kind of frames derived from the Tibetan videos dataset as well as the good detection obtained with a YoloV5 that has not been retrained on the Tibetan videos dataset.

For object tracking, we used StrongSORT (Du et al., 2023), which is an improved version of DeepSORT (Wojke et al., 2017). StrongSORT has two parts: a position predictor and an appearance comparator. When YOLO detects objects, StrongSORT stores them and estimates their next positions. StrongSORT then associates boxes between consecutive frames by comparing current detected positions with the predicted position while also comparing appearances. We used the GitHub<fnref n="2" /> repository Yolov5 + StrongSORT with OSNet, that integrate YoloV5, and StrongSORT using an appearance descriptor given by the OSNet model (Zhou et al., 2019). Note that the appearance descriptor (from OSNet) is trained for person re-identification, therefore it is not expected to work well on dogs. We thus decided to lower the appearance cost term.

### 3.2. Construction of the YT-BB-Dog dataset

To train a network to the general re-identification problem, we require a dataset containing a sufficient number of different dog individuals and, for each individual, a sufficient number of different thumbnails. The Stanford Dogs (Khosla et al., 2011), Columbia Dogs (Liu et al., 2012), or Tsinghua Dogs (Zou et al., 2020) datasets are dedicated to dog breed identification. Each breed contains dozens of images of different dogs but not several images of the same individual. Conversely, Flickr-dog and Snoopybook image datasets (Moreira et al., 2017) contain a few thumbnails per dog individual, but they focused only on the head. These datasets cannot be used for re-identification where the full-body of dogs is acquired, in any position.

The MPDD-192 (He et al., 2023) is one of the first datasets of full-body dog re-identification, comprising 1657 images and 192 identities.

Leveraging from web-scraping techniques, we aim to create a larger dataset. One idea is to use videos of dogs to access several thumbnails of the same dog individual. We thus looked at video datasets and found that the Youtube-BoundingBoxes (YT-BB) dataset (Real et al., 2017) offered indirect access to a dog dataset useful for the re-identification training.

The YT-BB dataset is a human-annotated dataset with 380,000 video clips of 15-20s extracted from 240,000 public videos from YouTube, and containing 23 object classes, including dogs, that form a subset of the classes of the COCO dataset. This dataset is given with the bounding boxes of the various present objects.

We select all the video clips of the class ‘dog’ and automatically retrieve the Sequences of Thumbnails of the Same Dog (STSD) using the bounding box annotations for each frame. We remove those whose image resolution is under 50 × 50 or with less than 5 thumbnails. This led us to 9411 STSDs, which corresponds to different dog-individuals, for a total of 89,772 images.

We then noticed that about 4/5 of the STSDs was of poor quality or showed irrelevant parts of a dog, thus it was impossible to identify any individual. We thus decided to filter the dataset by keeping the STSDs whose thumbnails own at least 10 keypoints extracted by a pose estimator. We used MMPose toolbox (Chen et al., 2019) for the pose estimation and selected a ResNet152 model (He et al., 2016) trained on the AnimalPose dataset (Cao et al., 2019). This model was chosen for its good performance on animal pose estimation, without any specific preference. Keypoints were considered as valid if their threshold probability is greater than 40%. After running this filtering, we went from 9411 to 2723 dogs and 27,036 images, resulting in the YT-BB-Dog dataset.

This dataset contains a wide variety of images since videos were posted on YouTube by different people worldwide, using various types of cameras, under various acquisition conditions (angle, luminosity, background, inside/outside, etc.), and with various types of dogs (see Fig. 4). The YT-BB-Dog represents a very diverse short-term re-identification dataset, making it possible for our feature extractor to learn the important fine-grained features of dogs.

### 3.3. The Tibetan dataset

The Tibetan dataset is derived from a conducted data collection of ANR-SEAdogSEA<fnref n="3" /> project (de Garine-Wichatitsky et al., 2024) (Socio-Ecological Approach of Dog-borne diseases in South-East Asia), coordinated by CIRAD/UMR ASTRE. In total we disposed of 11,278 video clips, each was 5 s long with a 720 × 480 pixel size. The video clips were taken by 14 camera traps (although we only detected dogs in 12 cameras), placed at variable heights depending on the field, but usually around 1 m from the ground. The recording was done over 5 days of September 2019. Meta-data such as the temperature, the date, the hour, and the coordinates are also available but were not used here. As shown in Fig. 3, these videos are mid-quality with many visible compression artifacts.

We only considered sequences that had only one dog identified in order to build all the STSDs. Thus, we did not consider sequences of frames that included more than one dog, avoiding wrong associations and multiple threads for the same dog. We indeed observed in some videos that many threads were created for the same dog when there were multiple dogs. Additionally, we ignored video clips recorded at night.

After having detected, tracked (YoloV5 + StrongSORT with OSNet), kept only STSDs consisting of 5 or more thumbnails, and suppressed sequences with less than 10 keypoints obtained with a pose estimation (MMPose toolbox), we obtained 228 STSDs for a total of 1788 thumbnails. In order to evaluate our model’s performance on such a

<footnote n="1">¹ https://github.com/ultralytics/yolov5</footnote>

<footnote n="2">² https://github.com/mikel-brostrom/Yolov5_StrongSORT_OSNet</footnote>

<footnote n="3">³ https://anr.fr/Projet-ANR-19-ASIE-0002</footnote>
<figure><img src="image_7.png" /><figcaption>Fig. 4. Illustration of the diversity of images in the YT-BB-Dog dataset.</figcaption></figure>

complex scenario, the dataset was annotated and verified in collaboration with veterinarians of the CIRAD/UMR ASTRE. After eliminating false positives detections and excluding dogs that were too difficult to annotate, the result is a total of 59 dogs, 223 STSDs and 1755 images across 12 cameras, constituting the *Sibetan* dataset. Fig. 2 illustrates an example of a dog detected in different cameras with a difference of 5 days. To the best of our knowledge, Sibetan is the largest long-term dog re-identification dataset in a cross-camera scenario.

Table 1 gives a summary of our two datasets. We reinforce that both datasets are composed only of dog “crops” extracted from raw camera-trap videos, and no sensitive data (e.g., images of people or license plates) are published. The YT-BB dataset is licensed by Google under a Creative Commons Attribution 4.0 International License,<fnref n="4" /> and the Sibetan dog images are used with the owners’ permission. Consequently, no ethical concerns arise from their use.

# 4. Method

## 4.1. Triplet loss principle

Most modern approaches for ReID rely on the use of the triplet loss concept introduced by Schroff et al. (2015). The idea is to build a

<table><caption>**Table 1**
Comparison of our datasets.</caption><thead><tr><th></th><th>YT-BB-Dog</th><th>Sibetan</th></tr></thead><tbody><tr><td>Images</td><td>27,036</td><td>1755</td></tr><tr><td>STSDs</td><td>2723</td><td>223</td></tr><tr><td>Ids</td><td>2723</td><td>59</td></tr><tr><td>Days</td><td>–</td><td>5</td></tr><tr><td>Cameras</td><td>–</td><td>12</td></tr><tr><td>Source</td><td>Youtube videos</td><td>Sibetan, Bali, Indonesia</td></tr></tbody></table>

feature extractor that discriminates different individuals by increasing the distance between their feature vectors. The training consists of reasoning on three feature vectors extracted from three thumbnails, hence named triplet loss. Those three thumbnails (in our case, each of them showing an individual dog) are named the anchor, i.e. the thumbnail showing an individual dog, the positive, i.e. another thumbnail that shows the same individual, and the negative, i.e. a thumbnail showing another individual. The three corresponding feature vectors *a*, *p* and *n* define the following loss function:

$$L_{a,p,n} = \max(0, dist(a, p) - dist(a, n) + m), \quad (1)$$

where *dist* is a distance function, *m* is a scalar defining the margin, and *max* is the maximum function. By minimizing the sum of the triplet loss for all the individuals, we force the positive samples to be closer to the

<footnote n="4">⁴ https://creativecommons.org/licenses/by/4.0/</footnote>
<figure><img src="image_8.png" /><figcaption>Fig. 5. Dogs 2269 and 2652 in the test set of the YT-BB-Dog. Most part of the dogs of the dataset are like the 2652, with low background variation between thumbnails.</figcaption></figure>

anchors, while forcing the negatives samples to be more distant than a given margin.

To ensure efficient learning, we must carefully select both positive and negative samples. Indeed, selecting a triplet whose 3 vectors are already correctly distant in the feature space results in a null triplet loss, and is thus unnecessary for learning. Usually, we select the anchor and the positive to be very close (in our case, anchor and positive are two thumbnails of the same STSD), and focus on the research of semi-hard or hard negative samples (Schroff et al., 2015). A semi-hard or hard negative sample corresponds to thumbnails whose distance to the anchor is either within a defined margin or very close to it.

## 4.2. The selection of triplets

As discussed by Zhai et al. (2019), a good choice of triplets is very important in the domain of re-identification. To enhance the choice of triplets, they propose the formation of mini-batches by randomly sampling *P* individuals and *K* thumbnails of each individual, resulting in a mini-batch of *PK* images. Given an anchor, the set of positives and negatives are then defined by:

$$P = \{ \mathbf{p} \in \text{batch} | \text{id}(\mathbf{p}) = \text{id}(\mathbf{a}), \mathbf{p} \neq \mathbf{a} \}, \quad (2)$$

$$\mathcal{N} = \{ \mathbf{n} \in \text{batch} | \text{id}(\mathbf{n}) \neq \text{id}(\mathbf{a}) \}, \quad (3)$$

with *id(.)* being the identity of the dog denoted by the input vector.

Within mini-batches, online hard mining (Radenović et al., 2016; Shrivastava et al., 2016) is applied as a training strategy to select the most challenging examples. It involves, at each epoch and for each anchor vector *a*, selecting the positive sample with the highest distance and the negative sample with the lowest distance:

$$\mathbf{p}_{\text{hard}} = \arg \max_{\mathbf{p} \in \mathcal{P}} \text{dist}(\mathbf{a}, \mathbf{p}), \quad (4)$$

$$\mathbf{n}_{\text{hard}} = \arg \min_{\mathbf{n} \in \mathcal{N}} \text{dist}(\mathbf{a}, \mathbf{n}). \quad (5)$$

## 4.3. The influence of the background

The influence of non-causal features, that are not directly correlated with the output prediction but learned by models due to their presence in the training data, is a widely studied problem. Previous works have highlighted the influence of the background in image recognition tasks like classification (Xiao et al., 2020; Lynch et al., 2023). Tian et al. (2018) demonstrated the background bias in person ReID, showing that this problem is hardly observed when training and testing are conducted on datasets that were captured by a small number of cameras of nearly homogeneous backgrounds. However, the performances significantly deteriorate if models were then tested on datasets with strong background variability.

As discussed in Section 3.2, the YT-BB-Dog dataset is composed of dog thumbnails extracted from Youtube videos, where the background variability between individuals is very high. While there is a significant variability between two different individuals, this does not extend to the frames of the same dog. Often, the dataset includes dogs like the dog 2652 in Fig. 5, which remains relatively static, in this case, in a pool with minimal movement throughout the video. This is not always true. In the same figure we note that the dog 2269 moves a lot, transitioning from a red background to the grass and back to the same red background. For our goal, the cross-camera environment such as the Tibetan dataset, it is preferable for our model to perform well on dogs like the 2269, because it represents the background variations that we can expect in such contexts.

We can then distinguish between two types of training samples in the YT-BB-Dog dataset: some dog individuals have samples with nearly homogeneous backgrounds, which are the most common. On the other hand, there are also individuals that exhibit a background variability.

To illustrate the influence of background, we performed a ranking experiment based on MPFNet, a dog ReID method, trained on YT-BB-Dog. In a ranking experiment (see Fig. 6), we use MPFNet to extract features of all the images in the dataset, given a query image (shown on the left), we compute the feature distances of this image to all other images in the gallery (dataset). The retrieved images are then displayed from left to right in decreasing order of similarity. A clear influence of the background can be observed in Fig. 6. While the results are perfect for dog 2652, when using an image of dog 2269 on the grass as query, in the top-2 rank we retrieve only images of the same dog on the grass. In the top-6, no image of dog 2269 on the red background can be found (Fig. 5). The remaining images in the ranking are of dogs that are not very similar to the query but share similar background (grass). These results emphasize that re-identification algorithms, which are generally based on diverse metric learning techniques, tend to heavily encode background information into the feature vectors of the dogs. In datasets like the YT-BB-Dog, where most positive samples lack significant background variability, the background can, indeed, become a clue for a network to improve its performances. But the high performances in this case are biased, since the models are in fact not well representing what a dog and its visual aspect are. This leads to a reduction of generalization property when performing on an unseen domain.

## 4.4. BIFOR: A three-step method to mitigate the influence of the background

To generalize from the YT-BB-Dog to non-controlled, cross-camera environments, we need to define a feature extractor that is robust to background variations. Rather than relying on random identity sampling, the idea is to select triplets that discourage the model from relying on the background cues and guide toward a latent space representation focused on the dogs.
<figure><img src="image_9.png" /><figcaption>Fig. 6. Ranking of MPFNET on the individuals 2652 and 2269, showing a tendency to rank images based on their backgrounds.</figcaption></figure>

<figure><img src="image_10.png" /><figcaption>Fig. 7. The training sequence. This scheme shows the two consecutive updates of the feature extractor $f^{(0)}$ (leading to $f^{(1)}$, and then $f^{(2)}$).</figcaption></figure>

For this, we propose a three-step method BIFOR outlined in Fig. 7. It is based on the general pre-trained network $f^{(0)}$ which is, in our experiments, a ConvNext-small (Liu et al., 2022), pre-trained on ImageNet-1k.

**Step 1: defining a feature extractor based on the background.** Yan et al. (2022) demonstrated that the triplet loss is not effective on properly penalizing the network for missing fine-grained differences between two individuals, instead focusing on high-level differences. In this first step, we employ an auxiliary network dedicated to a task which is a proxy of the final dog re-identification. This network is used to select the hard negatives of our first optimization. Therefore, setting a high triplet loss margin during this phase, we ensure that the loss remains elevated, penalizing high-level differences between individuals. As the triplets selected by the auxiliary network are all similar and given the characteristics of the YT-BB-Dog dataset, large differences can be seen in the background.

<figure><img src="image_11.png" /><figcaption>Fig. 8. Illustration of the thumbnails most similar to a query thumbnail using the feature extractor $f^{(1)}$.</figcaption></figure>

We use as auxiliary network an EfficientNet (Tan and Le, 2019) that learned to distinguish dog breed via preliminary supervised learning on the Stanford Dogs dataset (Khosla et al., 2011).

After performing the first triplet optimization, the updated feature extractor $f^{(1)}$ tends to focus less on the object and more on the background. This behavior is illustrated in Fig. 8. On the left, we show that when using a dog on grass as the query, $f^{(1)}$ considers other dogs also on grass as being similar.

**Step 2: sampling mini-batches to mitigate the influence of the background.** The idea is now to train $f^{(0)}$ with mini-batches where the images of different individuals present a similar background. Let $\mathcal{X} = \{x_1, x_2, ..., x_N\}$ be the set of $N$ images, we use the feature extractor $f^{(1)}$ to extract the feature set $\mathcal{V} = \{v_1, v_2, ..., v_N\}$ described as $v_i = f^{(1)}(x_i)$. Given $\mathcal{D}$ unique identities, we represent each identity by a single vector. Thus, we define the set $\tilde{\mathcal{V}} = \{\bar{v}_1, \bar{v}_2, ..., \bar{v}_D\}$, where $\bar{v}_i$ is the mean of features with identity $i$ defined by:

$$ \bar{v}_i = \frac{\sum_{\{k|id(k)=i\}} \mathcal{V}_k}{|\{x \in \mathcal{X} | \text{id}(x) = i\}|} \qquad (6) $$

Following the PK sampling (Zhai et al., 2019), each mini-batch contains $P \times K$ images, with $P$ identities and $K$ images per identity. Grouping the images respecting a given number of individuals per mini-batch is not a straightforward optimization problem. We solve it by a constrained clustering algorithm, the constrained k-means (Bradley et al., 2000) algorithm.<fnref n="5" /> The key change of the algorithm compared to k-means is in the assignment step of the Expectation-Maximization (EM) algorithm, that is reformulated as a minimum cost flow optimization problem, allowing to specify both a minimum and a maximum number of elements per cluster. Setting both minimum and maximum bounds to $P$, the $D$ individuals are gathered into $\frac{D}{P}$ clusters.

Nevertheless, the number of images available per identity in the dataset is not equal. In order to maximize the usage of the available images and, consequently, the number of mini-batches, we do an artificial re-balance, assuring that each identity has the same number of images. Thus, we define an augmentation factor $r = \lceil \frac{N}{D \cdot K} \rceil$. In effect, $r$ is the

<footnote n="5">⁵ https://pypi.org/project/k-means-constrained/</footnote>
<figure><img src="image_12.png" /><figcaption>Fig. 9. Three rows of different mini-batches and the respective identities. The individuals inside a mini-batch share similar backgrounds.</figcaption></figure>

<figure><img src="image_13.png" /><figcaption>Fig. 10. Ranking of BIFOR on the individuals 2652 and 2269, showing a lower influence of the background on the results when compared to MPFNet.</figcaption></figure>

rounded-up average number of mini-batches per identity that we can form. Every identity is augmented (through replication or sampling) to have exactly $r \times K$ images, ensuring a balanced training on which each cluster can form $r$ mini-batches. As a result, we compose a total of $\frac{D}{P} \times r$ mini-batches.

As $f^{(1)}$ is heavily focused on the background, we hypothesize that individuals with close mean features may share a similar background. In Fig. 9 we show an example of mini-batches that we have formed setting $P = 4$. Notice that, as we were expecting, all individuals within a mini-batch share similar backgrounds.

**Step 3: Obtaining a Background Invariant Feature extractOR.**
After forming the mini-batches, we run the second triplet optimization of $(f^{(0)}$, applying online hard mining (refer to Section 4.2). We obtain our final feature extractor $f^{(2)}$, which we call BIFOR (Background Invariant Feature extractOR).

To demonstrate the reduction of the influence of the background, we compare again the closest images to the dogs 2652 and 2269 (refer to Fig. 5), but this time extracting the features using BIFOR trained on YT-BB-Dog. Fig. 10 shows a good result on dog 2652, but the results are even more interesting for the dog 2269, since now we are able to find all the correct images in the top 6, including the images on the red background. It is also noticeable that the dog found in top-4 and top-5 is very similar to the query dog.

Note that in the first step we used dog breed classification as a proxy task. However, depending on the final objective, alternative proxy tasks can be selected, allowing our method to be applied to different tasks or to mitigate the influence of different non-causal features.

In the next section, we present the implementation details, a benchmark evaluation comparing with some SOTA across different object re-identification methods on the YT-BB-Dog and Sibetan datasets, along with an ablation study.

# 5. Experiments, results and discussions

## 5.1. Evaluation metrics and datasets

As stated in Section 3, in this paper we utilize the YT-BB-Dog and the Sibetan datasets, both developed by us. The YT-BB-Dog is used for both training and evaluation, it contains 27,036 images of 2723 dogs, which we split into 2000 dogs for training and 723 dogs for evaluation. The Sibetan dataset is only used for evaluation and contains a total of 59 dogs, 223 STSDs and 1755 images. We thus exclude dogs that appear in only one STSD as we do not cover the open-set scenario (Liao et al., 2014), so the probe STSD must have corresponding STSDs in the gallery to find a match. We used 39 dogs, comprising 203 STSDs and a total of 1603 images.

For evaluation, we use classical person re-identification metrics such as the CMC rank-k and mean Average Precision (mAP) (Zheng et al., 2015). For a given query image $X_q$ of identity $d$, we first order a ranking $R_q = \{X_1, X_2, \dots, X_N\}$ of the most similar images to $X_q$. We assume that $D$ images inside $R_q$ have the same identity as $X_q$. Given a delta function $\delta(i)$:

$$ \delta(i) = \begin{cases} 1 & \text{if } id(X_i) = d, \\ 0 & \text{otherwise} \end{cases} \qquad (7) $$

The accuracy of identifying at least one correct match within the top-k images in $R_q$ is:

$$ Acc_k = \begin{cases} 1 & \text{if } \delta(i) = 1 \text{ for any } i \le k, \\ 0 & \text{otherwise} \end{cases} \qquad (8) $$

The rank-k is calculated as the mean $Acc_k$ across all queries.

The AP (average precision) represents the area under the Precision-Recall curve, which for the query $X_q$ is approximately given by $\sum_{k=1}^{N} p_q(k) \Delta r_q(k)$. Here, $p_q(k)$ denotes the precision, which is the ratio
of true positives (i.e., images that share the same ID as $X_q$) among the top k images in $R_q$. The $\Delta r_q(k)$ is the variation of recall defined as:

$$ \Delta r_q(k) = \begin{cases} \frac{1}{D} & \text{if } \delta_q(k) = 1, \\ 0 & \text{otherwise} \end{cases} \qquad (9) $$

Therefore, the AP can be written as:

$$ AP_q = \sum_{k=1}^{N} \frac{p_q(k)}{D} \delta_q(k) \qquad (10) $$

The mAP is computed as the mean value of the AP across all the queries.

It is important to notice that in the YT-BB-Dog the sequences of images of a specific dog come from the same video, so we evaluate our performances by comparing images individually. Given a probe image, we consider as gallery all the remaining images. In the Tibetan dataset we compare the sequences of images (STSDs) that are originated from different videos. We represent a STSD by calculating the mean of the features of its thumbnails. For a query STSD, we compare it with the 202 remaining STSDs.

QAConv and TransMatcher (Liao and Shao, 2019, 2021) scores are given by a matcher. That means, instead of extracting the embeddings for each image and calculating the similarity, they directly output a score of similarity between two images given as entry. For these methods specifically, we compute the scores between each thumbnail and then calculate the mean score between STSDs.

## 5.2. Implementation details

Starting by the proxy task of dog breed classification, it is performed using an EfficientNet trained on the Stanford dogs dataset, which contains 20,580 images and 120 dog breeds. We randomly split 70% of the images for training, 30% for test, and train the network using a cross-entropy loss with a learning rate of $1 \times 10^{-3}$, reaching 85% test accuracy. This network is used to form triplets for the step 1 of our three-step method.

To obtain $f^{(1)}$ and our final model $f^{(2)}$, we use Convnext-Small (Liu et al., 2022) as the backbone, which is an updated version of ResNet incorporating design elements of the Swin transformer (Liu et al., 2021). At a reasonable amount of parameters, convnext achieves good results across diverse image recognition tasks, making it a suitable choice for us.

The images are resized to $224 \times 224$ and we apply image normalization with mean [0.485, 0.456, 0.406] and standard deviation [0.229, 0.224, 0.225]. We use Adam as optimizer with a maximum learning rate of $1 \times 10^{-4}$, a one cycle learning rate scheduler with cosine annealing and a warm-up of 10%. As loss function, we employ the triplet loss with online hard triplet mining, using cosine similarity as similarity metric and a margin of 0.2. We select the best model based on the highest mAP on the YT-BB-Dog dataset. We run a training of 200 epochs, although our model converges quickly.

The main difference between the training of $f^{(1)}$ and $f^{(2)}$ lies in the triplet loss: to obtain $f^{(1)}$, we use offline triplet mining with cosine distance and, as described in Section 4.4, we adopt a large margin of 1, a considerable value given that the cosine distance ranges from 0 to 2.

After a tuning of hyper-parameters, we observed better results on Tibetan (higher mAP) when training $f^{(2)}$ using a batch size of 32 and 8 images per individual. This is likely due to having fewer individuals per mini-batch, which allows us to cluster more similar backgrounds within a mini-batch.

Our trainings were conducted on a NVIDIA RTX A6000.

<table><caption>**Table 2**
Comparison of performances on the YT-BB-Dog dataset (original) and after replacing the test set backgrounds with random images.</caption><thead><tr><th rowspan="3">Method</th><th colspan="9">YT-BB-Dog</th></tr><tr><th colspan="4">Original</th><th colspan="4">Random Background</th></tr><tr><th>R-1</th><th>R-5</th><th>R-10</th><th>mAP</th><th>R-1</th><th>R-5</th><th>R-10</th><th>mAP</th></tr></thead><tbody><tr><td>MPFNet</td><td>99.1</td><td>99.6</td><td>99.8</td><td>93</td><td>30.2</td><td>47.7</td><td>56.1</td><td>16.0</td></tr><tr><td>NWPU_ASGO</td><td>98.5</td><td>99.4</td><td>99.6</td><td>90.4</td><td>29.1</td><td>47.0</td><td>55.4</td><td>15.0</td></tr><tr><td>MegaDescriptor-B</td><td>97.8</td><td>99.0</td><td>99.3</td><td>87.6</td><td>30.2</td><td>46.0</td><td>53.8</td><td>14.4</td></tr><tr><td>LMBN</td><td>99.3</td><td>99.6</td><td>99.8</td><td>93.7</td><td>32.4</td><td>49.5</td><td>58.6</td><td>18.1</td></tr><tr><td>TransReID<sub>x=12</sub></td><td><b>99.5</b></td><td><b>99.8</b></td><td><b>99.9</b></td><td><b>96.1</b></td><td>35.5</td><td>54.5</td><td>64.2</td><td>21.3</td></tr><tr><td>QAConv<sub>x=152</sub>-GS</td><td>99.2</td><td>99.6</td><td>99.7</td><td>92.4</td><td>35.6</td><td>51.9</td><td>60.9</td><td>23.6</td></tr><tr><td>TransMatcher-GS</td><td>99.3</td><td>99.7</td><td>99.8</td><td>94.9</td><td>34.4</td><td>50.7</td><td>60.1</td><td>21.8</td></tr><tr><td>PAT</td><td>99.3</td><td>99.7</td><td><b>99.9</b></td><td>94.7</td><td>41.9</td><td>61.4</td><td>70.3</td><td>27.4</td></tr><tr><td>BIFOR (Ours)</td><td>98.9</td><td>99.5</td><td>99.6</td><td>91.3</td><td><b>62.6</b></td><td><b>80.1</b></td><td><b>86.2</b></td><td><b>44.8</b></td></tr></tbody></table>

## 5.3. Comparison with state-of-the-art algorithms

We compare our method with existing SOTA approaches. For easier reproducibility and fair comparison, we focus on methods that have public codes, we resize images to $224 \times 224$ while preserving the originally optimized hyperparameters. To demonstrate the effectiveness of our technique, we compare it with methods developed for different categories, like Megadecipher (Cermák et al., 2024) for animal ReID. NWPU_ASGO (Yu et al., 2019) for ATRW ReID (Li et al., 2020). MPFNet (He et al., 2023) for dog ReID. LMBN (Herzog et al., 2021), and TransReID (He et al., 2021a) for person ReID. QAConv, Transecnatcher (Liao and Shao, 2019, 2021), and PATransformer (Ni et al., 2023) for generalizable person ReID.

We do not implement the ensemble strategy proposed by Yu et al. (2019). Instead, we use a DenseNet-169, which is the model with best reported mAP in the paper.

Note that in all the tables presented in this section, R-1, R-2, R-5 and R-10 correspond to Rank-1, 2, 5 and 10, respectively.

On the left side of Table 2, we present the results on the original YT-BB-Dog dataset test set. The person re-identification methods are the ones that achieve highest mAP scores, specially TransReID (He et al., 2021a), achieving top-1 performance in all the tested metrics.

To demonstrate that, despite their strong performances, SOTA methods are influenced by the background, we modified the backgrounds of the test set. For each image, we applied dichotomous object segmentation using BiRefNet (Zheng et al., 2024), then randomly replaced the background with one of 100 pre-selected images. Some poorly segmented images were manually removed, reducing the test set from 7104 to 7064 images. The right side of Table 2 reveals a significant performance decline of all methods compared to the original test set. However, it is important to notice that BIFOR shows the best performances under these conditions.

We test the trained models on the Tibetan dataset. Table 3 shows that our method achieves SOTA results, surpassing 82% of rank-1 and achieving almost 70% of mAP. Compared to PAT (Ni et al., 2023), the second best method, we achieve 7.3% higher rank-1 and 4.5% higher mAP.

To better illustrate why BIFOR achieves lower scores on YT-BB-Dog but outperforms SOTA approaches under change of domain, Fig. 11 presents a ranking comparison on the Tibetan dataset between BIFOR and TransReID, the method that shows the best performances on YT-BB-Dog. In this example, among all the STSDs of dog 33, the query STSD 221 is the only one captured on grass. BIFOR correctly retrieves the same individual within the first two ranks, whereas TransReID retrieves different dogs that share the same grass background, notably the rank-1 (STSD 218) which is visually different but captured in the exact same location as the query. As discussed in Section 4.3, SOTA approaches tend to rely on background cues when trained with classical random
<figure><img src="image_14.png" /><figcaption>Fig. 11. Ranking comparison between BIFOR and TransReID on the dog 33 of the Sibetan dataset. The query (STSD 221) is the only image of the dog on the grass. BIFOR correctly retrieves the target individual within the first two ranks, while TransReID retrieves images of different dogs on the grass.</figcaption></figure>

<figure><img src="image_15.png" /><figcaption>Fig. 12. Illustration of our multi-step pipeline on the Sibetan dataset.</figcaption></figure>

<table><caption>**Table 3**

Comparison of performances on the Sibetan dataset.</caption><thead><tr><th rowspan="2">Method</th><th colspan="5">YT-BB-Dog → Sibetan</th></tr><tr><th>R-1</th><th>R-2</th><th>R-5</th><th>R-10</th><th>mAP</th></tr></thead><tbody><tr><td>MPFNet</td><td>62.6</td><td>73.4</td><td>87.2</td><td>93.1</td><td>50.1</td></tr><tr><td>NWPU_ASGO</td><td>61.6</td><td>75.9</td><td>86.7</td><td>92.6</td><td>49.8</td></tr><tr><td>MegaDescriptor-B</td><td>63</td><td>79.8</td><td>89.6</td><td>95.5</td><td>55.7</td></tr><tr><td>LMBN</td><td>64.5</td><td>76.8</td><td>90.1</td><td>95.6</td><td>55.1</td></tr><tr><td>TransReID<sub>x=12</sub></td><td>69.4</td><td>83.7</td><td>91.1</td><td>96</td><td>61.1</td></tr><tr><td>QAConv<sub>152</sub>-GS</td><td>63.5</td><td>79.8</td><td>91.6</td><td>94.6</td><td>53.4</td></tr><tr><td>TransMatcher-GS</td><td>64.5</td><td>77.3</td><td>90.1</td><td>96.1</td><td>55</td></tr><tr><td>PAT</td><td>75.4</td><td>86.7</td><td>90.6</td><td>94.6</td><td>65.3</td></tr><tr><td><strong>BIFOR (Ours)</strong></td><td><strong>82.7</strong></td><td><strong>88.2</strong></td><td><strong>93.6</strong></td><td><strong>96.1</strong></td><td><strong>69.8</strong></td></tr></tbody></table>

identity sampling, which explains their superior performance on short-term datasets such as YT-BB-Dog (which show a lower background variability), but also their limited capacity to generalize.

By guiding our model into focusing more on the individual than in the background, we successfully achieve the best performances on Sibetan. This demonstrates the ability of our model to adapt from short-term to long-term scenarios, and a high capacity to generalize to unknown domains.

## 5.4. Ablation studies

Table 4 presents the impact of each step of our sampling method (refer to Section 4.4). Starting from the ImageNet pretrained ConvNeXt-Small ($f^{(0)}$, line 1), fine-tuning on YT-BB-Dog under the same hyperparameters as BIFOR but using random identity sampling, as in SOTA methods, results in a strong improvement of 7.2% mAP. By applying the step 1 of our method (sampling based on features from

<table><caption>**Table 4**

Ablation studies on the Sibetan dataset. We test the influence of each step of our method.</caption><thead><tr><th rowspan="2">Model</th><th colspan="5">Sibetan</th></tr><tr><th>YT-BB-Dog</th><th>Step 1</th><th>Steps 2-3</th><th>R-1</th><th>R-2</th><th>R-5</th><th>mAP</th></tr></thead><tbody><tr><td>$f^{(0)}$</td><td></td><td></td><td></td><td>76.8</td><td>83.7</td><td>90.6</td><td>57.1</td></tr><tr><td>$f^{(0)}$</td><td>✓</td><td></td><td></td><td>73.4</td><td>85.2</td><td>92.1</td><td>64.3</td></tr><tr><td>$f^{(1)}$</td><td>✓</td><td>✓</td><td></td><td>24.1</td><td>36.4</td><td>57.1</td><td>21.8</td></tr><tr><td>$f^{(2)}$ (BIFOR)</td><td>✓</td><td>✓</td><td>✓</td><td><strong>82.7</strong></td><td><strong>88.2</strong></td><td><strong>93.6</strong></td><td><strong>69.8</strong></td></tr></tbody></table>

an EfficientNet trained on a proxy task) the intermediate network $f^{(1)}$ (line 3) performs poorly, as expected, since it is intentionally designed to retrieve images with similar backgrounds. Finally, BIFOR, which combines the clustering-based batch formation using $f^{(1)}$ (step 2) achieves the best overall results, improving rank-1 by over 9% and mAP by 5.5% compared to the baseline (line 2).

## 6. Conclusion

In this paper, we present a way to re-identify dogs when the image contents are diverse and of poor quality. Our approach differs from most part of other works on animal re-identification, as we use bigger and more diverse datasets, also by the fact that we used the whole dog's appearance and not only its face. In this context, we present two datasets dedicated to dog re-identification: the YT-BB-Dog and the Sibetan datasets. The first one includes a high number of very diverse dogs, we use it to train our robust feature extractor. The second one, which we annotated with the aid of specialists, to evaluate our method in a scenario that is very close to most part of real-world applications.
Our algorithmic proposition involved building a feature extractor able to discriminate between various similar individuals in a covariate shift scenario. We used a triplet loss and proposed a clever way to automatically define hard examples. Through two-step training, the feature extractor focuses on the particularities of each dog. We were able to demonstrate that our method is less influenced by the background content than the SOTA re-identification methods, being more prepared to be applied in unknown domains. Note that while we apply our mini-batches formation strategy to the task of dog re-identification, it could be applied to other re-identification tasks.

We therefore demonstrate the feasibility of a dog re-identification pipeline, composed of a detection and tracking step using off-the-shelf YOLOv5 and StrongSORT without any retraining, a filtering step that employs a pretrained animal pose-estimation model to filter occluded detections, and re-identification using BIFOR. We exemplify the feasibility of the pipeline, described in Fig. 12, on the Tibetan dataset. Re-identification relies solely on features learned from the YT-BB-Dog dataset, and our results are very promising, achieving 82.7% of rank-1 accuracy and almost 70% of mAP, overcoming all tested methods.

There are many potential ways to improve the results, first, the background is not the only non-causal feature that influences the re-identification. Other non-causal features like the illumination, pose, and camera perspective could be explored. We could also use a combination of loss metrics instead of relying solely on the triplet loss, experimenting if our mini-batches formation is capable of improving the performances of more complex metric learning techniques. In further studies, we could investigate the impact of replacing the proxy task of dog breed classification by more downstream tasks such as pose estimation, semantic segmentation, etc. Another idea would be to use poses to improve the feature extractor, leveraging the most important keypoints for the re-identification of dogs. Other ideas are related to the use of unsupervised domain adaptation to adapt the feature extractor to manage the covariate shift on datasets such as Tibetan, and use metadata like the geographical displacements of dogs between cameras, etc.

## CRediT authorship contribution statement

**Eugênio Dias Ribeiro Neto:** Writing – review & editing, Writing – original draft, Validation, Supervision, Software, Methodology, Investigation, Formal analysis, Data curation.
**Cyril Barrelet:** Writing – review & editing, Writing – original draft, Validation, Methodology, Investigation, Formal analysis.
**Marc Chaumont:** Writing – review & editing, Writing – original draft, Validation, Supervision, Resources, Project administration, Methodology, Investigation, Funding acquisition, Formal analysis, Conceptualization.
**Gérard Subsol:** Writing – review & editing, Writing – original draft, Supervision, Project administration, Methodology, Investigation, Funding acquisition, Formal analysis, Conceptualization.
**Muhammad Nur Faiz Mahfudz:** Validation, Data acquisition, Data curation.
**Muhammad Najib Arung Petana Raja Bone:** Validation, Data acquisition, Data curation.
**Barandi Sapta Widartono:** Validation, Data acquisition, Data curation, Conceptualization.
**Hery Wijayanto:** Validation, Data acquisition, Data curation.
**Dyah Ayu Widiasih:** Validation, Data curation.
**Mia Nur Farida:** Validation, Data acquisition, Data curation.
**Wayan Tunas Artama:** Validation, Project administration, Data acquisition, Data curation, Conceptualization.
**Thibaut Langlois:** Validation, Data curation.
**Hélène Guis:** Validation, Supervision, Project administration, Data curation.
**Etienne Loire:** Validation, Supervision, Funding acquisition, Conceptualization.
**Michel de Garine-Wichatitsky:** Writing – review & editing, Writing – original draft, Validation, Supervision, Project administration, Methodology, Investigation, Funding acquisition, Data acquisition, Data curation, Conceptualization.

## Declaration of competing interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

## Acknowledgments

The authors would like to thank the GENCI for granting access to their resources using HPC resources from GENCI-IDRIS (Grant 2024-AD011015410).

The authors would also like to thank the interns who worked on this project: Hoang Duong, Alexandre Hayoun, and Antoine Hounsi. Their contributions on annotating the data and refining our data processing pipeline were very valuable to this project.

## Data availability

Both the YT-BB-Dog and Tibetan datasets, along with the trained model weights, are publicly available at https://www.lirmm.fr/YT-BB-Dog_Siberan/. The code is publicly available at https://github.com/eugeniodias5/BIFOR/.

## References

* Borwarginn, P., Kusakunniran, W., Karnjanapreechakorn, S., Thongkanchorn, K., 2021. Knowing your dog breed: Identifying a dog breed with deep learning. Int. J. Autom. Comput. 18 (1), 45–54. http://dx.doi.org/10.1007/s11633-020-1261-0

* Bradley, P.S., Bennett, K.P., Demiriz, A., 2000. Constrained k-means clustering. Microsoft Res. Redmond 20, URL https://www.microsoft.com/en-us/research/wp-content/uploads/2016/02/tr-2000-65.pdf

* Cao, J., Tang, H., Fang, H.S., Shen, X., Tai, Y.W., Lu, C., 2019. Cross-domain adaptation for animal pose estimation. In: 2019 IEEE/CVF International Conference on Computer Vision. ICCV, pp. 9497–9506. http://dx.doi.org/10.1109/ICCV.2019.00959

* Čermák, V., Picek, L., Adam, L., Papafitsoros, K., 2024. Wildlifedatasets: An open-source toolkit for animal re-identification. In: 2024 IEEE/CVF Winter Conference on Applications of Computer Vision. WACV, pp. 5941–5951. http://dx.doi.org/10.1109/WACV57701.2024.00585

* Chen, K., Wang, J., Pang, J., Cao, Y., Xiong, Y., Li, X., Sun, S., Feng, W., Liu, Z., Xu, J., Zhang, Z., Cheng, D., Zhu, C., Cheng, T., Zhao, Q., Li, B., Lu, X., Zhu, R., Wu, Y., Dai, J., Wang, J., Shi, J., Ouyang, W., Loy, C.C., Lin, D., 2019. MMDetection: Open MMLab detection toolbox and benchmark. http://dx.doi.org/10.48550/arXiv.1906.07155, arXiv:1906.07155

* Ciaparrone, G., Luque Sánchez, F., Tabik, S., Troiano, L., Tagliaferri, R., Herrera, F., 2020. Deep learning in video multi-object tracking: A survey. Neurocomputing 381, 61–88. http://dx.doi.org/10.1016/j.neucom.2019.11.023

* de Garine-Wichatitsky, M., et al., 2024. One health in practice: Seadogsea! a socio-ecological approach for the study and management of zoonotic diseases associated with free-roaming domestic dogs in south-east Asia. In: 3rd Joint AITVM-STVM International Conference. Montpellier, France, Poster Presentation. URL https://cnam.hal.science/RIIPCAMBODGE/hal-04637825v1

* Du, Y., Zhao, Z., Song, Y., Zhao, Y., Su, F., Gong, T., Meng, H., 2023. StrongSORT: Make DeepSORT great again. IEEE Trans. Multimed. 25, 8725–8737. http://dx.doi.org/10.1109/TMM.2023.3240881

* Fu, D., Chen, D., Bao, J., Yang, H., Yuan, L., Zhang, L., Li, H., Chen, D., 2021. Unsupervised pre-training for person re-identification. In: 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition. CVPR, pp. 14745–14754. http://dx.doi.org/10.1109/CVPR46437.2021.01451

* Gong, Y., Zeng, Z., 2021. An effective data augmentation for person re-identification. http://dx.doi.org/10.48550/arXiv.2101.08533

* Gray, D., Brennan, S., Tao, H., 2007. Evaluating appearance models for recognition, reacquisition, and tracking. Proc. IEEE Int. Work. Vis. Surveill. Perform. Eval. Track. Surveill., Oct.

* Harley, D., Eyre, A., 2024. Ten years of camera trapping for a cryptic and threatened arboreal mammal – a review of applications and limitations. Wildl. Res. 51, http://dx.doi.org/10.1071/WR23054

* He, T., Jin, X., Shen, X., Huang, J., Chen, Z., Hua, X.S., 2021b. Dense interaction learning for video-based person re-identification. In: 2021 IEEE/CVF International Conference on Computer Vision. ICCV, pp. 1470–1481. http://dx.doi.org/10.1109/ICCV48922.2021.00152

* He, S., Luo, H., Wang, P., Wang, F., Li, H., Jiang, W., 2021a. TransReID: Transformer-based object re-identification. In: 2021 IEEE/CVF International Conference on Computer Vision. ICCV, IEEE Computer Society, Los Alamitos, CA, USA, pp. 14993–15002. http://dx.doi.org/10.1109/ICCV48922.2021.01474
He, Z., Qian, J., Yan, D., Wang, C., Xin, Y., 2023. Animal re-identification algorithm for posture diversity. In: ICASSP 2023 - 2023 IEEE International Conference on Acoustics, Speech and Signal Processing. ICASSP, pp. 1-5. http://dx.doi.org/10.1109/ICASSP49357.2023.10094783.

He, K., Zhang, X., Ren, S., Sun, J., 2016. Deep residual learning for image recognition. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition. CVPR, pp. 770-778. http://dx.doi.org/10.1109/CVPR.2016.90.

Herzog, F., Ji, X., Teepe, T., Hormann, S., Gilg, J., Rigoll, G., 2021. Lightweight multi-branch network for person re-identification. In: 2021 IEEE International Conference on Image Processing. ICIP, IEEE, http://dx.doi.org/10.1109/icip4298.2021.9506733.

Jang, D.H., Kwon, K.S., Kim, J.K., Yang, K.Y., Kim, J.B., 2020. Dog identification method based on muzzle pattern image. Appl. Sci. 10 (24), http://dx.doi.org/10.3390/app10248994.

Jiao, B., Liu, L., Gao, L., Wu, R., Lin, G., Wang, P., Zhang, Y., 2023. Toward re-identifying any animal. In: Oh, A., Neumann, T., Globerson, A., Saenko, K., Hardt, M., Levine, S. (Eds.), In: Advances in Neural Information Processing Systems, vol. 36, Curran Associates, Inc., pp. 40042-40053, URL https://proceedings.neurips.cc/paper_files/paper/2023/file/7df69dbf39705c7a39b40f2d70e806c1-Paper-Conference.pdf.

Khosla, A., Jayadevaprasak, N., Yao, B., Fei-Fei, L., 2011. Novel dataset for fine-grained image categorization. In: Proceedings of the First Workshop on Fine-Grained Visual Categorization, FGVC'2011, IEEE Conference on Computer Vision and Pattern Recognition, CVPR'2011. Colorado Springs, CO.

Lai, K., Tu, X., Yanushkevich, S., 2019. Dog identification using soft biometrics and neural networks. In: 2019 International Joint Conference on Neural Networks. IJCNN, pp. 1-8. http://dx.doi.org/10.1109/IJCNN.2019.8851971.

Li, S., Li, J., Tang, H., Qian, R., Lin, W., 2020. ATRW: A benchmark for amur tiger re-identification in the wild. In: Proceedings of the 28th ACM International Conference on Multimedia. MM '20, Association for Computing Machinery, New York, NY, USA, pp. 2590-2598. http://dx.doi.org/10.1145/3394171.3413569.

Liao, S., Mo, Z., Zhu, J., Hu, Y., Li, S.Z., 2014. Open-set person re-identification. URL https://arxiv.org/abs/1408.0872. arXiv:1408.0872.

Liao, S., Shao, L., 2019. Interpretable and generalizable deep image matching with adaptive convolutions. arXiv abs/1904.10424. URL https://api.semanticscholar.org/CorpusID:128358623.

Liao, S., Shao, L., 2021. Transmatcher: Deep image matching through transformers for generalizable person re-identification. In: Neural Information Processing Systems. URL https://api.semanticscholar.org/CorpusID:244921162.

Liao, S., Shao, L., 2022. Graph sampling based deep metric learning for generalizable person re-identification. In: 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition. CVPR, pp. 7349-7358. http://dx.doi.org/10.1109/CVPR52688.2022.00721.

Lin, T.Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., Zitnick, C.L., 2014. Microsoft COCO: Common objects in context. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (Eds.), Computer Vision - ECCV 2014. Springer International Publishing, Cham, pp. 740-755. http://dx.doi.org/10.1007/978-3-319-10602-1-48.

Liu, J., Kanazawa, A., Jacobs, D., Belhumeur, P., 2012. Dog breed classification using part localization. In: Fitzgibbon, A., Lazebnik, S., Perona, P., Sato, Y., Schmid, C. (Eds.), Computer Vision - ECCV 2012. Springer Berlin Heidelberg, Berlin, Heidelberg, pp. 172-185. http://dx.doi.org/10.1007/978-3-642-33718-5_13.

Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., Guo, B., 2021. Swin transformer: Hierarchical vision transformer using shifted windows. In: 2021 IEEE/CVF International Conference on Computer Vision. ICCV, pp. 9992-10002. http://dx.doi.org/10.1109/ICCV48922.2021.00986.

Liu, Z., Mao, H., Wu, C.Y., Feichtenhofer, C., Darrell, T., Xie, S., 2022. A ConvNet for the 2020s. In: 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition. CVPR, pp. 11966-11976. http://dx.doi.org/10.1109/CVPR52688.2022.01167.

Liu, C., Zhang, R., Guo, L., 2019a. Part-pose guided amur tiger re-identification. In: 2019 IEEE/CVF International Conference on Computer Vision Workshop. ICCVW, pp. 315-322. http://dx.doi.org/10.1109/ICCVW.2019.00042.

Liu, N., Zhao, Q., Zhang, N., Cheng, X., Zhu, J., 2019b. Pose-guided complementary features learning for amur tiger re-identification. In: 2019 IEEE/CVF International Conference on Computer Vision Workshop. ICCVW, pp. 286-293. http://dx.doi.org/10.1109/ICCVW.2019.00038.

Lynch, A., Dovonon, G.J.S., Kaddour, J., Silva, R., 2023. Spawrous: A benchmark for fine control of spurious correlation biases. URL https://arxiv.org/abs/2303.05470. arXiv:2303.05470.

Moreira, T., Perez, M., Werneck, R., Valle, E., 2017. Where is my puppy? Retrieving lost dogs by facial features. Multimedia Tools Appl. 76, http://dx.doi.org/10.1007/s11042-016-3824-1.

Mougeot, G., Li, D., Jia, S., 2019. A deep learning approach for dog face verification and recognition. In: Nayak, A.C., Sharma, A. (Eds.), PRICAI 2019: Trends in Artificial Intelligence. Springer International Publishing, Cham, pp. 418-430. http://dx.doi.org/10.1007/978-3-030-29894-8_4_34.

Nepovinnykh, E., Chelak, I., Eerola, T., Immonen, V., Kälviäinen, H., Kholiavchenko, M., Stewart, C.V., 2024. Species-agnostic patterned animal re-identification by aggregating deep local features. Int. J. Comput. Vis. 1-16. http://dx.doi.org/10.1007/s11263-024-02071-1.

Ni, H., Li, Y., Gao, L., Shen, H.T., Song, J., 2023. Part-aware transformer for generalizable person re-identification. In: 2023 IEEE/CVF International Conference on Computer Vision. ICCV, pp. 11246-11255. http://dx.doi.org/10.1109/ICCV51070.2023.01036.

Otranto, D., Dantas-Torres, F., Mihalca, A.D., Traub, R.J., Lappin, M., Baneth, G., 2017. Zoonotic parasites of sheltered and stray dogs in the era of the global economic and political crisis. Trends Parasitol. 33 (10), 813-825. http://dx.doi.org/10.1016/j.pt.2017.05.013.

Radenović, F., Tolias, G., Chum, O., 2016. CNN image retrieval learns from BoW: Unsupervised fine-tuning with hard examples. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (Eds.), Computer Vision - ECCV 2016. Springer International Publishing, Cham, pp. 3-20. http://dx.doi.org/10.1007/978-3-319-46448-0_1.

Ravoor, P., Tsb, S., 2020. Deep learning methods for multi-species animal re-identification and tracking - a survey. Comput. Sci. Rev. 38, 100289. http://dx.doi.org/10.1016/j.cosrev.2020.100289.

Real, E., Shlens, J., Mazzocchi, S., Pan, X., Vanhoucke, V., 2017. Youtube-BoundingBoxes: A large high-precision human-annotated data set for object detection in video. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition. CVPR, pp. 7464-7473. http://dx.doi.org/10.1109/CVPR.2017.789.

Redmon, J., Divvala, S., Girshick, R., Farhadi, A., 2016. You only look once: Unified, real-time object detection. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition. CVPR, pp. 779-788. http://dx.doi.org/10.1109/CVPR.2016.91.

Schroff, F., Kalenichenko, D., Philbin, J., 2015. Facenet: A unified embedding for face recognition and clustering. In: 2015 IEEE Conference on Computer Vision and Pattern Recognition. CVPR, pp. 815-823. http://dx.doi.org/10.1109/CVPR.2015.7298682.

Shrivastava, A., Gupta, A., Girshick, R., 2016. Training region-based object detectors with online hard example mining. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition. CVPR, IEEE Computer Society, Los Alamitos, CA, USA, pp. 761-769. http://dx.doi.org/10.1109/CVPR.2016.89.

Simões, F., Bouveyron, C., Precioso, F., 2023. DeepWILD: Wildlife identification, localisation and estimation on camera trap videos using deep learning. Ecol. Informatics 75, 102095. http://dx.doi.org/10.1016/j.ecoinf.2023.102095.

Tan, M., Le, Q.V., 2019. Efficientnet: Rethinking model scaling for convolutional neural networks. CoRR abs/1905.11946. URL http://arxiv.org/abs/1905.11946. arXiv:1905.11946.

Tanwar, K., Sadhu, A., Jhala, Y., 2021. Camera trap placement for evaluating species richness, abundance, and activity. Sci. Rep. 11, http://dx.doi.org/10.1038/s41598-021-02459-w.

Tian, M., Yi, S., Li, H., Li, S., Zhang, X., Shi, J., Yan, J., Wang, X., 2018. Eliminating background-bias for robust person re-identification. In: 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 5794-5803. http://dx.doi.org/10.1109/CVPR.2018.00607.

Wang, G., Lai, J., Huang, P., Xie, X., 2018. Spatial-temporal person re-identification. CoRR abs/1812.03282. URL http://arxiv.org/abs/1812.03282. arXiv:1812.03282.

Wiezorek, M., Rychalska, B., Dabrowski, J., 2021. On the unreasonable effectiveness of centroids in image retrieval. CoRR abs/2104.13634. URL https://arxiv.org/abs/2104.13634. arXiv:2104.13634.

Wojke, N., Bewley, A., Paulus, D., 2017. Simple online and realtime tracking with a deep association metric. In: 2017 IEEE International Conference on Image Processing. ICIP, pp. 3645-3649. http://dx.doi.org/10.1109/ICIP.2017.8296962.

Wu, D., Zheng, S.J., Zhang, X.P., Yuan, C.A., Cheng, F., Zhao, Y., Lin, Y.J., Zhao, Z.Q., Jiang, Y.L., Huang, D.S., 2019. Deep learning-based methods for person re-identification: A comprehensive review. Neurocomputing 337, 354-371. http://dx.doi.org/10.1016/j.neucom.2019.01.079.

Xiao, K., Engstrom, L., Ilyas, A., Madry, A., 2020. Noise or signal: The role of image backgrounds in object recognition. URL https://arxiv.org/abs/2006.09994. arXiv:2006.09994.

Xu, P., Zhu, X., 2023. Deepchange: A long-term person re-identification benchmark with clothes change. In: 2023 IEEE/CVF International Conference on Computer Vision. ICCV, pp. 11162-11171. http://dx.doi.org/10.1109/ICCV51070.2023.01028.

Yan, C., Pang, G., Bai, X., Liu, C., Ning, X., Gu, L., Zhou, J., 2022. Beyond triplet loss: Person re-identification with fine-grained difference-aware pairwise loss. IEEE Trans. Multimed. 24, 1665-1677. http://dx.doi.org/10.1109/TMM.2021.3069562.

Ye, M., Shen, J., Lin, G., Xiang, T., Shao, L., Hoi, S.C.H., 2022. deep learning for person re-identification: A survey and outlook. IEEE Trans. Pattern Anal. Mach. Intell. 44 (06), 2872-2893. http://dx.doi.org/10.1109/TPAMI.2021.3054775.

Yi, D., Lei, Z., Liao, S., Li, S.Z., 2014. Deep metric learning for person re-identification. In: 2014 22nd International Conference on Pattern Recognition. pp. 34-39. http://dx.doi.org/10.1109/ICPR.2014.16.
Yu, J., Su, H., Liu, J., Yang, Z., Zhang, Z., Zhu, Y., Yang, L., Jiao, B., 2019. A strong baseline for tiger re-ID and its bag of tricks. In: 2019 IEEE/CVF International Conference on Computer Vision Workshop. ICCVW, pp. 302–309. http://dx.doi.org/10.1109/ICCVW.2019.00040.

Zhai, Y., Guo, X., Lu, Y., Li, H., 2019. In defense of the classification loss for person re-identification. In: 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops. CVPRW, pp. 1526–1535. http://dx.doi.org/10.1109/CVPRW.2019.00194.

Zhang, Y.F., Zhang, Z., Li, D., Jia, Z., Wang, L., Tan, T., 2023. Learning domain invariant representations for generalizable person re-identification. IEEE Trans. Image Process. 32, 509–523. http://dx.doi.org/10.1109/TIP.2022.3229621.

Zheng, P., Gao, D., Fan, D.P., Liu, L., Laaksonen, J., Ouyang, W., Sebe, N., 2024. Bilateral reference for high-resolution dichotomous image segmentation. CAAI Artif. Intell. Res. 3, 9150038.

Zheng, L., Shen, L., Tian, L., Wang, S., Wang, J., Tian, Q., 2015. Scalable person re-identification: A benchmark. In: 2015 IEEE International Conference on Computer Vision. ICCV, pp. 1116–1124. http://dx.doi.org/10.1109/ICCV.2015.133.

Zhou, K., Yang, Y., Cavallaro, A., Xiang, T., 2019. Omni-scale feature learning for person re-identification. In: 2019 IEEE/CVF International Conference on Computer Vision. ICCV, pp. 3701–3711. http://dx.doi.org/10.1109/ICCV.2019.00380.

Zou, D.N., Zhang, S.H., Mu, T.J., Zhang, M., 2020. A new dataset of dog breed images and a benchmark for finegrained classification. Comput. Vis. Media 6, http://dx.doi.org/10.1007/s41095-020-0184-6.