dataesr/scientific-paragraphs-categorization huggingface dataset:
images with the identity (or an attribute) of each individual. The amount of data required to train a CNN is expected to be proportionally dependent on the difficulty of the classification challenge, that is, a bear and a bird would be easier to differentiate than two bears of the same species. Usually, CNNs that achieve large generalization capability need to be trained over thousands to millions of pictures  (Marcus, 2018). Such large datasets are required because the aim of using a CNN is to generalize recognition from the specific data that the CNN has been exposed to during training. For example, if a CNN was trained to distinguish two bears of the same species with only pictures of the individuals lying down, it might be unable to identify those same individuals from new pictures taken when the animals are standing up.
Additionally, if the pictures used for training were taken during a short period of time, it might lead the CNN to rely on superficial and temporary features for identification. For example, if pictures for training were taken when one of the individuals had a large wound or was going through moulting or shedding, it might result in a CNN that relies on those salient and temporary features, and thus perform badly when having to predict the identity of the individuals a few days later. Therefore, effectively making use of deep learning for individual identification, especially in the wild, requires new ways to collect training data that do not rely on individual manual image annotation. When working in captivity settings, such large labelled image datasets can be easily collected by temporarily isolating the animals in enclosures separated from the rest of the group while filming or photographing them.
However, such an approach is clearly not feasible for researchers working on wild populations, making collecting training data from wild animals much more challenging. For example, in birds, relying on human observers and colour rings, to photograph and manually label enough pictures to implement CNN for individual identification, would be extremely time-consuming. Furthermore, in longer term studies, animals can change their appearance over time (e. g. changing from juvenile to adult plumage in birds) or new individuals may join the population (e. g. immigrants or recruited offspring). These cases require that the process of identifying individuals and labelling photos is routinely repeated. Therefore, relying on human observers for collecting labelled data in this type of systems might hinder the widespread implementation of deep learning techniques for individual identification, or restrict its application to short-term projects.
Here we provide an efficient pipeline for collecting training data, both in captivity and in the wild, and we train CNNs for individual re-identification (i. e. machine recognition of a previously known set of individuals). We demonstrate the feasibility of our approaches using data from two wild populations of birds of two different species, the sociable weaver Philetairus socius and the great tit Parus major, and a population of captive zebra finches Taeniopygia guttata. We then show that CNNs trained on these species can successfully re-identify individuals across a range of different contexts. We start by (a) focusing on the problem of efficiently collecting large training datasets. We provide simple and automated methods for collecting a very large number of labelled pictures by using low-cost cameras that can be programmed to take labelled pictures of birds.
In captivity, we achieve this by temporarily isolating target individuals, and taking pictures using low-cost cameras. In the wild, we describe a solution using low-cost RFIDs and low-cost cameras that are programmed to take labelled picture when PIT-tagged birds land on an RFID-equipped feeder. We then (b) provide details of the steps involved with data pre-processing and the training of an adequate CNN. We further describe approaches for augmenting our training datasets using algorithms that add noise and make modifications to the original images. Next, we (c) evaluate the generalization performance of our CNNs to data collected in other contexts by evaluating the ability of our models to predict the identity of the birds in pictures collected using different cameras and in contexts that differ from the ones used for collecting the training datasets.
| Training data collection in the wild The collection of labelled pictures in the wild was automated by combining RFID technology (Priority1Design, Australia), single-board computers (Raspberry Pi), Pi cameras and artificial feeders. We fitted RFID antenna to small perches placed in front of bird feeders filled with seeds (Figure  1a-c ). The RFID data logger was then directly connected to a Raspberry Pi (detailed explanation of the developed setup is available at github. com/AndreCFerreira/Bird_individualID) which had a Pi camera (we used Pi camera V1 5mp and V2 8mp). When the RFID data logger detected a bird, it sent the individual's PIT-tag code to the Raspberry Pi, which was programmed to then take a picture. Because birds often spend some time on the feeder, we programmed the Raspberry PI to take a picture every 2 s while the bird remained present.
This interval was introduced in order to efficiently collect data while avoid having near-identical frames of the same bird as having too many near-identical pictures could increase the overfitting of the CNN, that is, the risk of the model 'memorizing' the pictures instead of learning features that are key for recognizing the individuals and thus jeopardize the generalization capability of the models (see Section 2. 3). Each picture file was automatically labelled with the bird identity, known from the RFID logger and the time of shooting in the filename. Training data collection was therefore automatized by linking the identity of the bird perching on the antenna while feeding to its pictures, without any need for human manual identification and annotation. When multiple birds perched on the feeder at the same time, it was not possible to determine which of the birds activated the RFID system.
| Training data collection in captivity We temporarily divided cages into equally sized partitions with a net, allowing us to take pictures from individual birds without completely socially isolating them. We collected data from 10 zebra finches (five males and five females). We placed two Raspberry Pi cameras on the roof of each partition to photograph (every 2 s) the birds sitting on the wooden perches (Figure  1d ). Each bird was recorded for 4 hr. Since we knew which Raspberry Pi photographed which bird, we avoided the need to manually link the identity of the birds to the pictures.
| Data pre-processing To efficiently train a CNN, the regions in the pictures corresponding to the birds should be extracted from the background (third step of Figure  2 ). A Mask R-CNN  (He, Gkioxari, Dollár, & Girshick, 2017)  was used to automatically localize and crop the bird in the pictures. For the sociable weavers, we used a Mask R-CNN model that had been trained on Microsoft COCO  (Lin et al. , 2014). Microsoft COCO is a generalist dataset which includes pictures of birds and therefore is able to localize the sociable weavers in the pictures (see github. com/ Andre CFerr eira/Bird_indiv idualID for details). Because the sociable For the sociable weavers and the great tits, if the Mask R-CNN identified more than one bird perching simultaneous at the RFID antenna, we automatically excluded that image.
We detected a total of 35 sociable weavers at the RFIDs antennas. Of these, 30 individuals with more than 350 pictures were used to train the classifier. In the great tit population, 77 birds were photographed, of which 10 had more than 350 pictures. These 10 individuals were used to train a CNN for each of the species. The remaining five sociable weavers and 67 great tits (with <350 pictures) were used to address the issue of working in open areas where new individuals can constantly be recruited to the study population (see Section 2. 5 below). For the zebra finches, we used all 10 individuals as our setup resulted in more than 2,000 pictures for each bird. We applied limited oversampling to our training dataset only. For nine sociable weavers and six great tits for which we did not have 1,000 images, we first selected 100 images for the validation dataset and then duplicated (through oversampling) the remaining pictures until 900 images were available for the training dataset  (Buda, Maki, & Mazurowski, 2018).
Oversampling was therefore restricted to the training dataset and not applied to the validation dataset in order to avoid overestimating the model's learning progress. For both species, in order to limit overfitting caused by having very similar pic- Working on the captive zebra finches, we could easily collect many images per bird. However, the problem of collecting data of animals that are in confined enclosures is that a significant number of pictures could potentially be near-identical, such as if an individual stays motionless for long periods of time. In our case, all birds were generally active and visited all the places in their cage (i. e. all wooden perches, floor, water and food plates). Nevertheless, to avoid potential overestimation of the model's accuracy, we used the images collected when the birds were in different partitions for training and validation datasets.
Additionally, to create a diverse set of validation pictures, we used a structural similarity index measure (SSIM;  Wang, Bovik, Sheikh, & Simoncelli, 2004)  to create a dataset with maximized pairwise dissimilarity among images (following a similar procedure as  Hansen et al. , 2018  for a pig dataset). We started by randomly selecting an image to include in the validation dataset. We then randomly sampled images and computed the SSIM between the new image and those already in the validation dataset.
| Convolutional neural networks If the SSIM value was smaller than a threshold, these new pictures were included in the validation dataset. This process was repeated by sequentially comparing a new picture to all the ones already in the validation dataset until we reached 160 images per bird. The threshold value used (0. 55) was empirically determined by trying different values and looking at the resulting datasets. For the training dataset, 1,600 images of each zebra finch were randomly selected without filtering for near-identical images. All birds had at least 1,600 images, except for one that had 1,197 for which oversampling was used by creating duplicates of 403 randomly sampled images. We used the VGG19 CNN architecture  (Simonyan & Zisserman, 2014 ) and initialized the model with the weights of a network pretrained on the ImageNet dataset (a dataset with more than 14 million pictures and 20,000 classes,  Deng et al.
, 2009). The main idea behind using networks pre-trained on other datasets is that features (such as colour or texture) that are important to distinguish multiple objects could also be useful to distinguish between individual birds. When using transfer learning, the bottom layers of the network can be frozen in order to mitigate overfitting, this is especially important when the training datasets are small. However, as freezing the layers prevent them from update their weights during the training process (and therefore could prevent the model from learning key features for performing the classification task) and considering the size of our training datasets, we decided to train the models without freezing any of the layers of the network. The fully connected part of the VGG19 CNN network (i.
e. the classifier part) was replaced by layers with random weights that fit our particular task of interest and the corresponding number of classes (i. e. number of different individuals; Figure  S1 ). To further increase our training sample, we then used a data augmentation procedure. This procedure consists of artificially increasing the sample size by applying transformations to an existing set of samples. Using the data generator available in Keras  (Chollet, 2015) , we randomly rotated (from 0 to 40°) and zoomed (zoom range of 0. 2) images of all species. We additionally applied horizontal and vertical flips to the great tits and zebra finches populations, and as contrary to the sociable weavers, these birds could be photographed from any orientation (as they perched all around the RFID antenna or the cage on which perch their bodies can be facing different directions).
These transformations were applied randomly to every single One dropout layer was added just before the first dense layer (see github. com/Andre CFerr eira/Bird_indiv idualID and Figure  S1  for details on the network architecture). Dropout layers are used to limit overfitting by randomly ignoring units of the CNN (i. e. neurons) during the training process (see  Srivastava, Hinton, Krizhevsky, Sutskever, & Salakhutdinov, 2014  for details on dropout). For the sociable weavers and the zebra finches, the dropout layer had a value of 0. 5, while for the great tits it was reduced to 0. 2 (i. e. less units are being ignored in order to facilitate the training process) as the model did not improve the accuracy from a random guess for 10 epochs when the dropout was at an initial value of 0. 5. We used a softmax activation function for the classifier and ADAM optimizer  (Kingma & Ba, 2014)  with a learning rate of 1 e-5.
90% accuracy with the validation dataset, the accuracy was significantly lower when generalizing to other contexts (see Sections 2. 4 and 3). We suspected that such differences could be due to the lower quality of images collected in those other contexts (with different cameras, capture distances and conditions; see Section 2. 4). To account for this possibility, we trained a model using the same setting parameters that yielded the best results, and applying further transformation. In order to simulate the lower quality of the pictures taken in other contexts, we applied Gaussian blur, motion blur, Gaussian noise, resizing transformations and a random combination of two of these four transformations (see github. com/ Andre CFerr eira/Bird_indiv idualID for details on the transformations applied to the images) to each of the images in the dataset used to train the models (Figure  3 ).
The idea is that even if the overall quality of the pictures in the dataset used for training slightly differs from pictures which are of interest for a research question, this training dataset can be transformed in order to be more similar to the pictures collected in distinct contexts for which the classifier could be applied on. Blur and noise transformations were not used for the great tits and zebra finches as there were no differences in the overall quality of the pictures used for training and for testing the model generalization capability (see Section 2. 4).
SD). As a result, we did not attempt to make a balanced dataset and, therefore, used all the 94 pictures collected at this new feeding setup. For the zebra finches, we did not have a second setup that differed from the one used to collect the pictures to train a CNN and that could be used for testing the CNN generalization. Instead, we ran an additional trial which consisted of recording the birds together to see how well the model would predict the identity of each individual when they are in small groups interacting with each other (Figure  4f ). Since these birds did not have any visual tags and it was not possible to distinguish them when in group, we used one flock of three birds and another flock of two birds for each sex. This allows us to estimate the model's accuracy by calculating the number of times that the CNN wrongly attributed the identity of a bird as being an individual that was not actually present in that flock.
| New birds In the wild, it is common for new individuals to join a population during the course of a study. These new individuals may challenge the performance of a CNN, because the model outputs a vector from a softmax layer that indicates probabilities of presence for every individual present during training, with the sum of these probabilities being one (see 'classification' stage in Figure  2 ). In order to study this potential issue, we used the already trained CNNs from the subset of identities, where we had to predict the identity of birds that were not included in the training datasets. For the sociable weavers, we had a scenario in which a CNN that was trained to identify a relatively large number of individuals (30) was then exposed to a small number of new individuals (5). For the great tits, we had the opposite scenario in which a CNN that was trained for a small number of individuals (10) was then exposed to a large number of new individuals (67).
For the sociable weavers, we selected 50 pictures of each of the five birds (a total of 250) that were not in the training dataset and 250 random images from the pool of birds that were included in the training data. For the great tits, we selected 250 random images from the pool of 67 individuals that were not in the training dataset, and kept a random set of 250 images from the birds in the training data. We limited the number of pictures from the same individual to a maximum of eight (3. 91 ± 1. 67 M ± SD) in order to keep a large number of different individuals in this dataset (64 out of the 67 individuals were used). Shannon's entropy  (Shannon, 1948)  of each of the distributions was calculated from the classification (softmax) output to empirically determine a confidence threshold to consider a bird as part of the training dataset.
F I G U R E 4 Examples of data collected across different contexts. For the sociable weaver, we collected images at the feeders from the RIFD feeder set-up from (a) close or (b) far perspectives. We then collected data at a feeding plate on the floor, which we recorded from (c) a ground perspective and (d) a top perspective. For the great tits, we (e) recorded from a top perspective feeding at a table on which we placed an RFID antenna. Finally, for zebra finches we (f) collected data from social groups.
| New birds The entropy of the softmax outputs (i. e. probabilities) was smaller when predicting the identity of birds present in the training dataset, compared to when predicting the identity of new birds (Figure  5 ). This is due to the fact that when predicting the identity of a bird from the training dataset, there is usually one that stands out with very high probability (thus successfully indicating the bird's identity) and the remaining probabilities are very low (other birds' identities). In contrast, when predicting the identity of a new bird, the probabilities were usually more equally distributed across all classes, all with low values. For the sociable weavers, 90% of entropies were below 0. 75 when predicting the identity of birds from the training dataset and only 17% of them were under this value when predicting the identity of new birds.
| D ISCUSS I ON Deep learning has the potential to revolutionize the way in which researchers identify individuals. Here we propose a practical way of collecting large labelled datasets, which is currently the main bottleneck preventing the application of deep learning for individual identification in animals  (Schneider, Taylor, Linquist, & Kremer, 2019). We also show the steps required to train a classifier for individual re-identification. To our knowledge, this is the first successful attempt of performing such an individual recognition in small birds. Using data collected with automatized procedures, CNNs proved to be effective for re-identifying known individuals in three different bird species, including two species that are among the most commonly used models in the field of behavioural ecology (great tits and zebra finches).
One solution to this is to ensure that the training dataset contains sufficient variation to capture the broad range of contexts that the CNN is required for. The arrival of new individuals to the study population is another challenge that needs to be carefully addressed. If these new birds are marked with a PIT-tag, the CNN could be updated similarly to the problem of changes in appearance discussed above. However, in many cases new individuals will not be marked. Such a problem fits in the anomaly  (Chandola, Banerjee, & Kumar, 2009)  and novelty  (Pimentel, Clifton, Clifton, & Tarassenko, 2014)  detection domain. Here we explored a simple approach involving investigating the entropy of classification probabilities. Our solution appears useful if the CNN was trained on a relatively large number of individuals and if immigrants are uncommon in the population, like in the sociable weaver example.
However, for some studies, such conditions might not be met and, as it was the case of the great tit scenario, where we had a low number of individuals in the training dataset and observed a large number of new birds. Nevertheless, the identification accuracy of a CNN should also be considered from a post-detection analysis perspective. While some studies will benefit from maximize the number of identifications made, in other studies it may be more costly to have misidentified individuals For example, misidentifications are very costly when constructing social networks  (Davis, Crofoot, & Farine, 2018 ), while at the same time social networks are very identification hungry  (Farine & Strandburg-Peshkin, 2015). Thus, exploration of the entropy distribution and other approaches, and subsequent trade-offs, should be considered.
In addition, the error rate might be also reduced through post-processing. For example, if the identification is based on a collection of frames (e. g. images extracted from a short video recording of the animal) instead of single image, then the sequence of detections (and assignment probabilities) can be quantified over subsequent frames, and the detection can be kept or discarded depending on the overall confidence in the sequence of detected identification. The field of deep learning progresses rapidly and almost continuously provides solutions to seemingly challenging problems. However, this is facilitated by the existence of large and freely availed databases, which are used to try different approaches for a wide range of classification problems. For example, the ImageNet database  (Deng et al.
Since manually labelling the regions of interest is time-consuming, we started by training the model for 10 epochs (i. e. passing the entire dataset through the neural network 10 times) with 200 manually labelled pictures. If the model was found to perform badly, additional pictures were manually labelled and added to the training dataset. This process was repeated until a satisfactory performance was achieved. For the great tits, we needed 500 pictures in the training data and 125 for validation (see Section 2. 3 below for explanation on training and validation datasets). The zebra finch data required 400 pictures for training and 100 for validation.
Training a CNN requires both a training and a validation dataset. The training dataset is the set of samples that the neural network repeatedly uses to learn how to classify the input images into different classes (in our case, different individuals). The validation dataset is an independent set of samples that is used to compute the accuracy and loss (estimation of the error during training) of the model. This validation dataset is used to assess the learning progress of the neural network. As the network never trains on or sees the validation data, this validation dataset can indicate if the model is overfitting the training data and not learning features that are key for recognizing the individuals. It is generally difficult to anticipate the minimum number of images needed from each individual to obtain high performance for individual recognition.