Data Pruning Can Do More: A Comprehensive Data Pruning Approach for Object Re-identification
Authors: Zi Yang
Haojin Yang
Soumajit Majumder
Jorge Cardoso
Guillermo Gallego
Venue: Accepted by TMLR ()

Abstract: Previous studies have demonstrated that not each sample in a dataset is of equal importance during training. Data pruning aims to remove less important or informative samples while still achieving comparable results as training on the original (untruncated) dataset, thereby reducing storage and training costs. However, the majority of data pruning methods are applied to image classification tasks. To our knowledge, this work is the first to explore the feasibility of these pruning methods applied to object re-identification (ReID) tasks, while also presenting a more comprehensive data pruning approach. By fully leveraging the logit history during training, our approach offers a more accurate and comprehensive metric for quantifying sample importance, as well as correcting mislabeled samples and recognizing outliers. Furthermore, our approach is highly efficient, reducing the cost of importance score estimation by 10 times compared to existing methods. Our approach is a plug-and-play, architecture-agnostic framework that can eliminate/reduce 35%, 30%, and 5% of samples/training time on the VeRi, MSMT17 and Market1501 datasets, respectively, with negligible loss in accuracy (< 0.1%). The lists of important, mislabeled, and outlier samples from these ReID datasets are available at https://github.com/Zi-Y/data-pruning-reid.

Published in Transactions on Machine Learning Research (04/2024)
Data Pruning Can Do More: A Comprehensive Data Pruning Approach for Object Re- identification
Zi Yang Technical University of Berlin Huawei Munich Reseorch Center
zi. zi.ya @campus tu-berlin de
Haojin Yang
Haojin. Yang@hpi.de de
Hasso Plattner Institute
Soumajīt Majumder Huowei Munich Research Center
sommaii..ncjomder@@wwwwe...cco
Jorge Cardoso
Jorge. Cwwdoso@hawwwii.coooommmmmm
Huowes Munich Research Center CISUC, Universily of Coimbra
Guillermo Gallego
guillerno.galleggotttebeeeee
Technical University of Berlin
Einstein Center Digital Future
Science of Intelligence Excellence Cluster
Reviewed on OpenReview:
Abstract
Previous studies have demonstrated that not each sample in a dataset is of equal importance during training. Data pruming aims to remove less important or informative samples while still achieving comparable results as training on the original (untruncated) dataset, thereby reducing storage and training costs. However, the majority of data pruning methods applied image classification tasks. To our knowledge, this work is the first to explore the fcasibility of these pruning methods applicd to object ro-identification (ReID) tasks, while also presenting more comprehensive data pruning approach. By fully leveraging the logit history during training, our approach offers a more accurate and comprehensive metric for quantifying sample importance, as well as correcting mislabeled samples and rec- ognizing outliers. Furthermore, our approach is highly efficient, reducing the cost of impor- tance score estimation by 10 times compared to existing methods. Our approach and-play, architecturo-agnostie framework that can comminate/reduce 35%, 30%, and 5% of Mumples/traine time on the VeRi, MSMT17 and Market datasets, respectively, with negligible loss in accuracy (< 0.1%). The lists of important, mislabeled, and outlier samples from these ReID datasets are available at https://githubb.com /iiY wttp/ddaaaaapnn-reeee
plug-
1 Introduction
Object re-identification (ReID) is computer vision task that aims to identify the same object across multiple images. RelD has wide range of applications in security surveillance (Khan et al., 2021), robotics (Wenge- feld al., 2016), and human-computer interaction (Wang et al., 2019) among others. Similar several other downstream computer vision applications, the performance of ReID algorithms is contingent upon the quality of data. ReID datasets constructed by first collecting pool of bounding boxes containing the

---

Published in Transactions on Machine Learning Research (04/2024)
Standard data pruning approach
a (msittple) models
Rvaluste simple importance
srratch on the primed darzeet
Performance eraluzticn
Importont samples
Raw dataset
(a) A standard workflow of data pruning.
Images data method
O ur data pruning approach
Eveluste sample importazce
impoctasi semples
Les Inogee sumples
Identify mistabeeed samples
wrog
lew epochs
imagos
samples
culliees
Outhees
Redbiy label and
(b) Our comprehensive data pruning approach.
Ac) example of data pruning approach.
(a) The workflow data pruning. (b) Our data pruning approach not only identifies less important samples, but also rectifies mislabeled samples and removes outliers (boxes highlighted in turquoise). (c) demonstrates an example of our method, where all images are from the same person (id 630, MSMT17 dataset). Our approach serves as "pre-processing" step, redueing the dataset size to save storage and training costs of RelD models while having minimal impact their accuracy.
Figure 1:
intended object (e.g, people, vehicles); these images are typically obtained by applying category-specife object detectors to raw videos (Zheng et al., 2015; Fu et al., 2022). Then, the bounding boxes are assigned identity labels by manual annotators. ReID datasets created by this process generally present two issues:
Less informative samples: owing to the mostly fixed position of cameras, extracted bounding boxes from a sequence of images can contain redundant information, as these bounding boxes are likely to have the same background, lighting condition, and similar actions, e.g., Fig. 1(c). Such less informative samples from RelD datasets do not provide additional value to model training, Instead, they increase the training time leading to lowered training efficiency (Toneva et al., 2018; Feldman & Zhang, 2020; Paul et al., 2021; Sorscher et al., 2022).
2.
Noisy labels and outlier samples: RelD) datasets, typically containing large number of people or vehicles with distinct appearances that require manual labeling, often suffer from noisy labels and outlier samples due to human annotation errors, ambiguities in images, and low-quality bounding boxes (Yu et al., 2019; Yuan et al., 2020). Such noisy samples and outliers can reduce inter-class separability in RelD models, thereby reducing their accuracy (Yu et al., 2019).
The issue of less informative samples not limited ReID datasets. Several methods (Toneva et al., 2018: Feldman & Zhang, 2020; Paul al., 2021; Sorscher al., 2022) have shown that significant portion of training samples in standard image classification benchmarks (Krizhevsky et al., 2009; Deng et al., 2009) be pruned without affecting test accuracy. To this end, various data pruning approaches (Sener & Savarese, 2017; Ducoffe & Procioso, 2018; Paul et al., 2021; Yang et al., 2022; Sorscher et al., 2022) have been proposed to remove unnecessary samples. A standard data pruning workflow is depicted in Fig. 1(a): 1) One or multiple models are trained on a raw dataset. 2) After training for some or all epochs, the importance score of cach sample estimated based on metrics, c.g., EL2N (Paul al., 2021) L. norm of error, forgetting score (Toneva al., 2018) the number of times sample has been forgotten during the training process. Generally, the more difficult the sample, the more important it is (Toneva al., 2018:
2

---

Published in Transactions on Machine Learning Research (04/2024)
Paul et al., 2021; Yang et al., 2022). 3) The samples with low importance scores are considered easy less informative samples and are discarded (Toneva et al., 201 8; Feldman & Zhang, 2020; Paul et al., 2021). Steps 1 to 3 constitute a complete data pruming approach. 4) As final step, we train the model from scratch on the pruned datasct and evaluate its performance. Despite these methods contributing significantly to finding less informative samples, they have been applied primarily to classification tasks. To the best of our knowledge, no existing studies have explored the application of these methods in ReID tasks, and their effectiveness in this new context still needs to be determined. Our work is dedicated bridging the gap in applying (general) data pruning methods to ReID datasets.
Furthermore, the majority of these data pruning methods quantify the importance of samples at a specific training step (Paul et al., 2021; Yang et al., 2022; Sorscher et al., 2022) and tend to overlook the sample's behavior during the training, i.e., how the logits (un-normalized class probability scores) vary and evolve across epochs during training. We illustrate such a scenario in Fig. 2 and observe that the EL,N score (Paul et al., 2021) cannot diffcrentiate between two hard samples, i.c., (b) and (c), as it relies solely on the model's prediction at the last training epoch, discarding the sample's training history. On the other hand, the forgetting score (Toneva et al., 2018) estimates the importance score by recording how many times the model forgets this sample, i.c., how many times a sample experiences transition from being classified correctly to incorrectly during training. However, the forgetting score is still coarse grasse,r as it exclusively focuses on tracking the number of forgetting events and does not leverage the full training information.
A further limitation of standard data pruning techniques is that they primarily emphasize the removal of easy samples and do not explicitly discard noisy samples or rectify mislabeled samples (Toneva et al., 2018; Paul et al., 2021; Sorscher et al., 2022). However, mislabeled samples and outliers are commonly found in RelD datasets (Yu et al., 2019; Yuan et al., 2020), which lead to performance degradation.
To address the limitations of these standard methods, we propose a comprehensive data pruming approach and apply it to ReID tasks. Our method not only prunes less informative samples more efficiently, but also corrects mislabeled samples and removes outliers in an end-to-end framework, as shown in Fig. 1(b). We hypothesize that instead of quantifying sample importance using discrete events (Toneva et al., 2018) or coarse inter-class relations (Paul al., 2021; Sorscher al., 2022) during training, fully leveraging the logit trajectory offers more insights. Accordingly, we propose novel data pruning metric by fully utilizing the logit trajectory during training. Let us summarize our contributions:
1. To the best of our knowledge, we are the first to investigate the effectiveness of standard data pruming methods applied to RelD) tasks in reducing dataset size while maintaining accuracy.
2. We propose a novel pruning metric that utilizes soft labels generated over the course of training to estimate robust and accurate importance score for each sample. Our method also demonstrates significant efficiency, reducing the cost of importance score estimation by a factor of ten compared to existing state-of-the-art methods.
3. Our proposed framework not only removes less informative samples but also corrects mislabeled samples and eliminates outliers.
4. We benchmark our comprehensive data pruning approach on three standard ReID datasets. It eliminatess/edduces 35, 30,, and 5% of comples/traininng time on VeRi, MSMT17 and Market1501 dataset, respectively, with negligible loss in accuracy (< 0.1%).
2
Related Works
Data Pruning aims to remove superffuous samples in dataset and achieve comparable results when trained on all samples (Toneva et al., 2018; Feldman & Zhang, 2020; Paul et al., 2021; Sorscher et al., 2022). Different to dynamic sample selection (Chen el. al., 2017; Hermans el al., 2017) and online hard example mining (Shrivastava et al., 2016), which emphasize on informative sample selection for a mini-batch at each iteration, (static) data pruning aims to remove rodundnn or less informative samples from the datasct in one shot. Such methods can be placed in the broader context of identifying coresets, i.e., compact and informative data, subsets that approximate the original dataset. K-Center (Sener & Savarese, 2017) and
3

---

Published in Transactions on Machine Learning Research (04/2024)
supervised prototypes (Sorscher et al., 2022) employ geometry-based pruning methods, which estimate the importance of a sample by measuring its distance to other samples or class centers in the feature space. The forgetting score (Toneva et al., 2018) tracks the number of "forgetting events", as illustrated in Fig. 2 (c). GraNd and EL2N scores (Paul et al., 2021) propose to estimate the sample importance using gradient norm and error L,2 norm with respect to the cross-entropy loss. The main limitations of these approaches are: (i) their importance estimations may be highly affected by randomness due to underestimation of the influence of the whole training history, leading to insufficient accuracy; and (ii) the significantly high computational overhead for the metric estimation required by these methods. For instance, the forgetting score (Toneva et al., 2018) and supervised prototype score estimation (Sorscher et al., 2022) require an additional number of training epochs equivalent to the whole training time.
Object Re-identification (ReID) aims at identifying specific object instances across different viewpoints based on their appearance (Zheng et al., 2016; Yadav & Vishwakarma, 2020; Zahra et al., 2022). ReID datasets (Zheng et al., 2015; Liu et al., 2016; Wei et al., 2018) consist of images of the object instances from multiple viewpoints typically extracted from image sequencos. Therefore, images from the same sequence can often exhibit high similarity and contain redundant information i.e., the same background, consistent lighting conditions, and mostly indistinguishable poses. Furthermore, due to the subjective nature of manual annotation, RelD datasets often suffer from label noise and outliers (Yu et al., 2019; Yuan et al., 2020), such as heavily occluded objects and multi-target coexistence. During training, ReID tasks share similarity with the image classification task - given a datasel with a finite number of classes or identities, and models are trained to differentiate these distinct identities (Zheng et al., 2016; Yadav & Vishwakarma, 2020). As a result, the majority of RelD methods adopt the classification loss as an essential component (Wu et al., 2019; Si et al., 2019; Quispe & Pedrini, 2019; He et al., 2020). Owing to such similarity, several existing data pruning methods (Toneva et al., 2018; Paul et al., 2021; Sorscher et al., 2022) designed for image classification can directly be applied to RelD. As first step, we apply existing pruning methods (Toneva et al., 2018; Paul et al., 2021) designed for the image classification task to ReID tasks. We observe that a portion of RelD datasets can be pruned without any noticeable loss in accuracy. Next, we propose a novel approach that offers improvements in terms of importance score estimation and computational overhead over existing methods. To the best of our knowledge, our work is the first to comprehensively study the efficiency of data pruning applied RelD tasks.
3 Identifying Important Samples
3.1 Preliminaries
Our work focuses on supervised RoID. Similar to image classification (Zeng ct al., 2014; Rawat & Wang, 2017), a standard ReID network includes a feature extraction backbone, c.g., CNN-based (He et al., 2016) or transformer-based model (Dosovitskiy et al., 2021), followed by a classification head (Song et al., 2019; Liao & Shao, 2022). The backbone network extracts representations from images of persons (Wei et al., 2018) or vehicles (Liu et al., 2016). The classification head, typically fully connected layer, maps these extracted features to class labels, which in ReID) refer to the individuals or vehicles in the respective dataset. The train- ing of ReID models varies with respect to classification counterparts in the type of loss functions. In addition to the standard cross-entropy loss, ReID training often involves metric losses, e.g., triplet loss (Schroff et al., 2015) and center loss (MacDonald, 2013), to learn more discriminative features: LREID = LCE + aLmmetrie: where a a hyper-parameter to balance the loss terms.
3.2 Motivation
Typically, data pruning approaches estimate the sample importance after training for several or all epochs. However, the majority of data pruning meth do not fully exploit or take into account the training dynamics in evaluating sample importance. We observe that underestimating the training dynamics can lead to less accurate and unreliable estimations of the sample's importance. Figure 2 illustrates the trajectories of logits for three samples: (a) a simple sample, (b) hard sample, and (c) a 'harder' sample. Let us consider a classification task with three distinct classes, and each logit trajectory corresponds to one class. The difficulty
4

---

Published in Transactions on Machine Learning Research (04/2024)
Logfito resigred ID
of asagned
Logh of asaggeed II)
Ooger
Max other lagit
7.1
Max odher odhelogit logit logst
6.7
Max oder odarlogit
logt
6.7
Furgetting evert
Logit
6
Logit
3.1
3.1
0.4
0.4
12
Training Epoch
Trarinng Spoch
Triring Epoch
(a) Easy sample.
(b) Hard sample.
(c) Harder sample.
Forgetting score: 0
Forgetting score: 0
Forgetting score: 2
EL2N score: 0.01
EL2N score: 0.04
EL2N score: 0.04
Ours: H 0.09
Ours: H- = 0.30
Ours: H = 0.66
Figure 2: Logit trajectories for three samples (i.e., the evolution of the log probabilities of each sample belonging to each class over the course of training): (a) easy sample, (b) hard sample and (c) harder sample. There are three classes total and each logit trajectory corresponds to one of such classes. The difference between the logits for each class can reflect the level of difficulty of this sample. In general, the more difficult the sample, the more important it is. The forgetting scores cannot distinguish (a) and (b), while the EL2N score relies solely the model's prediction at the last epoch (thus without considering the history or "training dynamics"), hence it cannot differentiate between (b) and (c). Our approach fully exploits the training dynamics of a sample by utilizing the average logits values over all epochs to generate more robust soft label. Then, the entropy of this soft label is employed to summarize the importance of the sample.
of sample can be inferred by analyzing the discrepancy in logit values among different classes. Generally, the more difficult the sample, the more important it is (Toneva et al., 2018; Paul et al., 2021; Yang et al., 2022). EL2N (Paul ct al., 2021) and other geometry-based methods (Sorscher et al., 2022: Yang et al., 2022) utilize the output of a trained model after a few or all training epochs to evaluate the importance of samples. However, a comprehensive importance evaluation cannot be achieved solely based on the model's output a single epoch. As shown in Fig. 2 (b) and (c), although the last output logits of two hard samples are equal, the difficulties of learning them should not be equal: the model can correctly classify sample (b) from the early stage of training, while it struggles significantly and even mis-classifies sample (c) before the final training stage. Consequently, sample (c) in Fig. 2 should be considered more difficult or important. Inspired by this observation, we fully exploit the complete logit trajectories in order to achiove a more robust importance estimation, which we verify empirically. To be specific, for each sample, we leverage the complete logit. trajectory to generate a soft label, which captures the relationship between a sample and all target classes. The entropy of the sample's 's soft label can represent the degree of confusion between different classes. A sample with high entropy could be wavering between different classes fuzzy boundary sample, thus making it more difficult or important.
3.3 Methodology
Let (x, E Dtrain be an image-label pair and (x) Re be the logit vector of dimension e at epoch t, i.e., the un-normalized output of the classification head (pre-softmax). Here refers to the number of classes in the training set. A logit value, presented on logarithmic scale, represents the network's predicted probability of particular class (Anderson et al., 1988), with large values indicating high class probability. We first calculate the average logits of a sample over all training epochs T. We then apply Softmax a(.) to generate the soft label y of this sample, which portrays the relationship between this sample and all classes,
T
1
y =a a
2 (xx)
E ER..
(1)
T
This soft label ý summarizes the model's predictions at each training epoch. We conjecture that this soft label can effectively and accurately depict the ground truth characteristies of sample. Next, quantify
5

---

Published in Transactions on Machine Learning Research (04/2024)
Easy sample
Hard sample irnowr--
Mislabeled sample
Images fram Bio
Multi-target consistence
18:33
4.11
(99
Ove sortes
Aell we
Lis
AU4
6:34
6:37
61: 1066
10: 663
10: 6M4
ID:532
ED:72
(a) Easy sample: H 0.24
(b) Hard sample: H = 4.52
(c) Mislabeled sample: H = 5.18
(d) Outlier: = 5.91
Figure 3: Illustration of the generated soft labels averaged over 12 training epochs for different sample types (d) and their entropies. Multi-target coexistence (d) is one such type of outlier. We show the top 3 identities. Notably, our soft label can accurately indicate the ground-truth label of the mislabeled sample (c) without being influenced by the erroncous label. Images are from Market1501 dataset.
the importance score of a sample using the soft label entropy,
H(y) Σ ps loga Ps:
where y (PI..... ........................ PT) and the total number of classes. The importance of sample can generally be evaluated by the difficulty of this sample (Sener & Savareso, 2017; Toneva et al., 2018; Paul et al., 2021). The entropy is a straight metric to quantify the difficulty of sample: A sample with small entropy value implies that it contains the typical characteristics of single class, and the classifier this sample. In contrast, samples with larger entropy values could have different characteristies of more than one class at the same time (e-g., samples near classification boundaries), thus making it more difficult or important. Figure 3 illustrates our generated soft labels and the corresponding entropy of several typical samples. Please refer to Appendix E for more examples.
easily distinguish
The Area Under Margin (AUM) (Pleiss et al., 2020) also leverages training dynamics, but for the iden- tification of noisy samples in dataset. The AUM is defined the difference between the logit values for sample's assigned class and its highest non-assigned class and averaged over several epochs, i.e., AUM(X, y) = + E'= (x) maxi 2 2 (x)) where 2 (x) corresponds to logit value class i at epoch t. This scalar metric primarily captures the knowledge related to the assigned class while ignoring the information across non-assigned classes. However, knowledge from non-assigned classes can potentially be of greater importance than that from the assigned class, demonstrated in (Zhao et al., 2022). Different from AUM, our method utilizes information from all classes to depict a comprehensive relationship between sample and the label space. Additionally, as the AUM metric does not fully encapsulate the characteristics of the sample, it is only capable of identifying mislabeled samples, but it cannot correct them. In the next section, we elaborate further on our approach for label correction and outlier detection.
4 Data Purification
Although most samples with high importance scores contribute significantly to model training, some high- scoring samples may be noisy (Feldman & Zhang, 2020; Paul et al,, 2021) and degrade model performance. To address this limitation, in this section we further exploit the generated soft label to "purify" samples in two ways: by correcting mislabeled samples and by eliminating outliers.
Dataset Noise. Noise RelD datasets can be broadly categorized into (Yu et al., 2019): (i) label noise (i.c., mislabeled samples) caused by human annotator errors, and (ii) data outliers caused by object detector errors (e.g., heavy occlusion, multi-target coexistence, and object truncation; see examples in Appendix C). Our method aims to reuse mislabeled samples by label correction and eliminating outliers.

---

Published in Transactions on Machine Learning Research (04/2024)
Correcting Mislabeled Samples. We hypothesize that the soft label generated from the average logits of sample can reflect the ground-truth characteristics of this sample accurately. Accordingly, we directly utilize soft label y from Eq. to identify whether a sample is mislabeled or not. When the assigned label differs from the class with the highest score in the soft label, i.., argmax(}) * y, the sample is considered as mislabeled. The ground-truth label of this sample is then accordingly corrected to argmax(()). Indeed, our ablation experiments offer empirical evidence that accumulated logits over epochs lead to much more accurate prediction of the ground-truth label than solely relying on logits from single epoch (more details in Sec. 5.6).
Identifying Outliers. The entropy of the soft label reflects the difficulty of sample, allowing us to infer whether it is an outlier or not. A straightforward approach is to set a threshold on the entropy: if the entropy of the sample surpasses the predefined threshold, the sample is likely an outlier. However, theoretically, such an entropy threshold should be changed depending on the number of classes: the upper bound of the entropy is determined by the number of classes, i.c., ($) S logz(c), rendering unsuitable for use across datasets with different numbers of classes. Therefore, we employ the highest class score as an indicator to determine outliers. If the highest class score of a soft label is relatively low, ie., max(ş) S 8, it is highly likely that the sample is an outlier. We conduct sonsitivity analysis to evaluate the impact of the threshold 8 in Sec. 5.6 and observe our method is robust to this choice.
5 Experiments
We demonstrate the effectiveness of our method via detailed experimentation. In Sec. 5.3, we evaluate our pruning metric on ReID and classification datascts, and verify that it effectively quantifies the sample imporlance. In Sec. 5.4, we train models on noisy datasets after correcting labels and removing outliers lo validate the data purification ability. In Sec. 5.5, we present the results of our comprehensive data pruning method, which includes removing less important samples, correcting mislabeled samples, and climinating outliers. Finally, we perform several detailed ablation studies in Sec. 5.6 to evaluate the impact of hyper- parameters and verify the importance of logit, accumulation.
5.1 Datasets and Evaluation
Datasets. We evaluate our method across three standard ReID benchmarks: Market1501 (Zheng et al., 2015) and MSMT17 (Wei al., 2018) for pedestrian RelD), and Velti-776 (Liu al., 2016) for vehicle RelD, Market1501 is a popular ReID dataset consisting of 32, 668 images featuring 1, 501 pedestrians, captured across six cameras. Compared to Market1501, MSMT17 is more challenging and large-scale person RelD dataset (Wei et al., 2018), comprising 126, 441 images from 4, 101 pedestrians. VeRi-776 is widely used vehicle RelD benchmark with a diverse range of viewpoints for each vehicle; it contains 49, 357 images of 776 vehicles from 20 cameras.
Evaluation. Given a query image and a set of gallery images, the ReID model is required to retrieve a ranked list of gallery images that best matches the query image. Based on this ranked list, we evaluate RelD models using the following metrics: the cumulative matching characteristies (CMC) at rank-1 and mean average precision (mAP), which are the most commonly used evaluation metrics for RelD tasks (Bedagkar- Gala & Shah, 2014; Ye et al., 2021; Zheng et al., 2016). As an evaluation metric, we adopt the mean of rankl accuracy and mAP (He et al., 2020), i.e. 1
2 [rank]] + mAP), to present the results in a single plot.
5.2 Implementation Details
Estimating the Sample Importance Score. For RelD tasks, we follow the training procedure in (Luo et al., 2019): ResNet50 (He et al., 2016) pretrained on ImageNet (Deng et al., 2009) is used as the backbone and trained on a RelD dataset for 12 epochs to estimate sample importance. For oplimization, we use Adam (Kingma& & Ba, 2014) with a learning rate of 3.5x10-4, We do not apply any warm-up strategy or weight decay. The batch size is set to 64. Following Luo et al. (2019), We use a combination of the cross-entropy loss and the triplet loss as the objective function. During training, we record the logit values of each sample after each forward pass. Then, we generate the soft label of each sample based Eq. 1 and
7

---

Published in Transactions on Machine Learning Research (04/2024)
Table 1: Extra training epochs and time needed for metric estimation on a RelD dataset. The total training epochs for a ReID model is 120.
Method
EL2 models)
Forgetting score
Supervised prototypes
Ours
Extra training epochs
240
120
120 2.6 hours
12 15.8 mins
Extra training time'
5.2 hours
2.6 hours
calculate its entropy as the importance score. For classification tasks, we use the model architecture and training parameters from Toneva et al. (2018) for experiments on CIFAR100 dataset and from Boudial et al. (2020) CUB-200-2011 dataset. In contrast to the Rell tasks, we do not employ any pre-trained models in classification tasks, i.c., the model weights are randomly initialized. Please refer to Appendix A and B for more implementttion details.
Data Pruning. Given the importance scores all samples, we sort them to obtain a sample ordering from easy (low importance score) to difficult (high importance score). Next, given the pruning rate, we remove the corresponding number of easy samples from the original training set. For instance, if the pruning rate is 10%, remove the top 10% of the casiest samples. Subsequently, following Luo et al. (2019), we train four RelD models from scratch report their mean accuracy. The total number of training iterations is linearly proportional to the number of samples. Therefore, a reduction in sample size leads to a proportionate decrease in training time.
this pruned dataset independently, each with different random seed, and
5.3 Find Important Samples
This section aims to verify the effectiveness of our proposed importance score by data pruning experiments. Because our work predominantly concentrates on data pruning applied to the RelD tasks, we demonstrate the efficacy of our proposed importance score on three ReID datasets Sec. 5.3.1. Considering the substantial similarity ween RelD and classification tasks, we also extend our data pruning experiments to CIFAR-100 and CUB-200-201 datasets in Sec. 5.3.2, thereby ensuring more comprehensive validation of our method.
5.3.1 Data Pruning on RelD Datasets
We observe that the majority of data pruning methods have been evaluated on standard image classification datasets. However, to date, pruning metric has been evaluated RelD datasets. To this end, we perform systematic evaluation of three standard pruning metrics on RelD datasets: EL2N score (Paul et al., 2021), forgetting score (Toneva et al., 2018), supervised prototypes (Sorscher et al., 2022). Following the standard protocol (Toneva et al., 2018; Sorscher et al., 2022), the forgetting scores and supervised prototype scores are calculated at the end training. The EL2N score (Paul et al., 2021) and our proposed metric are estimated early in training (i.e., after 12 epochs which is 10% of the total training epochs and recommended by the standard EL2N protocol). For EL2N, we report two scores: one generated by single trained model and the other one by 20 trained models, each trained with diffcrent random seed. Training epochs for the importance score estimation of slandard data pruning methods are shown in Tab. 1. In addition to standard data pruning methods, we also conducted additional comparisons with RoCL (Zhou et al., 2020), which is a curriculum learning method designed for learning with noisy labels but also utilizes training dynamics to select samples. To fairly compare with it, we utilize its core idea of moving average loss as the pruning metric to realize static data pruning- The training sets are constructed by pruning different fractions of the lowest-scored samples. In all experiments, training on the full dataset (no pruning) and a random subset of the corresponding size (random pruning) are used as baselines (Sorscher et al., 2022). We present the results of data pruning approaches on three different RelD datasets in Fig. 4.
Better Pruning Metric by Leveraging the Training Dynamics. The results in Fig. 4 validate our assumption that fully exploiting training dynamics can lead to a more precise and comprehensive evaluation of the sample importance. Our method surpasses all competing methods across all datasets. The importance
Extra training time is measured using a single NVIDIA V100 GPU MSMT17 dataset.
8

---

Published in Transactions on Machine Learning Research (04/2024)
VeRi + ResNet50
MSMT17 ResNet50
Market1501 ResNet50
Rand + +6660
87.0
Aug acouaayy (85 (Hark] mal00
63.5
-90.5
88.5
63.0
62.5
89.5
86.0
praning
62.0
Fan pranting
praming sid)
89.0
random pruning
pruniigg
mondomn pruning
forgetting score
$1.5
fongctinng score
68.5
Torgetting score
EL2N (1 model)
ELZN (1 mope@
EL2N (1 modell
(20 models)
61.0
medets)
EL2N (20 models)
84.5
supervised protetypes
60.5
moking avg
supervised protonyces
67.5
87.0
moving avg loss our pruning method
s-periised protetypes
B4.0
our prining mathod
moving wog loss
60.0
our pruning method
10
20 30
40
10
15
20
25
30
25 Fraction of Dataset Pruned (%)
10 15
20
30
Fraction of Dataset Pruned (%)
Fraction of Dataset Pruned (96)
Figure 4: Data pruning on ReID datasets. We report the mean of Rankl and mAP on 3 ReID datasets (labeled at the top), obtained by training the pruned datasets. For each method, carry out four independent runs with different random seeds and report the mean.
of incorporating the training dynamics is also observed in the forgetting score and moving avg. loss. Both methods outperform single model EL2N and surpass the random pruning bascline at low pruning rates on all datasets.
EL2N Score Suffers from Randomness. We observe that EL2N (1 model) under-performs all three datasets (Fig. 4), being even worse than random pruning. The primary reason is that the EL2N score approximates the gradient norm of a sample at a certain epoch, and it assumes that the importance score of a sample remains static throughout the training process, ignoring the impact of evolving training processes. Disregarding the training dynamics leads to unreliable estimations that are highly susceptible to random variations. By training multiple models (20 models) and using the mean E2LN score, the performance improves significantly. However, it comes at the price of a notable growth in training time: it necessitates 20 times the training time required by our method.
Differences in Datasets. We notice considerable variations in the levels of sample redundancy across the three ReID datasets (Fig. 4). Using our method only allows for the removal of 5% of samples from Market 1501 without compromising accuracy. While on MSMT and VeRi datasets our method can climinate 15% and 30% of the samples, respectively. We hypothesize this is due to the rigorous annotation process of Market1501 dataset. Additionally, Market1501 is considerably smaller in scale, ie., 1arket1501 training set is merely 43% the size of MSMT17 and 34% the size of VeRi, thereby reducing the likelihood of redundant or easy samples.
Generalization from ResNet50 to Deeper ResNet and VIT. Previous experiments have demonstrated that our proposed metric can better quantify the importance of samples. Next, we validate if the ordering (ranking) obtained using a simpler architecture, ie., ResNet50 (He et al., 2016), can also work with a more complex architecture, ie., deeper ResNet and ViT (Dosovitskiy et al., 2021). To this end, we train a ResNet101 and vision transformer VIT-B/16 16 on a pruned dataset using the sample ordering from ResNet50. We plot generalization performance of ResNet101 and ViT in Fig, 5. We observed that the sample ordering obtained using ResNet50 is still applicable to ResNet101 and ViT. For instance, using ResNet101, 15% of MSMT17 samples can be removed, whereas with ViT, 30% of the VeRi samples can be climinated without any reduction in accuracy. These results verify that our metric can reflect the ground-truth characteristics of samples, independent of the model architecture used for training.
5.3.2 Data Pruning on Classification Dataset
To comprehensively verify the effectiveness of our proposed metric, we conduct supplementaty experiments on two classification datasets. We demonstrate the efficacy of our method on CIFAR-100 (Krizhevsky et al., 2009) and PUB-200-2011 (Wah et al., 2011) datasets and compare our method with the forgetting (Toneva et al., 2018) and EL2N (Paul al., 2021) scores in Fig. 6. For a fair comparison, all methods employ the same training time cost to estimate the importance scores of samples. In detail, following the standard protocol
9

---

Published in Transactions on Machine Learning Research (04/2024)
MSMT17 ResNet101
Market1501 + ResNet101
VeRi + ResNet101
200 Wctuee (S6 (bandt: +-0Mth12
(Bank) moM0i
91.00
03.0
64.5
90.50
64.0
30.00
(9
8664
63.5
8662
89.75
pruning
pruning
pruning
twnsom pruning
89.50
rendom praning
random pruning
62.5
0u1 praring method
out pruning mathod
855.8
our proning mathod
15
20
25
5
10
15
20
25
30
10
20
30
60
Fraction of Dataset Pruned (%)
Fraction of Dataset Pruned (%)
Fraction of Dataset Pruned (%)
MSMT17 + VIT-B/16
Market1501 + VIT-B/16
VeRi + VIT-B/16
Avg arcuney (M (tanet mAM2
71.5
90.0
87.4
90.6
87.2
71.0
(S)) (Band
90.4
87.0
70.5
90.2
66.8
30.0
86.4
pruning
pruning
00 pruning
rondom cm pruning
rondom aruning
random pruning
69.5
001 praning
our praning method
pruning method
1o
15
20
86.0
20
Fraction of Dateset Pruned (%)
Fraction Dataset Pruned (96
Fraction of Dataset Pruned (%)
Figure 5: Generalization performance. We train a ResNet101 and a ViT model using the sample ordering of ResNet50 on the MSMT17 dataset. For each method, we carry out four independent runs with different random seeds and we report the mean values. Shaded areas mean +/- one standard deviation of four runs.
CIFAR100
CUB-200-2011
77
Accurity (96)
Roal Test Accumsy
76
68
75
no pruning
pruning
random pruning
random pruning
forgetting model)
forgetting (1 model)
EL2N model)
EL model)
our pruning method
our pruning method
15
25
Fraction of Dataset Pruned (%)
Fraction of Dataset Pruned (%)
Figure 6: Data pruning on classification datascts. We report the mean of final test accuracy on two clas- sification datasets (labeled at the top), obtained by training on the pruned datasets. For each method, we carry out four independent runs with different random seeds and report the mean.
of EL2N (Paul al., 2021) we solely train model for 20 epochs CIFAR-100 and 3 epochs CUB-200- 2011, equivalent to 10% of all training epochs, and then apply three methods, i.c., forgetting, EL2N and our scores to estimate the sample importance. In line with the results in ReID tasks, our method consistently demonstrates superior performance over two competing methods. Notably, we intentionally exclude pre- training methodologies for both classification experiments; specifically, the model weights are randomly initialized. These results confirm the consistent efficacy of our approach, regardless of the utilization of pre-trained models.
5.4 Robust Training on Noisy Datasets
We test the efficacy of our data purification method on random noisy datascts. Herein, we start with the original images with the assigned labels. certain percentage (i.e., from 10% 50%) of training images are randomly selected and intentionally assigned the wrong labels. We compare approach with state-of-
10

---

Published in Transactions on Machine Learning Research (04/2024)
the-art method for identifying mislabeled samples, i.c., AUM (Pleiss et al., 2020), and evaluate each of our used components, i.e., label correction and removing outliers.
Market1501 + ResNet50
From Fig. 7, we observe that label correction plays a pivotal role, while merely removing outliers only slightly improves the perfor- mance. The reason that outlier removal merely discards a few data points, leaving behind a substantial amount of mislabeled data, which ultimately results in minimal model improvement. Besides, we ob- served these two components interdependent and complementary to each other, and the simultancous utilization of them significantly boosts the performance.
Dog. acoures (gi (Rankd mapi2
80
70
60
50
basaline
4.0
AUM
oues: label correction
20
removal nutliers
20
(bit)) removing correcting
30
Random Noise Ratio (%)
Our proposed data purification approach, removing and correcting samples, demonstrates superior performance compared to the AUM metric (Pleiss al., 2020). The rationale behind this advantage lies in the fact that the AUM metric tends to discard greater amount of method selectively removes fewer samples
Figure 7: Data purification on noisy datasets. We report accuracy under different random noise ratios. Each curve represents the mean accuracy over four independent runs.
training data, whereas and capitalizes on mislabeled data by employing label correction to
maximize the benefit of each sample.
5.5 Putting It All Together
By integrating data purification we build comprehensive data pruning approach. Samples are removed or rectified according to the sequence below: 1) all outliers (i.e,, samples whose highest class score is lower than 10%) are removed; 2) all samples with incorrect labels are rectified; 3) easy samples (i.e., samples whose soft labels have low entropy) are pruned. Please note that the total number of pruned samples equals the sum of all outliers and the pruned easy samples (i.e., all outliers are removed). We test our approach on three RelD datasets and evaluate the benefit of our data purification approach. In Fig. 8, observe marked performance differences between datasets. Data purification on MSMT17 shows notable effect, potentially due to the presence of more outliers and mislabeled samples in this dataset. The performance on VeRti is slightly improved as well. However, the impact on Market 1501 is comparatively limited, plausibly owing to stricter annotation protocols and fewer outliers compared to the other datasets. In sunmary, the seamless integration of data purification and data pruning can effectively boost overall performance. Especially, on MSMT17 our approach even eliminate/reduce 30% of comples//rrininn time with almost. compromise in performance.
MSMT17 + ResNet50
Market1501 + ResNet50
VeRi + ResNet50
Arg. (ecuragy/81 iRand mm2222
90.5
Anr (S) m.0022
87.2
90.0
86.8
86.6
89.5
86.4
pruning s00)
no pruning [=] 22
no pruning (al s00)
1 randumptin pruring
89.0
landom pruhing
86.2
ferdom pruning
our pruning method
our pruning method
out prining method
pruning method data purficcation
method with puréeation
methed with data pur fication
85.8
Fraction Dataset Pruned
Fraction Dataset (%)
Fraction Dataset Pruned (%)
Figure 8: Data pruning with data purification. Accuracy is achieved by training on the pruned dataset (pruning ratios on the X-axis). We report the mean values from four independent runs with different seeds.
5.6 Sensitivity Analysis and Ablation Studies
Our approach involves two hyper- paceoppprametess:: (1) the number of training epochs required to generate the soft. labels, i.e., T from Eq. 1, and (ii) the threshold o to remove outliers. We first explore the impact of the
11

---

Published in Transactions on Machine Learning Research (04/2024)
hyper-parameter T. Next, verify the importance of logit accumulation through ablation experiments, and finally analyze the effect of the threshold d on the performance of data purification.
Impact of Score Computation Epoch T. We investigate how early in training our metric is effective at estimating the importance scores of samples. In Fig. 9, we compare the model performance from training on 85% training data but pruned based on the for- getting score (Toneva. et al., 2018), EL.2N score (Paul et al., 2021) and our scores computed at different epochs. All scores are esti- mated using the identical random seed. We observe that after 12 epochs, the model performance of using our pruning method essen- tially stabilizes and surpasses that of random pruning well as other competing methods. Considering EL2N's vulnerability to random- ness, a single model is often insufficient to precisely estimate sample importance. The forgetting score typically necessitates many train- ing epochs (Toneva et al., 2018), thereby hindering its capability to estimate sample importance in early training stages accurately. Ad- ditionally, we explore the impact of score computation epoch T on label correction and present results in Appendix D.1. We observe training for 12 epochs is sufficient lo achieve desirable results of label
MSMT17 ResNet50
Aog. accurssy (88: Bhank moant
02.5
62.0
61.5
pruming
forgetting score model
610
E12N model]
our pruning method
60.5
10 15
20
25
30
Score Computation Epoch
Figure 9: Impact of score computation epoch T. Model performance achieved by training on 85% training data com- prised of examples with maximum for- getting, EL2N, and our scores com- puted at different epochs. correction as well.
Importance of Logit Accumulation. We conduct two experiments to demonstrate the importance of logit. accumulation in (i) importance score ion for data pruning, and (ii) label correction on noisy datasets. In both experiments, our soft labels are generated using different frequencies of logit accumulation, i.c., once at the 12th epoch and every 1-6 epochs. For example, "every 4 epochs" " means we use logits at the 4th, 8th, and 12th epochs to calculate our proposed importance scores. Likewise, "every 6 epochs" means logits are accumulated at the 6th and 12th epoch. Figure 10 depicts the results of data pruning experiment, and observe the progressive improvement in pruning performance with increasing frequency of logit accumulation. Consistent with this result, Figure 11 illustrates that as more logits at different epochs are accumulated, the performance of label correction improves. Both results validate our claim that the soft label generated using the full logit trajectory can better capture the ground-truth characteristics of a sample.
MSMT17 ResNet50
Market1501 ResNet50
AUg. accuracy Pha:: (hanna mo4222
63.0
02.5
42.0
n2 pruning (21 sk5)
no noise
bascline
only
AVg accuracy
only
0.0
ablution: every spech
ablation every 5 spocha
abiistin: every 4 epach
actatton eveyy epochs
abill kin every 2 spach
ablation: every 2 spochs
spach
spach
10 Fraction of Dalasel Pruned (%6)
30
30
Random Noise Ratio (%)
Figure 10: Ablation study of logit ac- cumulation for dala pruning. Model performance when trained with the importance scores computed using dif- ferent frequencies of logit accumula- tion.
Figure 11: Ablation study of logit cumulation on noisy dalasels. Model performance only using label correc- tion and the soft labels are generated using different frequencies of logit ac- cunulation.
Impact of Outlier Removal Threshold 8. In our method, samples whose highest class score lower than 8 are marked as outliers and removed (Sec. 4). We perform the ablation study on Market1501 dataset to quantify the impact of 8 under varying levels of noise. We keep the noise ratios fixed at 10%, 20% and
12

---

Published in Transactions on Machine Learning Research (04/2024)
Random noise ratio: 10%
Random noise ratio: 20%
Random noise ratio: 30%
Avg. accuracy (hox (Hami) m142022
90
(Ramk 0A2222
90
05
80
70
80
70
accuracy
60
50
no nolse
notce
10 nolse
baselire
30
baseine
Threshold for removal
Threshold outierr
Threshold removal (6)
Figure 12: Impact of outlier removal threshold 6. Model performance under noise ratio 10%, 20% and 30%.
30% while varying 8. As observed in Fig. 12, setting 8 10% yiclds slightly superior performance. Overall, 5 has limited impact on the final performance, especially when ò lies between 5% and 15%. It demonstrates that our approach is robust and insensitive to the threshold values.
6 Discussion and Conclusion
In this work, we have addressed two issues of RelD datasets (less informative samples and noise) by proposing plug-and-play architecture-agnosti data pruning framework. Firstly, by leveraging the training dynamics, we have provided a more accurate and robust pruning metric with extremely low computational overhead. In the generalization test, we have demonstrated that our metric reflects the ground-truth characteristics of samples independently of the model architecture used for training, ie., the sample ordering (ranking) obtained via a ResNet remains effective training a vision transformer. For completeness, we have also verified method on image classification datasets, observing that our approach exhibits superior per- formance over competing methods as well. Secondly, have proposed an efficient data purification method, enabling the correction of mislabeled samples and the removal of outliers. Experiments on noisy datasets validate that our data purification method exhibits robustness, achieving remarkable results. Finally, by integrating data purification we have built a comprehensive data pruning framework and demonstrated that it achieves state-of-the-art performance when pruning RelD datasets, allowing for the removal of 35%, 30%, and 5% of samples from the VeRi, MSMT17, and Market1501 datasets respectively, thereby leading to an equivalent reduction in training time. Like previous works (Toneva al., 2018; Paul al., 2021), data pruning method requires labels. In the future, intend to investigate data pruning methods based on logit trajectories in an unsupervised setting.
References
Simon P Anderson, André De Palma, and J-F Thisse. A representative consumer theory of the logit model. International Economic Review, pp. 461-466, 1988.
Apurva Bedagkar-Gar Gala and Shishir K Shah. A survey of approaches and trends in person re-identification. Image and vision computing, 32(4):270-2)) 286, 2014.
Malik Boudiaf, Jérôme Rony, Imtiaz Masud Ziko, Eric Granger, Marco Pedersoli, Pablo Piantanida, and Ismail Ben Ayed. A unifying mutual information view of metric learning: cross-entropy vs. pairwise losses. In European Conference on Computer Vision, pp. 548 564. Springer, 2020.
Weihua Chen, Xiaotang Chen, Jianguo Zhang, and Kaiqi Huang. Beyond triplet loss: deep quadruplet network for person re-identification. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 403-412, 2017.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A largo-scale hicrarchical image database. In 2009 IEEE conference on computer vision and pottern recognition, pp. 248-255 Iece, 2009.
13

---

Published in Transactions on Machine Learning Research (04/2024)
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Un- terthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby, An image is worth 16x16 words: Transformers for image recognition at scale. JCLR, 2021.
Melanic Ducoffe and Frederic Precioso. Adversarial active learning for deep networks: margin based approach. Xixi preprinl orXiv: 188000909984, 2018.
Vitaly Feldman and Chiynan Zhang. What neural networks memorize and why: Discovering the long tail via inffuence estimation. Advances in Neural Information Processing Systems, 53:2881-289) 2020.
Dengpan Fu, Dongdong Chen, Hao Yang, Jianmin Bao, Lu Yuan, Lei Zhang, Houqiang Li, Fang Wen, and Dong Chen. Large-scale pre-training for person re-identification with noisy labels. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2476 2476-2486 2022.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pallern recognilion, pp. 770 778, 2016.
Lingxiao He, Xingyu Liao, Wu Liu, Xinchen Liu, Peng Cheng, and Tao Mei. Fastreid: A pytorch toolbox for general instance re-identification. arXiv preprint XrXi::2006.024 02631, 2020.
Alexander Hermans, Lucas Beyer, and Bastian Leibe. In defense of the triplet loss for person re-identification. arXiv preprint
2017.
Samee Ullah Khan, Tanveer Hussain, Amin Ullah, and Sung Wook Baik. Deep-reid: Deep features and autoencoder assisted image patching strategy for person re-identification in smart cities surveillance. Mul- timedia Tools and Applications, pp. 1-22, 2021.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. aiv preprinl arXiv:1412. 6980, 2014.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images, 2009, URL wtt.::/....ooooo..uuu.u.uu
features- 20w.uess--00--111 200--020099-R00 TR.
Shengeai Liao and Ling Shao. Graph sampling based deep metric learning for generalizable person re- identification. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7359 7368, 2022.
Xinchen Liu, Wu Liu, Tao Mei, and Huadong Ma. A deep learning- based approach to progressive vehicle re-identification for urban surveillance. In Computer Vision-ECCo 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part 11 14, pp. 869 884. Springer, 2016.
Hao Luo, Youzhi Gu, Xingyu Liao, Shenqi Lai, and Wei Jiang. Bag of tricks and strong baseline for deep person re-identification. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, June 2019.
Norman MacDonald. Time lags in biological models, volume 27. Springer Science & Business Media, 2013.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, bigh-performance deep learning library. In Advances in Newral Information Processing Systems 32, pp. 1024-8033 Curran Associates, Inc., 2019. URL http:// papers.neurips.ce.
high- htwp://pppiii...iiiooooooooornne
Mansheej Paul, Surya Ganguli, and Gintare Karolina Dziugaite. Deep learning on a data diet: Finding important examples early in training. Advances in Neural Information Processing Systems, 34:20596- 20607, 2021.
14

---

Published in Transactions on Machine Learning Research (04/2024)
Geoff Pleiss, Tianyi Zhang, Ethan Elenberg, and Kilian Q Weinberger. Identifying mislabeled data using the area under the margin ranking. Advances in Neural Information Processing Systems, 33:17044-17056, 2020.
Rodolfo Quispe and Helio Podrini. Improved person re-identification based on saliency and semantic parsing with deep neural network models. Image and Vision Computing, 92:103809, 2019.
Wascem Rawat and Zenghui Wang. Deep convolutional neural networks for image classification: A compre- hensive review. Neural compulation, 29(9):2352 2449, 2017.
Florian Schroff, Dmitry Kalenichenko, and James Philbin. Facenet: A unified embedding for face recognition and clustering. In Proceedings of the IEFE conference on computer vision and pattern recognition, pp- 815 823, 2015.
Ozan Sener and Silvio Savarese. Active learning for convolutional neural networks: A corc-set approach. arXiv preprint Xrv: 101.00..0000 00489 2017.
Abhinav Shrivastava, Abhinav Gupta, and Ross Girshick. Training region- object detectors with online hard example mining. In Proceedings of the IEEE conference on compuler vision and pollern recognilion, pp. 761-769, 2016.
Tongzhen Si, Zhong Zhang, and Shuang Liu. Compact triplet loss for person re-identification in camera sensor networks. Ad Hoe Networks, 95:101984, 2019.
Jifei Song, Yongxin Yang, Yi-Zhe Song, Tao Xiang, and Timothy M Hospedales. Generalizable person re-identification by invarian mapping network. In Proceedings of the IEEE/CVF conference on Computer Vision and Pattern Recognition, pp. 719-728, 2019.
Ben Sorscher, Robert Geirhos, Shashank Shekhar, Surya Ganguli, and Ari Morcos. Beyond neural scaling laws: beating power law scaling via data pruning. Advances in Neural Information Processing Systems, 35:19523 19536, 2022.
Mariya Toneva, Alessandro Sordoni, Remi Tachet des Combes, Adam Trischler, Yoshua Bengio, and Ge- offrey J Gordon. An empirical study of example forgetting during deep neural network learning. arXie preprint crXfc:1812.05159, 2018
Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The CalD birds-200-2011 dataset. 2011.
Yujiang Wang, Jie Shen, Stavros Petridis, and Maja Pantic. A real-time and unsupervised face re- identification system for hunan-robot interaction. Pallern Recognition Letlers, 128:559 568, 2019.
Longhui Wei, Shiliang Zhang, Wen Gao, and Qi Tian. Person transfer gan to bridge domain gap for person re-identification. Tn Proceedings of the IEEE conference on compuler vision and pallern recognition, pp. 79 88, 2018
Tim Wengefeld, Markus Eisenbach, Thanh Q Trinh, and Horst-Michacl Gross. May i be your personal coach? bringing together person tracking and visual re-identification on mobile robot. In Proceedings of ISR 2016: 47st International Symposium on Robolics, pp. 1-8. VDE, 2016.
Wu, Si-Jia Zheng, Wen-Zheng Bao, Xiao-Ping Zhang, Chang An Yuan, and De-Shuang Huang. novel deep model with multi-loss and efficient training for person re-identification. Newrocomputing, 324:69-75, 75, 2019.
Ankit Yadav and Dinesh Kumar Vishwakarma. Person re-identification using deep learning networks: A systematic review. XrXiv preprint arXiv:2012.133) 2020.
Shuo Yang, Zeke Xie, Hanyu Peng, Min Xu, Mingming Sun, and Ping Li. Dataset pruning: Reducing training data by examining generalivation influence. aix preprint arXin:2205.0933 2022.
15

---

Published in Transactions on Machine Learning Research (04/2024)
Mang Ye, Jianbing Shen, Gaojie Lin, Tao Xiang, Ling Shao, and Steven CH Hoi. Deep learning for person re-identification: A survey and outlook. IEFE transactions on paltern onalysis and machine intelligence, 44(6):2872 2893, 2021.
Tianyuan Yu, Da Li, Yongxin Yang, Timothy M Hospedales, and Tao Xiang. Robust person re-identification by modelling feature uncertainty. In Proceedings of the IEEE/CVF international conference computer vision, pp. 552 561, 2019.
Ye Yuan, Wuyang Chen, Yang Yang, and Zhangyang Wang. In defense of the triplct loss again: Learning robust person re-identiffeation with fast approximated triplet loss and label distillation. In Proceedings of the IEEE/CWE Conference Computer Vision and Pattern Recognition Workshops, pp. 354-355, 2020.
Asmat Zahra, Nazia Perwaiz, Muhammad Shahzad, and Muhammad Moazam Fraz. Person re-identification: A retrospective on domain specific open challenges and future trends. ar preprint arXiv:2202.131 131211 2022.
Daojian Zeng, Kang Liu, Siwei Lai, Guangyou Zhou, and Jun Zhao. Relation classification via convolutional deep neural network. In Proceedings of COLING 2014, the 25th international conference on computational linguistics: technical popers, pp. 2335-2344, 2014.
Borui Zhao, Quan Cui, Renjie Song, Yiyu Qiu, and Jiajun Liang, Decoupled knowledge distillation. In Proceedings of the IEEE/CVF Conference on computer vision and pattern recognition, pp. 11953 11962, 2022.
Liang Zheng, Liyue Shen, Lu Tian, Shengjin Wang, Jingdong Wang, and Qi Tian. Scalable person re- identification: A benchmark. In Computer Vision, IEEE International Conference on, 2015.
Liang Zheng, Yi Yang, and Alexander G Hauptmann. Person re-identiffcation: Past, present and future. arXiv preprint 1r:: 02984, 2016.
Tianyi Zhou, Shengjie Wang, and Jeff Bilmes. Robust curriculum learning: from clean label detection to noisy label self-correction. In International Conference on Learning Representations, 2020.
16

---

Published in Transactions on Machine Learning Research (04/2024)
A Implementation Details for RelD Tasks
A.1 Estimating the Importance Score of Samples from RelD datasets
All experiments are implemented in PyTorch (Paszke et al., 2019) using the FastRelD (He et al., 2020) toolbox on 4 NVIDIA Tesla V100 GPUs. A general workflow for supervised ReID is shown in Fig. 13. In our work, we follow the training procodure? of Luo et al. (2019). For feature extraction, we employ a 50 (He et al., 2016) pre-trained on ImageNet (Deng et al., 2009); the model is trained for 12 epochs. All images from M5rket1501 and MSMT17 are resized to 256 128 pixels, while images from VeRi are resized 256 x 256 pixels. During training, we record the logits for each sample after each forward pass. For optimization, we use combination of the cross-entropy loss and the triplet loss, i.e., Lerain Lee + Leriplet
Input Images
Features
FC Layers
Classification Loss
Backhone
Triplet Loss
Figure 13: A general supervised RelD workflow.
A.2 Existing Methods
We use the same model configuration and settings as in our approach, except for the loss function. When calculating the EL2N score, we strictly follow its procedure and definition (Paul et al., 2021), i.c., we only use the cross-entropy loss and do utilize additional loss functions (e-g., metric loss). We train till 12 epochs and then compute the EL2N score for each sample. For EL2N (20 models), run the same experiments 20 times but with different random seeds and obtain the score by averaging over the runs.
E2LN.
Forgetting Score. We train single model for 120 epochs using the default settings². During training, record the number of forgetting events for each sample.
Supervised Prototypes. To ensure a fair comparison with other methods, replace the self-supervised learning proposed in Sorscher et al. (2022) with supervised learning. Using the default settings and configuration", train single model for 120 epochs. After training, we calculate the supervised protolype score of each sample, which is defined by the L2 distance between sample and its class centroid (Sorscher et al., 2022). Additionally, following Sorscher et al. (2022), we also apply a simple 50% class balancing ratio.
A.3 Data Pruning Experiments RelD datasets
Following the default model configuration and settings?, train four independent RelD models using different random seeds on the pruned dataset, where the samples are pruned based different types of importance score, i.e., the forgetting, EL2N, supervised prototype and our scores. We then report the mean performance across these four models.
Phttps:/ htttpss//ghthub..omm/DDA. DDA.aa/////DDA--T///tttt-t-eelee CV/fast-reid/bbbl http://bbbb/asttt//o///i/////// Base- BaaB---aggtrikeee bagtricks.yml parameter setting.
Model configuration and hyper-
17

---

Published in Transactions on Machine Learning Research (04/2024)
A.4 Generalization from ResNet to ViT
We initialize vision transformer ViT-B/16 (Dosovitskly al., 2021) with its weight pre-trained on Ima- geNet21K and use the default configuration The model is trained a pruned dataset, which is pruned based the ranking list of samples generated by ResNet50. We report the average performance across four runs with different random seeds.
A.5 Robust Training Noisy Datasets
To evaluate the performance of AUM, we follow their proposal (Pleiss el al., 2020) to fine-tune the threshold value using threshold samples. For our method, we set the threshold of outlier removal to 10% on all three ReID datascts. After the outlier removal and label correction, train four ReID models using the default settings and present the mean accuracy derived from these four runs.
A.6 Putting It All Together
In our proposed framework, samples are either removed or rectified according to the sequence below:
1. Eliminating all outliers according to the highest class scores of the samples, as described in Sec. 4.
Removing easy samples, as described in Sec. 3.3.
Rectifying mislabeled samples. Details are presented in Sec. 4.
Please note that the total number of pruned samples equals the sum of all outliers and the pruned easy samples (i.e., all outliers are removed). For instance, if we set the threshold å 10% the noisy sample rate of the MSMT17 dataset is 1.1%. If 10% of the samples are removed, includes 1.1% outlier samples and 8.9% casy samples.
B Implementation Details for Classification Tasks
For image classification on CIFAR-100 and 200-220-2001 datasets, we begin with computing the forgetting score (Toneva et al., 2018), EL2N score (Paul et al., 2021), and our proposed metric for each sample. For a fair comparison, all methods employ the same training time or cost to estimate the importance scores of samples. For experiments on CIFAR-100, we use the model architocture and training parameters' specified in Toneva al. (2018), while the model architecture and training parameters specified in Boudial et al. (2020) are used for experiments on CUB-200 11B-200-2011. Following the standard protocol of Paul et al. (2021), solely train model for 20 epochs CIFAR-100 and 3 epochs CUB-200-2011, equivalent to 10% of all training epochs. After computing the individual metrics for each sample, we remove the corresponding number of easy samples from the original training set. For instance, the pruning rate is 20%, we remove the top 20% of the easiest samples (samples with the lowest importance score). Following Toneva et al. (2018), we train four models from scratch (i.e., weights are randomly initialized without the use of any pre-trained model) on this pruned dataset independently. For each run, we use different random seed. Finally, we report the mean test accuracy across these four independent runs.
C Examples of different outliers from MSMT17.
In Fig. 14 we present three different types of outliers from MSMT17, i.e., heavy occlusion, multi-target coexistence and object truncation
hhttee////taaalooom////cc/////// httpe///ihhuu..cmmmmmmmmww/a/aa t 5 https:///hhhbbcmm//eemmmmmmmmnn entropy
vit ViT configuration
18

---

Published in Transactions on Machine Learning Research (04/2024)
(a) Heavy occlusion
(b) Multi-targel coexistence
(c) Object truncation
Figure 14: Examples of different outliers from MSMT17.
Random noise ratio: 10%
Random noise ratio: 20%
Random noise ratio: 30%
50
Aug securcy Okn
no
baseine
base fne
baseline
ours (both)
20
ours (both)
ours (both)
10
10
10
25
Score Computation Epoch
Score Computation Epach
Score Computation Epach
Figure 15: Accuracy achieved by training on the Market1501 dataset under different noise ratios only using label correction, where the soft labels are generated at different epochs.
Ablation Study
D.1 Impact of Score Computation Epoch on Label Correction
Sec. 5.6, we investigate how early in training our metric is effective at estimating the importance scores of samples and observe that after 12 epochs, the importance scores of samples estimated by our method tend to be stabilized. However, it beckons the consideration: Are the generated soft. labels trained for 12 epochs sufficient achieve good performance in label correction? To answer this question, we explore the impact of score computation epoch T, which is the soft labels generation epoch as well, label correction. We present results in Fig. 15. Our sensitivity analysis reveals that the soft labels generated between the 8th 20 epochs demonstrate superior label correction capability, even approaching the model performance trained on the clean dataset. This observation clarifies our earlier question: Even in the relatively early stages of training (alter epochs), the soft labels we proposed still exhibit a notable capability of label correction.
Furthermore, observe that the label correction capability of soft labels generated after 20 epochs exhibits decline, particularly pronounced under high noise ratios such 30%. The underlying rationale is that, as the model undergoes prolonged training of mislabeled samples, leading to fitting these erroneous labels (Pleiss et al., 2020). Overall, our approach T, particularly within an acceptable range of noise ratios., c.g. 10%,
noisy dataset, the model starts memorizing a huge amount
exhibits robustness to 20%.
E Example Images
We present some samples from three RelD datasets and CIEAR-100 dataset sorted from easy to hard based on our proposed pruning metric. From Figs. to 19, we observe that the samples with the smallest importance scores tend be simple and are canonical representations of each class. In contrast, the samples with higher scores harder to identify: they have different backgrounds
suffer from occlusion truncation.
19

---

Published in Transactions on Machine Learning Research (04/2024)
Figure 16: Samples from 15rket1501 sorted from easy (the first column) to hard (the last column) based on our proposed importance scores. Images in each row belong the same identity.
20

---

Published in Transactions on Machine Learning Research (04/2024)
Figure 17: Samples from MSMT17 sorted from easy (the first column) to hard (the last column) based on our proposed importance scores. Images in each row belong to the same identity.
21

---

Published in Transactions on Machine Learning Research (04/2024)
Figure 18: Samples from VeRi sorted from easy (the first column) to hand (the last column) based our proposed importance scores. Images in each row belong to the same identity.
22

---

Published in Transactions on Machine Learning Research (04/2024)
Figure 19: Samples from CIFAR-100 sorted from easy (the first column) hard (the last column) based on our proposed importance scores. Images in each row belong the same identity.
23


Reviews:
[{"review_id": "YNiifPndXb", "reviewer": "Reviewer_8NyC", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\n\n- Dataset quality is a realistic concern for all learning based approaches. Methods tackling data pruning has many downstream applications.\n\n- Compared with previous works, this work jointly considers multiple factors in data pruning, including removing easy samples, correcting mislabels and eliminating outliers.\n\nWeakenss:\n\n- The proposed method is too naive. Correcting mislabels is simply achieved by resorting the pseudo label with the highest probability. This naive way of self-training is subject to confirmation bias [1] where the performance of network could be severely harmed by incorrect pseudo labels. Moreover, the pseudo label correction requires very careful calibration of model training procedure. Introducing the label correction too early may fail to fit the hard samples while too late could overfit to the label noise.\n\n- The outlier detection approach relies on a predefined threshold. This also introduces additional hyper-parameter to tune and there is no principled way introduced for this hyper-parameter tuning.\n\n- The description of pruning easy samples is not clear. A discussion of existing work AUM is brought in but the methodology section is expected to be self-contained. It is very hard to fully understand how this step is implemented in the current form.\n\n- Comparisons with state-of-the-art methods are missing. In particular, comparisons with learning from noisy labeled data and outlier detection methods are missing, which are the core contributions of this work.\n\n[1] Arazo E, Ortego D, Albert P, et al. Pseudo-labeling and confirmation bias in deep semi-supervised learning[C]//2020 International Joint Conference on Neural Networks (IJCNN). IEEE, 2020: 1-8.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This work makes the observation that not all training samples are equally import for ReID tasks. The existing works often prune out easy samples to speed up training procedure. Instead of focusing on the easy samples, this work proposed a comprehensive approach towards data pruning by taking into consideration the samples importance, correcting mislabels and outlier removal. Extensive results suggest the effectiveness with a large percentage of data pruning on ReID tasks.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- Additional comparisons with state-of-the-art methods are missing. In particular, comparisons with learning from noisy labeled data and outlier detection methods [1-3] are necessary.\n\n- Additional discussions of learning from noisy labeled data [4-5] is also necessary as the core contribution lies on learning from noisy labeled data and outlier detection. \n\n[1] Han B, Yao Q, Yu X, et al. Co-teaching: Robust training of deep neural networks with extremely noisy labels[J]. Advances in neural information processing systems, 2018, 31.\n[2] Ma X, Huang H, Wang Y, et al. Normalized loss functions for deep learning with noisy labels[C]//International conference on machine learning. PMLR, 2020: 6543-6553.\n[3] Li S, Xia X, Ge S, et al. Selective-supervised contrastive learning with noisy labels[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022: 316-325.\n[4] Song H, Kim M, Park D, et al. Learning from noisy labels with deep neural networks: A survey[J]. IEEE Transactions on Neural Networks and Learning Systems, 2022.\n[5] Wang H, Bah M J, Hammad M. Progress in outlier detection techniques: A survey[J]. Ieee Access, 2019, 7: 107964-108000.", "broader_impact_concerns": "There is no ethical concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "P1SD2w7MRe", "reviewer": "Reviewer_26U1", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The idea of using global training information is solid.\n2. The proposed method is clearly described.\n\nWeaknesses:\n1. Although this paper focuses on data pruning for object re-id, it is not clear why the proposed method is specifically designed for object re-id.\n2. In Sec.5.5 the authors use both easy sample and outlier pruning together. I wonder what exact strategy is used for these two methods. Does 10% prune rate mean deleting 5% easy samples and 5% outliers?\n3. In the proposed method the authors take $\\tilde{y}$ as ground truth pseudo labels which are used to justify whether a sample is mis-labeled. I wonder if there is any chance the wrong labels are easy to learn, which leads to $argmax(\\tilde{y})=y$ while it is still mis-labeled?\n4. It would be great if the authors could provide comparison among different model scales, such as ViT-S, ViT-B and ViT-L.\n5. The authors can also provide visualization of detected easy/outlier samples in CIFAR-10.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper mainly targets data pruning. The authors analyze the current data pruning methods and point out that these methods cannot handle noisy samples and ignore the full training information. To solve these problems, they propose a new method which records the pseudo labels trajectory during training and prunes data according to the confidence calculated by entropy of pseudo labels. The authors further propose several strategy to use their method to rectify mis-labeled samples and identify outliers. They conduct extensive experiments on both object re-id and image classification to prove the effectiveness of the proposed method.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please refer to the weaknesses and provide corresponding analysis and results.", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "z3Dd9vnEEX", "reviewer": "Reviewer_xbcP", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The proposed computation of soft labels as average logits over training epochs and the importance score as entropy of these soft labels is simple, intuitive and effective. It is also straightforward a metric as mentioned by the authors.\n2. Comprehensive experiment is one of the major strengths of the work. The approach shows its superiority on 3 re-id and one typical image classification dataset over a few state-of-the-art data pruning approaches. Leveraging the training dynamics helps not only to prune effectively for training in convnets but the pruned dataset is shown to be effective in ViT also. Another interesting experiment is label correction and outlier removal on noisy datasets. Sensitivity analysis on the hyperparameters $T, \\delta$ and frequency of logit accumulation are also worth mentioning.\n\nWeaknesses:\n1. The work looks related to coreset selection [a, b] and dataset distillation [c, d] in flavor (especially if I look at the goal of training with only a handful of important data). A discussion relating the proposed work to these types of works substantiating the similarities and differences would be good to have.\n2. Though the target scenario for the work is re-id, it would be good to apply the proposed approach to a largescale typical image classification dataset e.g., ImageNet or the fine grained classification dataset CUB-200. Though, experimental results show little superiority of the proposed approach on a small scale CIFAR-100 dataset, it would be good to have such experiments along with it a discussion on the approach’s success or failure as the case may be.\n3. Some minor typos:\n - Related work: First line – the word ‘as’ will not be there.\n - Section 4: “Dataset Noise” Last line: ‘eliminate’ -> ‘eliminating’\n - Section 5.3.1: “The disregard of training dynamics” -> “Disregarding the training dynamics”\n\n[a] K. Killamsetty, D. Sivasubramanian, G. Ramakrishnan, and R. Iyer, “Glister: Generalization based data subset selection for efficient and robust learning,”, AAAI, 2023.\n\n[b] B. Mirzasoleiman, J. Bilmes, and J. Leskovec, “Coresets for dataefficient training of machine learning models,” ICML, 2020.\n\n[c] T. Wang, J.-Y. Zhu, A. Torralba, and A. A. Efros, “Dataset distillation,” arXiv preprint arXiv:1811.10959, 2018.\n\n[d] J. Cui, R. Wang, S. Si, C.-J. Hsieh, “Scaling Up Dataset Distillation to ImageNet-1K with Constant Memory”, ICML 2023.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper addresses efficient training by discarding or pruning less important data from re-identification (re-id) datasets. The proposed approach, unlike previous approaches, not only prunes less informative data points, but also can discard outlier as well as rectify noisy labels. The approach is built on top of the intuition that a hard (informative) example remains a tough nut for most of its journey throughout the training. Mathematically, the authors measure this by taking average of the logits of such examples across different epochs and computing the entropy of this. This gives the importance score that forms the basis to categorize the example as hard or easy and also as noisy or not. The authors took re-id as the use case and showed the efficacy of the approach on re-id datasets along with well designed and executed ablation studies.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "The requested changes are mentioned in the weaknesses. I will be happy to change my rating on seeing these.", "broader_impact_concerns": "None.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "tT7mg5IoAe", "reviewer": "Reviewer_H1m6", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n1. The problem of data pruning in the RID task seems reasonable and its solution is useful in practice.\n\n2. The experiments show some promising results on three benchmarks under its model setting. \n\nConcerns:\n\n1. The connection between the proposed data pruning strategy and the RID task is weak. It lacks specific considerations for the RID task -- the entire design is based on the standard classification paradigm. As such, it is unclear why such a generic method is only evaluated on the RID task. \n\n2. Data selection is not a new problem for learning from data with noisy labels. The idea of using training dynamics have been explored before, which has not been discussed in the paper, for example:\n   Tianyi Zhou, Shengjie Wang, and Jeff Bilmes. Robust curriculum learning: from clean label detection to noisy label self-correction. In International Conference on Learning Representations, 2021.  \n\n3. The technical contribution is incremental: it is rather heuristic to use the averaged logits or its entropy for various data cleaning steps, and no theoretical analysis has been provided. What is the difference between the RID training and the standard classifier training in this problem setting? \n\n4. The experimental evaluation is less convincing in several aspects: \n    \n    a) As an empirically design strategy, more extensive experiments on different settings of backbone networks should be reported, such as deeper ResNet and larger ViT models, with different pre-training settings. \n\n    b) The efficiency achieved by such data pruning is underwhelming. At most 30% of the dataset can be removed. It should be compared with other few-shot or fine-tuning learning strategies regarding training efficiency. \n\n    c) The baseline methods are weak. As mentioned above, it is more convincing to compare this method to data selection strategies used in learning with noisy labels, which also ignore \"hard examples\" based on small loss heuristics.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper presents a data pruning strategy for training an object re-identification model. The proposed method leverages the logit trajectories of each training data instances and constructs a soft label based on the trajectory average. The soft label and its entropy are then used to perform data pruning, correcting incorrect labels or identifying outliers. The method is evaluated on three RID benchmarks using ResNet50/ViT-B/16 backbones.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. Clearer motivation for the RID task.\n2. Comparisons with previous work on data selection in training. \n3. As mentioned above, more comprehensive experimental evaluation.", "broader_impact_concerns": "No concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: **Strengths:**
1. **Realistic Problem Addressing:** The paper addresses the practical and significant issue of data pruning in the RID (Re-Identification) task, highlighting its applicability in real-world scenarios.
2. **Promising Experimental Results:** The experiments conducted demonstrate promising results across three benchmark datasets within the proposed model settings, showing the potential effectiveness of the methodology.
3. **Comprehensive Experimentation:** The paper presents a thorough experimental approach that tests the method's efficacy on multiple datasets, providing robust evidence of its superiority in specific contexts.
4. **Innovative Metric Use:** The utilization of soft labels as average logits and entropy as an importance score for labels is noted as simple, intuitive, and effective, contributing to the method's overall strength.

The method is simple, and yet effect. Two of reviewers vote the acceptance. Particularly, one concerned weakness is 
1. **Weak Connection to RID Task:** The proposed data pruning strategy's connection to the specific requirements of the RID task is weak, as it is largely based on a generic classification paradigm, making its specialized effectiveness for RID unclear.
2. **Typos and Minor Errors:** The paper contains various grammatical mistakes and typographical errors that could detract from its professional quality and clarity.
3. **Ambiguous Method Descriptions:** Descriptions of certain methodologies, such as pruning easy samples, are unclear and not self-contained, making it difficult for readers to grasp the implementation fully.

Please incorporate the rebuttal to improve the paper. Particularly, add some discussion in introduction or related work for Weakness 1 listed above. Overall, while the paper offers promising approaches and solid experimentation, it has incremental contributions that rely heavily on previous work.
Decision: Accept (poster)