Info-Coevolution: An Efficient Framework for Data Model Coevolution
Authors: Ziheng Qin
Hailun Xu
Wei Chee Yew
Qi Jia
Yang Luo
Kanchan Sarkar
Danhui Guan
Kai Wang
Yang You
Venue: ICML 2025 poster (2025)
Keywords: Dataset Efficiency; Annotation Efficiency;
Primary area: general_machine_learning
TL;DR: Efficient online selective annotation for both supervised and semi-supervised learning

Abstract: Machine learning relies heavily on data, yet the continuous growth of real-world data poses challenges for efficient dataset construction and training. A fundamental yet unsolved question is: given our current model and data, does a new data (sample/batch) need annotation/learning? Conventional approaches retain all available data, leading to non-optimal data and training efficiency. Active learning aims to reduce data redundancy by selecting a subset of samples to annotate, while it increases pipeline complexity and introduces bias. In this work, we propose Info-Coevolution, a novel framework that efficiently enables models and data to coevolve through online selective annotation with no bias. Leveraging task-specific models (and open-source models), it selectively annotates and integrates online and web data to improve datasets efficiently. For real-world datasets like ImageNet-1K, Info-Coevolution reduces annotation and training costs
by 32% without performance loss. It is able to automatically give the saving ratio without tuning the ratio. It can further reduce the annotation ratio to 50% with semi-supervised learning. We also explore retrieval-based dataset enhancement using unlabeled open-source data. Code is available at https://github.com/NUS-HPC-AI-Lab/Info-Coevolution/.

Info-Coevolution:
An Efficient Framework for Data Model Coevolution
Ziheng Qin 12 Hailun Xu 28
Wei Chee Yew?
Qi Jia
Yang Luo Yang You 't
Kanchan Sarkar
Danhui Guan 2
Kai Wang
Abstract
paradigms for large-scale dataset construction fall into two categories: (1) collecting and annotating data for fully super- vised training, (2) sourcing data from the web, followed by automated cleaning and
Machine learning relies heavily on data, yet the continuous growth of real-world data poses chal- lenges for efficient dataset construction and train- ing. A fundamental yet unsolved question is: given our current model and data, does a new data (sample/batch) need annotation/learning? Con- ventional approaches retain all available data, leading to non-optimal data and training effi- ciency. Active learning aims to reduce data re- dundancy by selecting subset of samples to an- notate, while it increases pipeline complexity and introduces bias. In this work, we propose Info- Coevolution, novel framework that efficiently enables models and data to coevolve through on- line selective annotation with no bias. Leverag- ing task specific models (and open-source mod- els), it selectively annotates and integrates on- line and web data to improve datasets efficiently. For real-world datasets like ImageNet-1 I K, Info- Coevolution reduces annotation and training costs by 32% without performance loss. able to automatically give the saving ratio without tun- ing the ratio. can further reduce the annota- tion ratio to 50% with semi-supervised learning. We also explore retrieval-based dataset enhance- ment using unlabeled open-source data. Code is available at http:///jitubbcoo///U---H--e Lab/Info-Coevolution.
training, and subsequently fine-tuning models on down- stream tasks with supervised data. However, both ap- proaches incur significant costs for both annotation and training, making them viable only for few well-resoureed institutions. As result, the construction of large-scale datasets remains insufficiently democratized and decen- tralized process.
Before 2010, popular datasets such as MNIST (Deng. 2012) and CIFAR-10/100 (Krizhevsky et al., a;b) each contained 60,000 hand-annotated low-resolution images, reflecting the modest scale of early machine learning datasets. The lack of data limited the development speed of machine learning at that time. In 2010, ImageNet- (Deng et al., 2009) was publicly released with more than IM human annotated sam- ples. It was the first large-scale supervised dataset, leading to the development of many modern deep-learning algo- rithms and architectures, boosting the development of AI research today. An estimated cost of building ImageNetI-K is at least 0.4 million USD for labeling the total 1.28m im- ages, and 4 million for 14m ImageNet- 12k, excluding other costs. With cloud GPU today, training a deep learning neural network on ImageNet-IK IK with A100 GPU takes about $240 to $1400, while labeling an ImageNet- scale (million-scale) data still takes more than 100 times this cost, ranging from $24,000 to more than $1m depending on annotators' degree of proficiency. Constructing supervised data is usually more expensive compared to training with them.
1. Introduction
Besides the supervised datasets, other types of datasets emerge later with corresponding training schemes. Bert (Devlin et al., 2019) and GPT (Brown et al., 2020) use large- scale unsupervised text corpus, such as BooksCorpus (Zhu, 2015) and Common Crawl (Raffel et al., 2020), to train with CL.IP
Machine learning is inherently data-driven and has nu- merous practical applications. Currently, the dominant
Project lead Core contribution National University of Singapore, School of Computing, 11 Research Link, Sin- gapore *ByteDance Ltd, Singapore. Conresoondence to: Kanchan Sarkar
masked language moddling/next--oken--reeeeee (Radford et al., 2021) scale the training on weakly super- vised (web-sourced image-text pair) data with cross-modal contrastive learning. Segment Anything (Kirillov et al., 2023) utilizes models and manpower together to increase the data scale and quality of segmentation data iteratively, These different types of data have various quality and cost.
Ziheng Qin <zihengggccomp.nnu www.woowww...uuueeeee <kanchan.cc wwacocaannnsrarreeee wewedanceeedoonnnnnnee com.> Danhui Guan <guandan- hui bytedance.com>>. Yang You <youy @ syouu@comp..nt comp.nus.eduu edw.c.d.sss......
Proceedings of the 42nd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Info-Coevolution: An Efficient Framework for Data Model Coevolution
Efficiency of Supervision ImageNet Accuracy
90
IN1K
IN1K
Finetune:
80
MAE Pretraining 2688 V100 hour
CLIP VIT-L/14+
70
-57000
73728 V100 hour
Supervised
-$18K
ImageNet Accuracy
60-
$500
CLIP ResNet50
50
CLIP VIT-L/14
CLIP
40
Supervised Training
2 -,0000
ResNet-50
MAE Finetune
Data Efficiency
30
IN1K Label Cost $192K-400K
Laion-400M
20
510-15K
WIT-400M
10
$100K
10 -3
10
10
10'
10
of images processed (millions)
Figure 1. The data scaling curve of different types of data. Supervised data has higher data efficiency than weak supervised data on specific downstream tasks (e.g. ImageNet), while incurring higher collection and annotation cost.
As shown in Iig.l, regarding the dask-speciffc/doowstrean against data amount) and data construction cost, supervised data > weakly supervised data > unsupervised/self- supeeviise data. The data scale and training cost are in a reversed order, where supervised << The
training with rounds of annotation, which makes the pipeline too complex for real-world applications. What's 's more, many active learning methods rely solely on the model's prediction for the sample's 's usefulness. It is prone to sample distribu- tion problems on harder tasks. Their design of re-training + re-inference limits its application to large-scale data. Due to these limitations, active learning is not yet an efficient and robust enough solution for real-world scenarios. Coreset selection methods (Guo al., 2022), the other hand, skip the model update and leverage the sample distribution to se- lect samples used for training. Therefore, it fails to leverage the model-specific information unless a model trained on fully annotated data is obtained initially.
data
weakly supervised < mnsupervised/sel--uprrvvii three paradigms provide different tradeoff's between data collection/amooatioon and training costs. In general, higher degree of supervision provides better data efficiency for downstream tasks with higher data cost per unit. Previously works like BLIP(Li et al., 2022) and Segment Anything (Kirillov et al., 2023) tried to boost the data quality with human+model annotation, which is somehow equivalent to improving the degree of supervision.
To address these challenges, we propose info-Coevolution. novel and efficient online framework for selective data collection/amnotation that integrates model-specifie estima- tion with distribution awareness. info-Coevolution enhances data annotation efficiency with minimal computational over- head. Leveraging Bayesian principles and our analysis of data locality, we propose a Mlta-based information gain es- timation strategy and Bayesian Prediction Fusion. This ap- proach improves the spocci- sample selection while reducing the need for frequent model updates during the selection process. The framework begins with an unsuper- vised pretrained backhone and a small, randomly initialized dataset, mimicking real-world scenarios. By utilizing an on- line Approximate Nearest Neighbor (ANN) structure, such as HNSW (Malkov Yashunin, 2018), Info-Coevolution achieves logarithmic scaling for information gain estimation. enabling efficiently handle growing data and increasing selective annotation.
In online and downstream real-world scenarios, data con- tinuously grows over time. The scale of unsupervised data could be immense, but the marginal gains from training diminish rapidly. Boosting its scale is not cost-efficient. Weakly supervised data involves a smaller overall volume with higher training efficiency per unit of data. In contrast, supervised data generally has the smallest total volume but incurs the highest annotation costs, leading to the most sig- nificant performance improvements for the same amount of data. As data grows in online scenarios, and usually the annotation cost is higher than training, it would be beneficial to improve the annotation efficiency and exlore the marginal benefits of supervised data.
To boost the annotation/sample efficiency, active learning was proposed, which aims select samples that better bene- fit training to annotate. However, many of the current active learning methods(Li al., 2024) involve frequent model
2

---

Info-Cocvolution: An Efficient Framework for Data Model Coevolution
On ImageNet-IK, Info-Coevolution achieves lossless perfor- mance with only 68% of the annotation cost, through just 4 rounds of continual supervised training (sum to <100% full training cost; training from scratch gets the same lossless result). The computational overhead primarily arises from model inference after each training round, totaling approxi- mately A A100 GPU hours for the entire process. Addition- ally, we developed an efficient batch sampling mechanism, allowing the selection process to be completed within minute on datasets at million-scale. Furthermore, Info- Coevolution provides both qualitative and quantitative esti- mates of a data sample's ' information gain, conditioned on the model and the previously collected data. This feature en- ables automatic stopping when performance gains plateau, eliminating the need for additional sample annotations to verify saturation.
learning, conducts full annotation each round with differ- ent quality and re-train the model for each round an increased amount/portion of data. Different to these works, our method targets an efficient online selective data collec- tion/annotation process, with far less cost in the loop. An illustration of the difference is shown in F
Acquisition Function
Estimated Gain
Select
New Annotation
Online Update
Online Update
Data's View (Online updalable)
Initialize
2. Related Works
Continual Model SudaeeUpdeeee
Extendable
Coreset Selection Methods Coreset selection methods fo- cus on filtering out low-quality or redundant samples, while reserving the most representative ones in the target dataset. The core of these approaches lies in the elaborate selection criteria, including geometry-based (Sener & Savarese, 2017; Sinha et al., 2020), error-based (Toneva et al., 2018; Paul et al., 2021). deccisio-bbuundary--ee based (Ducoffe& & Precioso, (Coleman et al., 2019), , gradient- matching (Mirzasolciman et al., 2020; Killamsetty et al., 2021a), bilevel optimization (Killamsetty et al., 2021b) and submodularity-
Model
Dataset
With new Data/Annotation
Online
OOttiona/Loowfree
New Data
Active Learning
Our Differences
Figure 2. A comparison of our method's difference to active learn- ing. Generally, active learning loops over 1. selecting samples, 2. annotating, 3. updating the model, and selecting samples again (with the updated model). contrast, our method doesn't 't have to update the model frequently. We can update the model optionally with continual supervised learning at a much lower frequency (so the training cost also low). And our algorithm is automatically class- balanced due to distribution awareness, therefore it doesn't tt suffer distribution shift active learning.
2018; Margatina et al., 2021), uncertainty-
methods (lyer et al., 2021; Zhou et al., 2023). Some of the approaches (Sener & Savarese, 2018) also combined the strength of active learning to make further improvements. In general, coreset selection methods do not force a strict constraint on full data/label access.
3. From the Perspective of Information Gain
Active Learning and Semi-Supervised Learning Active learning (Hino, 2020; Smith et al., 2023; Li et al., 2024) and semi-supervised learning (Sohn et al., 2020; Zhang et al., 2022; Wang et al., 2023; Cai et al., 2022) are two comple- mentary strategies for reducing the reliance on large-scale scale supervised datasets in machine learning. Active learning focuses on iteratively selecting the most informative sam- ples for annotation, enabling models to achieve higher per- formance with fewer labeled examples by optimizing the data-labeling process. On the other hand, semi-supervised learning leverages large pool of unlabeled data alongside a smaller labeled subset, using techniques such as pseudo- labeling, consistency regularization, or generative models to propagate label information and improve generalization. To- gether, these paradigms aim to maximize model efficiency and performance in data-scarce scenarios, often bridging the gap between fully supervised learning and real-world constraints. Segmenta anything, in way similar to active
In this section, we first derive from the theoretical aspect of how to estimate the information gain of sample to a dataset. Previously, an exact information gain was mainly defined upon an attribute of a decision tree. Here we extend it to more generalized form and provide a way to estimate it on broader tasks, e.g. a sample's ss information gain with respect to image classification (datasets). We analyze the influence of locality in the space to improve both the estima- tion quality and efficiency. Then we formulate the selective data selection/annotatioo problem as a recursive online prob- lem, each time adding one (batch of) sample/amotation to an already collected dataset. Based on this problem setup and our theoretical analysis, we propose our algorithm Info- Coevolution for efficient online selective annotation and dataset construction.
Theorem 3.1. For neural networks that generate repre- sentation and then make the prediction (for model M = g o 8), at certain distances e from a sample point x in
3

---

Info-Cocvolution: An Efficient Framework for Data Model Coevolution
the feature representation space, the similarity of sam- ples in feature space also leads similar model predic- tions (Vō, 3e s.t.a, x', g(f(x))) 6).
could have different certainty/confidence, should take the confidence (probability of prediction being true) into consideration of the interpolation.
(eft,,,,111(") 1I((2)) f(x)] S e (9(((x))
3.2. Confidence of label
We show proof of Theorem 3.1 with both 1.2 distance and cosine distance in Appendix. This theorem implies a kind of linearity among near-neighbor samples' predictions, and explains one of the reasons that KNN predictions work. We will leverage this theorem to design a parametrized estimation of a sample's information gain with locality ex- tendability, and efficiency.
By defining confidence for the prediction of sample, we integrate this value into the interpolation process as weighted NN prediction:
En simemm(e,))2 Sim(n 2)yx
Yestimation(2) =
(4)
En, Lz,sim(s,a)2 Og Sim(2z z)
where Yx the probability prediction vector of x's anno- tation. Then the entropy of point from data's view can be calculated based on it. Moreover, the semi-supervised learning's 's label and unlabeled datasets can be transferred into a continuous space defined on confidence a E [0, 1]. and we can estimate the confidence of an annotator and use models to perform annotation. An easy mapping in probability space can directly map accuracy to confidence: a (ace acCrandoco / acCrandom). One primary goal of semi-supervised learning under this interpretation could be making the a value converge while training the model with corresponding confidence without collapse.
3.1. Information carried by sample the dataset
In predictions with decision tree T, information gain IG with condition A is defined as:
A) H(T))
which is the reduced entropy given condition A. Here, we extend this information gain to broader tasks. Assuming the samples we are curious about (target distribution) is from distribution p. which is either uniform in the whole distribution space or IID to train data, then the information {2z, Zy to dataset sampled
gain of adding a sample from/ p
3.3. Bayesian prediction fusion
IG(D, (DDD) H(D =)
Therefore, one key question in semi-supervised learning is interpreted as value convergence. Suppose we have two independent predictors with confidence aj and 022, when the two predictors have same prediction y for sample x, the confidence value og of sample x should be annotated with y become
= 211 (H(x)= Hz)lz))p(a)) dr,
or IG(D, = H(D)- - H(D + 2)
Predictions are true, 0gy P( Predictions match
IID) = E
Σ
(M(x)-H(a))))
D
On 02
(5)
(3)
01 * 02 + Plmatch with other label)
Where E E I is the distance threshold we decide to use, and d E [0. 1] is similarity threshold. This distance threshold is inherited from 1, where two sample's prediction has low correlation beyond certain distance. During estimation, the H here is estimated on parameterized Ma.
In classification tasks, P(match with other label)
[0, (1 (1--)1)(1-00) az)] depending on distribution over classes. In the binary classification case P(malch with other) (1 0n a1)(1 -22), it requires smaller prediction confidence is larger than 0.5 so that the sample confidence is increased: min(a1, > 0.5 + Ozy > max((a, a2); if assuming the matching distribution is uniform over e classes, then P(match with other) = (1 - a1)( a2)/(c 1) and min(a), 02) > 1/c is the condition. no matching prob distribution is in other classes, then its value is O. To sum up, using (1 an (1 a2) gives lower bound estimation Gay- These conditions can be utilized to make the al- pha value converge, re-estimate value for samples in a dataset.
and the || is defined within space of f.
Then a question is how to estimate (x(22) given x + 2. As assumed, in certain distances the linearity dominates so that we can do interpolation and linear combination on logits/predictions. Within this space where linearity holds, interpolation between 2 and where V: y/ also gives the same label prediction, but with an increased certainty. Interpolation between and x where by # I, will split the prediction over the two, which needs careful calcula- tion. However, the interpolation itself is an estimation based on linearity assumption. As different samples' predictions
When predictions from different views diverge, if one is con- fident above the threshold and the other not, use the max

---

Info-Cocvolution: An Efficient Framework for Data Model Coevolution
Sample to estimate
Oeevieww How we get a fused prediction
Near Neighbor Samples Collected
(P,C.S)
(P,C,S)
(P,C,S)
Predicted Class:
(P.C) model
Confidence: 0.9
Similarity: 0.93
(P.C,S)
Model's View (Prediction)
redicted Class:
(P.C.S)
Confidence: 0.9
Similarity: 0.91
(P,C,S)
distance
Data's View:
Bayesian rediction Fusion
Prediction= max; Confidence;
Σxeknn, in e,px=i(CxSX)
Confidence ΣxEknn,x in CxSx
(P, C
merged
Figure The overview how we estimate sample's confidence with Bayesian Fused prediction both model's view and data's view. Generally, this estimation gives better evaluation of sample's uncertainty/confidenee than using model prediction only.
for the highest confidence. If both predictions confi- dent, then the sample's confidence is reduced. Supposing
re-annotating sample, the expected gain is [|IG(annolate z)] = (mrin (knn predictions)) ( (Cann)) -H(Ha
Q1 >
then
(7)
P(p is Irue)
P(pi) P(predictions unmatch)
If using entropy. one can use H(a) = -alog(a) Σe-1 _ log L-n ~ -alog(a) - (1 (01--)log(1 ao)oog(11 a) as an estimation neglecting the constant term. In practice, we found using H(x) = a a feasible proxy to choose sam- ples with less computation and higher compatibility. For sample selection, only need to know the positivity and relative magnitude of the samples' information gain, there- fore the confidence itself is good proxy.
0n o2))
(6)
a1 a2) P(other)
where P(other) [1. 01) )1)) In the P2 takes all the remaining confidences from the first predic- tor, P(other) (1 a1 )1);; if assuming the match- ing distribution is uniform over classes, P(other) = (1-aa a1)a2/( - 1). there are no prob distribution un- matching classes on other classes, P(other) = 0. Similarly, using (oother) (1 - an )a² gives lower bound on the updated confidence. The prediction divergence from confi- dent predictions means mar(p1) mar(p²) = (p') max( H (p)), ), H (p2)). Reducing the corresponding confi- dence can help with re-convergence.
3.5. The Algorithm
Selective Annotation When data to be annotated is greater than the capacity, for sample = (zt), to estimate the gain of annotating it given already collected data D, we have model m and get the feature f(z), model prediction ym. con- 1 1/reum_class). We then retrieve nearest neighbors of z (with and ANN search) and use Eq.4 get (knn) data-based prediction Vknn- Then we merge two predictions from model and dataset with Eq.5 and Eq.6 and get Umergeed Cmerged- The annotation gain of sample Z is then calculated as Cann - Cmerged where Cann. is the average accuracy of the annotator. For batched selection, one can sample with prob- ability proportional the gain and drop redundant samples.
fidence 6n = (pm - 1/numm_class
3.4. Benefit of reannotating one sample
I/knns Cknn-
As the model is to absorb the data distribution, the dataset itself also carries a prediction based on the near neighbor pre- diction within the distance where linearity holds. And sam- ple's s confidence changes during turns iteration. There- fore, taking the derivation from previous sections, when
5

---

Info-Cocvolution: An Efficient Framework for Data Model Coevolution
Data scaling curves on ImageNet
Sample Newly Annotated
Near Neighbor With Same Prediction
85.5
85.6
85.5
85.6
85
Near Neighbor With Different Prediction
(P.C.S)
(P.C.S)
ImageNet Accuracy
Predicted Class: P
(P.Ct.))
Confideece: 0.9 - 0.81
Similarity: 0.93
(P.C.S)
76-
Prodicted Class: P P Confidence: 0.9 0.93
Similarity: 0.91
Random Subset Supervised
(P.C.S)
distance
Adding Urtabeled Data to Seml VIT
Semi-Vi Adding Annotation Ratio
Info-Coevvlution Supervized
New Annotation am.C
Cnve-Coev Semi-VIT
66
Dynamic Rechecking
Percent of images
Figure 5. The scaling curve of different schemes. Cofe-Coevolution improves data efficiency for both pppervisee/Semmi-suprrvisss set- tings, and gets lossless performance at 68% and 50% annotation ratios respectively.
Figure The idea dynamic rechecking. When we get new annotation(s), update the sample estimation within the e-distance to better reflect the gain. will automatically balance classes and sample density. This step is efficient with klog(n) time.
tension, either distributed HNSW or hierarchical recursive clustering plus cluster-wise HNSW is a feasible solution.
Dynamic Rechecking After getting new annotation for = (a:;) with confidence Cann. we retrieve k nearest neighbors of z. For neighbors within the distance threshold, we recalculate their data's 's view prediction and update their (P.C) with our Bayesian Prediction Fusion Eq.5 and Eq.6 and update their gain. Each time we select the (batch) sam- ple with the highest annotation gain to annotate. This design is to avoid redundancy and ensure class balance during se- lective annotation, as near samples with similar predictions will be updated with a lower gain, while near samples with different predictions will be updated with a higher gain. We use online-ANe ANN based near neighbour search so that this process is efficient and extendable.
On ImageNet- K, we use image embedding from ImageNet- IK samples retrieve the k nearest neighbor in the superset, and then de-redundant the samples while filtering retrieved samples with cosine distance larger than 0.2.
4. Experiments
4.1. Datasets and Implementation Details
We evaluate our method on ImageNet- (Deng et al., 2009), CIFAAR10/100KKizhevskyy et al., atb), StanfordCars, Food-101 al., 2014), SVHN(Netzer al., 2011) under different annotation ratios and settings. We further extend the training data with data from CC3M, CC12M, SBU, Visual Genome, COCO, and LAION-400m to study the effect of scaling unlabeled data.
3.6. Using Public Data to Enhance Downstream Task
To further benefit downstream tasks and evaluate our method in the data/annotation growth setting, we construct super- set from multiple open-sourced (and mostly web-sourced) datasets (SC3M(Sharma et al., 2018), CCI2M(Changpinyo et al., 2021), SBU(Ordonez et al., 2011), COco(Chen et al., 2015), Visual Genome(Krishna et al., 2016), Laion- 400m(Schuhmann et al., 2021)) and retrieve related sam- ples to enhance downstream tasks. To reduce the peak GPU/RAM memory, we subsample 1% samples from all the samples and retrieve their feature using BLIP(Li al., 2022). Then we use K-means to get 500 clusters. We then follow (Qin et al., 2024) to do ANN-based de-redundancy and filtering within each cluster and construct the corre- sponding ANN index for later retrieval. This allows a higher parallelism of the algorithms. For future data scale ex-
Implementation Details.
The experiments on ImageNet- IK follows Semi-ViT ViT (Cai al., 2022), using ViT (Doso- vitskiy al., 2021) model with MAE (He et al., 2021) pre- trained backbone to conduct supervised and semi-supervised training. All other details can be found in the Appendix.
4.2. Data/Annotation Scaling
ImageNet-1k Results We show improvement in data/annotation efficiency here in Fig.5 and Tab. On 10% data setting of ImageNet, our selective annotation can in- crease the accuracy by 1.3% compared to the random base- line. With only 68% annotated samples from ImageNet- Ik, we can achieve lossless performance (85.6% Acc), surpass-
6

---

Info-Coevolution: An Efficient Framework for Data Model Coevolution
ing the 85.5% Acc of Semi-ViT with 80% labeled data and 20% unlabeled data. What's more, have an auto- matic stop criterion when the marginal gain of annotating more samples is negligible. The 68% ratio is given by the algorithm itself when it suggests to stop selecting sam- ples, instead of a annotation ratio tuning. can be seen that Info-Coevolution is compatible with Semi-Supervised learning, where Semi-Vir ViT trained with 50% ImageNet- IK annotations selected by rnfo-Coevolution can achieve almost lossless result (85.5%). Moreover, info-Coevolution is compatible with continual supervised training, without introducing a distribution shift.
Table 2. Scaling the dataset with retrieved data from our con- structed superset and training with Semi-ViT. Extending the un- labeled data can improve performance while using our selection would improve the data efficiency of unlabeled data. Additional Data
Labeled Data
Acc
1.28M
0 Unlabeled 2M Our selected unlabeled 1M
85.6
1.28M
86.0
1.28M
86.0
Generalization and Robustness Theoretically, the effec- tiveness of Info-Coevolution is model-agnostic and dataset- agnostic. We here verify its generalization and robust- ness across different datasets, architectures and other semi- supervised framework.
Table 1. Accuracy of training ViT-Large on ImageNet- Ik with continually increasing annotation and doing continual training- Info-Coevolution gets a 1.3% accuracy improvement in 10% Im- ageNet supervised setting with the proposed prioritized selective annotation.
We present our experimental result on CIFAR10 semi- supervised training with Fixmatch(Sohn et al., 2020) and ResNet-50x4 in table 3. Info-Coevolution improves the data efficiency and further reduces the annotation amount to achieve lossless performance (95.85% acc) from 4000 to
Method
Setting
1%
3%
5%
7%
10%
Random
Supervised Supervised
67.1
72.5
74.6
76.6
79.2
Ours
73.1
75.8
78.0
80.5
1000.
Data scaling curves ImageNat
Table 3. ResNet-50x4 with Fixmatch on CIFAR10
85.6
85
Data Selection
250
1000
4000
Full
85
Random
94.95
95.59
95.85
95.85
ImageNat, IaageNatAAccuui Accuracy
Info-Exolution
95.39
95.85
80
75
To analyze the generalization of info-Cooevolution across different datasets, we use Info-Coevolution with both su- pervised training and semi-supervised training by adapting pretrained ViT-L on CIFAR-10, 10, CIFAR-100, Standfordcars, fool101 and SVHN. As shown in Fig.7, info-Coevolution consistently improved both supervised and semi-supervised training performance all these datasets.
70-
Random Subeet Supervised
Cocvolution Supervised
DQ (Selectio-onll)
65
80
Percent of images
Figure 6. Compare with corset-selection SOTA method Dataset Quantization on INIK.
4.3. Ablation Experiments
Comparison with Coreset Selection In Fig.6, we compare with previous SOTA method Dataset Quantization(Zhou et al., 2023). With selection-only (not adding the MAE reconstruction), DQ achieves 85.5% Acc with 80% la- beled data. It shows that beyond the extendability of Info- Coevolution, it can also serves as good coreset selection method.
Table 4. Ablation of proposed components in the framework (1% model to select 10% annotation).
Component
ImageNet Ace. 10% label
Model
Data Dynamic
78.5
79.8
Tab.2 illustrates the effect of data scaling for further scaling cemi-supervised training with unlabeled data. With addi- tional unlabeled data, the acc can be further increased by 0.4% with supiriupee training. For unlabeled data selection, we adjust the equation to better capture useful unlabeled data. See Appendix for detail. It use half of the unlabeled data to enhance the performance as using all unlabeled data.
77. 77.1 12.1
79.7
79.8
80.2
Random Select
79.2
Ablation of Components. As our algorithm involves fusing
the prediction of model and KNN predictions with dynamic
7

---

Info-Cocvolution: An Efficient Framework for Data Model Coevolution
Infe-Coevolution's improvement on (Semi-JSupervised Training on different Datasets
Supervised Info Supervised
Semi-Superte Senl-Supervised Info-Sentl
96.1
+1.0
+1.8
96.1
+0.6
+1.5
+1.
97.9 +0.7
+0.6
93.2
+0.6
+1.2
+2.7 +3,4
+0.5
Accuracy
85
+1.7
62.0
10% CIFAR-10
10% CIFAR-100
50% Standfordcars
50% food101
10% svhn
10% svhn
Figure 7. Coo-Coevvlutionn consistently improve the annotation efficiency across different datascts with both superviseddsemiisuupriisss setting.
rechecking, we here ablate their corresponding influence on performance in Tab.4. The experiment is to choose annota- tion for 10% ImageNet data with 1% random data trained model. It can be seen that using only model prediction for sample gain estimation could fail to beat the random baseline (78.5% compared to 79.2%). As analyzed, purely pondel-uncertainty-bane sample selection is unaware of distribution, which could lead to distribution bias in hard problems. Using our data's view prediction gives a better prediction of information gain and gets 0.6% performance improvement: the dynamic rechecking which introduced locality consideration effectively mitigate the problem of model-based sample selection, improving performance by +0.5% (compared to model-only -0.7%). Our proposed mul- tiview prediction fusion combines both model and KNN prediction better decide the samples with higher entropy. Using the fused prediction alone will get severe distribu- tion balance problem, which lower down the performance by 2.1%. With dynamic rechecking involved to balance both density and class balance, it gets a 1.0% ACC improvement compared to the random baseline.
The Influence Frequency of Model Update
Frequent Model Update
One Choice
80.5
80.2
Random Baseline
ImageNet Accuracy
Baseline:79.2
Percent of samples seen (ImageNet-1K)
Figure 8. Ablation of model update frequency. When updating the model more frequently during selection, it shows improved data efficiency.
4.4. Task Generalization
Beyond the traditional classification task, we further ex- tend Info-Cocvolution to the semantic segmentation task. We experimented on ADE20k(Zhou et al., 2017) with UperNe((BEiT-v2 (Xcoo al., 2018; Peng et al., 2022). The UperNet has a backbone (BEiT-v2-large) and UperHead. We use the backbone feature from the last layer (dim 1024*16*10 16) and mean pooling it to dim 1024. We adapt Info-Coevolution accordingly, using pixel-wise confidence average as model confidence, class-wise simi- larity weighted confidence as knn confidence, and gain 1- L-avg(en (com www.foiddnnccccocoooo fidence con fidencegnn). The result for using 1% fata-trained model select 10% data is shown in
Ablation of Model Updating Frequency. We also study the influence of model fine-tuning frequency annotation- that, when add additional
efficiency. In Fi model updates during the loop, Info-Coevolution can select a better annotation set at the annotation ratio. When directly selecting 10% (0.15 188M annotations for ImageNet-I k with model trained on 1 % data, the performance is 80.2%; after adding two model updates in the middle, we get 80.5%.
8

---

Info-Cocvolution: An Efficient Framework for Data Model Coevolution
Impact Statement
Table Comparison of the performance of selected 10% data
ADE20K. Our method provides better data annotation selection.
This paper presents research aimed at advancing the field of efficient training in the context of continuously growing datasets. We conducted experiments using publicly avail- able, widely-used datasets in machine learning, without in- corporating any new data. As such, our work poses no risk of harmful societal consequences. Instead, it contributes to society by enabling the development of better neural models at a lower cost.
aAcc
mIoU
mAcc
Random
82.19
46.89
58.72
Ours
12.84
48.39 11.50
60.81 (2.09
table 5. Our method achieves significantly better perfor- mance at the same data amount.
References
5. Conclusion
Bossard, L.,, Guillaumin, M., and Van Gool, L. Food-101 mining discriminative components with random forests. In Fleet, D., Pajdla, T., Schiele, B., and Tuytelaars, (eds.), Computer Vision ECCV 2014, pp. 446-461, Cham, 2014. Springer International Publishing. ISBN 778-3-319-10599-4.
In this work, we extend Information Gain estimation to classification tasks with locality consideration. We pro- posed novel online efficient algorithm Info-Coevolution, to increase the annotation efficiency of supervised/semi- supervised training, Info-Coevolution is able to save the annotation ratio by 32% on ImageNet with a lossless per- formance and is compatible with Semi-Supervised learning to achieve almost lossless performance with 50% annota- tion. We also demonstrate how to enhance the downstream task dataset with open-source source data. As an online method, lnfo-Coevolution is efficient and extendable for real-world applications.
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, G., Henighan. T., Child, R., Ramesh, A., Ziegler, D. M., Wu,
J., Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S., Chess, B., Clark, J., Berner, C., McCandlish, S., Radford, A., Sutskever, I., and Amodei, D. Language models are few-shot learners, 2020. URL https:// a arxiv. org/ hasp//a///// 2005 14165.
Limitations and future works Our work mainly considers the same training data distribution as the target distribu- tion on the classification task. Using the large-scale weakly supervised data may using different training data distri- bution target. For unlabeled data and weakly-supervised data, it is possible to extend the framework further while have only done preliminary study. Tasks beyond classifi- cation are also worth further exploration.
Cai, Z.. Ravichandran, A., Favaro, P., Wang, M., Modolo, D., Bhotika, R., Tu, Z., and Soatto, S. Semi-supervised vision transformers at scale, 2022. URL https:// 05688.
Acknowledgement This work is supported by Bytedance, and was done when Ziheng Qin intern at Bytedance. Zi- heng Qin, Hailun Xu (experiments) and Kai Wang (writing) are core contributors. Kanchan Sarkar and Danhui Guan project leads. Kai Wang is supported by the National Research Foundation, Singapore under its AI Singapore Programme (AISG Award No: AISG2-PhD-2221--08-000 Yang You's research group is being sponsored by NUS startup grant (Presidential Young Professorship), Singapore MOE 1 grant, ByteDance grant, ARCTIC grant, SMI grant (WBS number: A-8001 104-00-00), Alibaba grant, and Google grant for TPU usage. We thank Dawei Du for valuable discussions and feedback.
Changpinyo, S., Sharma, P., Ding, N., and Soricut, R. Conceptual 12m: Pushing web-scale image-text pre- training to recognize long-tail visual concepts, 2021. URL https://arxiv> org/ abs/ 221222222222211
Chen, X., Fang. H., Lin, T.-Y., Vedantam, R., Gupta, S., Dollar, P., and Zitnick, C. L. Microsoft coco cap- tions: Data collection and evaluation server, 2015, URL https:/ aatpaaavvvvv.o/..//..a abs abtaaaas/..... 00325.....0...
Coleman, C., Yeh, C., Mussmann, S., Mirzasoleiman, B., Bailis, P., Liang, P., Leskovec, J., and Zaharia, M. Selec- tion via proxy: Efficient data selection for deep learning. arXiv preprint arxiv: 1906:190... 1829, 2019.
Deng, J., Dong, W., Socher, R., Li, L.J.., Li, K., and Fei-Fei, L. Imagenet: A large-scale hierarchical image database. In 2009 IEEE Conference on Computer Vi- sion and Pattern Recognition, pp. 248-255, 2009. doi: 10.1 10.1109/CVPR. 2009 5206848.

---

Info-Cocvolution: An Efficient Framework for Data Model Coevolution
Deng, L. The mnist database of handwritten digit images for machine learning research [best of the web]. IEEE Signal Processing Magazine, 29(6):141-14,, 2012. doi: 10.1 10.100//2S220122221144
D. Bernstein, M. S., and Li, F.-F. Visual genome: Con- necting language and vision using crowdsourced dense image annotations, 2016. URL https://arxia: orq/ hts/abs/1600 07332.
Devlin, J., Chang, M.-W., Lee, K., and Toutanova, Bert: Pre-training of deep bidirectional transformers for lan- guage understanding, 2019. URL org/abs/ /aabs/1810.. 04805.
Krizhevsky, A., Nair, V., and Hinton, G. Cifar- (canadian institute for advanced research). a. URL http:/ /www. cs c...orontoo.o wwtrrontto.duu-aoooo www.dd/uwriiiiiiieee
html.
Krizhevsky, A., Nair, V., and Hinton, G. Cifar-100 (canadian institute for advanced research). b. URL hett:://wwx cs ns.toronto..o edu/ krik aar...tmm....
Dosovitskiy, A., Beyer, L.,, Kolesnikov, A., Weissenbom, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M.. Heigold, G., Gelly, S., Uszkoreit, J., and Houlsby, N. An image is worth 16x16 words: Transformers for image recognition at scale, 2021. URL https: wwagg//a////// abs 20000.0001100 11929.
Li, D., Wang. Z. Chen, Y., Jiang, R., Ding, W. and Okumura, M. A survey on deep active learning: Re- cent advances and new frontiers, 2024. URL https: / //atxiv. watpxi..rg//bb//405...o 00334.
//arxiv. org/
Ducoffe, M. and Precioso, F. Adversarial active learning for deep networks: margin based approach. arXiv preprint arXiv:/802.0984/ 2018.
Li, Li, D., Xiong, C.,, and Hoi, S. BLIP: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In Chaudhuri, K., Jegelka, S., Song, L., Szepesvari, C., Niu, G., and Sabato, S. (eds.), Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 12888-12900, PMLR, 17-23 Jul 2022. URL https::/p//.. /ttpttp:::pppeeeee /ttp//eeeeeengsss./ mmlr. press/ / v1 62 / / 1122n. html.
Guo, C., Zhao, B., and Bai, Deepcore: A comprehensive library for coreset selection in deep learning, 2022. URL ht htttss://arxtoe
org/ abs/ 4204 08499.
He, K., Chen, X., Xie, S., Li, Y., Dollár, P., and Girshick, R. Masked autoencoders are scalable vision learners, 2021.
URL httttt:////aiitt /arxivv org. abs /etss./i..... 06377.
Malkov, Y. A. and Yashunin, D. A. Efficient and robust approximate nearest neighbor search using hierarchical navigable small world graphs, 2018. URL https:// arxiv. org/ ha///b/////// /11./660.0000 09320.
Hino, H. Active learning: Problem settings and recent developments. CoRR, abs/2012.04225, 2020, URL https:://axki org/ abs 201.000122222
Iyer, R., Khargoankar, N., Bilmes, J., and Asanani, H. Sub- modular combinatorial information measures with appli- cations in machine learning. In Algorithmic Learning Theory, pp. 722-754. PMLR, 2021.
Margatina, K., Vernikos, G., Barrault, I.,, and Aletras, N. Active learning by acquiring contrastive examples. arXiv preprint arXiv:2 109.03764 2021.
Mirzasoleiman, B., Bilmes, and Leskovec, Coresets for data-efficient training of machine learning models. In International Conference Machine Learning, pp. 6950 6960. PMLR, 2020.
Killamsetty, K., Durga, S., Ramakrishnan, G., De, A., and Iyer, R. Grad-match: Gradient matching based data sub- set selection for efficient deep model training. In Interna- tional Conference on Machine Learning, pp. -574. PMLR. 2021a.
Netzer, Y., Wang, T., Coates, A., Bissacco, A., Wu, B., and Ng. A. Y. Reading digits in natural images with unsupervised feature learning. In NIPS Workshop on Deep Learning and Unsuper- vised Feature Learning 2011, 2011. URL http: //ufldl. stanford. htwtu//huusnumbbrs/// nips2011 1a120111_hhuseumbbre.. .dd.
Killamsetty, K., Sivasubramanian, D., Ramakrishnan, G., and Iyer, R. Glister: Generalization based data subset selection for efficient and robust learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35-9, pp. 8110-8118, 2021b.
Kirillov, A., Mintun, E., Ravi, N., Mao, H., Rolland, C., Gustafson, I.,, Xiao, T., Whitehead, S., Berg, C., Lo, W.- Dollár, P. and Girshick, R. Segment abs/ anything, 2023. URL https://arxivv. 2304. 02643.
Ordonez, V., Kulkarni, G., and Berg, T. Im2text: Describing images using million captioned photographs. In Shawe-Taylor, J., Zemel, R., Bartlett, P., Pereira, F., and Weinberger, K. (eds.), Advances in Neural Information Processing Systems, volume 24. Curran Associates, Inc., 2011. URL httpttt/prrceeddingg www.:innnnn..uuu.eeeeeeem neurips. (/tpape_ffilee/pperr//011/1/1111
Krishna, R., Zhu, Y., Groth, O., Johnson, J., Hata, K., Kravitz, J., Chen, S., Kalantidis, Y., Li, L.-J., Shamma,
10

---

Info-Cocvolution: An Efficient Framework for Data Model Coevolution
Papper.
Smith, F. B., Kirsch, A., Parquhar, S., Gal, Y., Foster, A., and Rainforth, T. Prediction-orientedd bayesian active learning, 2023, URL https:/ https///axxvv.oooee org/abs/ abs/ 7304.08154
pdf.
Paul, M., Ganguli, S., and Dziugaite, K. Deep learning on data diet: Finding important examples early in training. Advances in neural information processing systems, 34: 20596-20600 2021.
Sohn, K., Berthelot, D., Li, C.-L., Zhang, Z., Carlini, N., Cubuk, E. D., Kurakin, A., Zhang, H., and Raf-
fel, C. Fixmatch: Simplifying semi-supervised learning with consistency and confidence, 2020. URL https: 07685.
Peng. Z., Dong, L.,, Bao, H., Ye, Q., and Wei, F. Beit v2: Masked image modeling with vector-quantized visual tokenizers, 2022. URL https: arxiv. org/abs/ 2208.06366
/ www.xiv.org/bb.... 276850061........
Toneva, M., Sordoni, A., Combes, R. T. d., Trischler, A.. Bengio, Y., and Gordon, G. J. An empirical study of example forgetting during deep neural network learning. arXiv preprint arXiv: arxxiv/812.000 181.055/9, 2018.
Qin, Z., Xu, Z., Zhou, Y., Zheng, Z., Cheng, Z., Tang, H., Shang. L., Sun, B., Peng, X., Timofte, R., Yao, H., Wang, K., and You, Y. Dataset growth. In Computer Vision ECCV 2024: 18th European Con- ference, Milan, Italy, September 29-October 4, 2024, Proceedings, Part IX, pp. 58-75, Berlin, Heidelberg, 2024. Springer-Verlage ISBN 978-3-031-72672-9, doi: 10. 10.10077978--3331--7677------662 URL https://doi. 11t--11000799-----------7-------
Wang, Y., Chen, H., Heng, Q. Hou, W., Fan, Y., Wu, Z., Wang, J., Savvides, M., Shinozaki, T., Raj, B., Schiele, B., and Xie, X. Freematch: Self-adaptive thresholding for semi-supervised learning, 2023. URL https:// 07246.
arxiv. arxiv.org/
/ttpabs/////oooooooe//o/////aaaa abs 2205.07246....
Xiao, T., Liu, Y., Zhou, B., Jiang, Y., and Sun, J. Uni- fied perceptual parsing for scene understanding. CoRR, abs/
Radford, A., Kim, W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., Krueger, G., and Sutskever, Learning transferable visual models from natural language supervision, 2021. URL www.tttps:/.m htt//aaxvivv. http///aaxvvv....aaa//// hbtp://bs//1000 2002.00020......
2018. URL
Zhang, B., Wang, Y., Hou, W., Wu, H., Wang, J., Oku- mura, M., and Shinozaki, T. Flexmatch: Boosting semi-supervised learning with curriculum pseudo label- ing, 2022. URL httpst:: //axivv httg/bb///111 08263.
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., and Liu, P. J. Exploring the limits of transfer learning with unified text-to-text transformer. Journal of machine learning research, 21 1140):1-67, 2020.
Zhou, B., Zhao, H., Puig, X., Fidler, S., Barriuso, A., and Torralba, A. Scene parsing through ade20k dataset. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017.
Schuhmann, C., Vencu, R., Beaumont, R., Kaczmarczyk,
R., Mullis, C., Katta, A., Coombes, T., Jitsev, J., and Komatsuzaki, A. Laion-400m: Open dataset of clip- filtered 400 million image-text pairs, 2021. URL https: //arxiv org/ abs/ / 2212111111111 02114.
Zhou, D., Wang, K., Gu, J., Peng, X., Lian, D., Zhang, Y., You, Y., and Feng, J. Dataset quantization, 2023. URL https http://axxvvv.ovv waspr///////////e/////eeee/eaaee abs///b//2220 2208. 10524.
Sener, O. and Savarese, S. Active learning for convolutional neural networks: core-set approach. arXiv preprint 1rXiv:1708.0000 2017.
Zhu, Y. Aligning books and movies: Towards story-like vi-
sual explanations by watching movies and reading books. arXiv preprint / 06724, 2015.
Sener, O. and Savarese, S. Active learning for convolu- tional neural networks: A core-set approach, 2018. URL https: //tp:://axivvvv/ii http//aaxvvvgaa/aaaaaa abs wttp////.wr///////////// 00489.
0048.
Sharma, P., Ding, N., Goodman, S., and Soricut, R. Con- ceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. In Proceedings of ACL, 2018.
Sinha, S., Zhang, H., Goyal, A., Bengio, Y., Larochelle, H., and Odena, A. Small-gan: Speeding up gan training using core-sets. In International Conference on Machine Learning, pp. 9005-9015. PMLR, 2020.

---

Info-Cocvolution: An Efficient Framework for Data Model Coevolution
A. Proof
We show our proof of Theorem3.1 here as follows: For the model M = g o f. assume g is Lg -Lpischcit,,, i.e. for all 21,22 in the feature space, we have
l19(=1) [(22)]l S 4 - =2ll
(8)
Then,
(1((2)) - ((2)) S Lalle1 22 S Lge
(9)
Softmax is known to be I-Lipschitz which does not further change the bound of logits.
When using cosine distance, we know that
< 21, 22 >
cosine dis((2,, =1
(10)
| |2
If features
||=1|||||2||| are normalized (as in ViT networks), then
|| 22ll v cosine coosioeedssteee dist(21, 22)
(11)
And
cosine dist(21, 22) < = (1œ(21) - g(>2)ll <= L,,
(12)
For a single linear layer (g) with W (RRdxC) and b, its Lipschitz constant is bounded by the spectral norm of W, which is ||W]|2. However, for a Gaussian weight with (0, o²), could still have a bound of O(a(vd + vC)). In practice, this still quite big. and only serves as a worst-case bound.
There is tighter bound using the local gradient assuming smoothness (which is true for most type of g):
(1g(21) - g( S (uPxejo.n]|[Vg(()) ||2| - =22ll S supxe(o_]] (Vxex)l
(13)
where 2A = A21 + (1 - 2)22-
Empirically, the linearity mainly takes effect when the predictions are reasonably good. It is because
On one hand, when the prediction is poor, the neighbour could be more random and the predicted knn- aan-prediction will give high entropy itself as well as the model, which will not contradict our purpose of the algorithm (the corresponding sample would very likely have high entropy and gain, and the theoretical guarantee of knn pred is from the k value);
On the other hand, when the prediction is good (e.g. P.0.7 or higher) in the region, the local gradient will be smaller, and the softmax backward propagation further suppresses the bound with ||p well <= 1.
So in the cases where the local gradient might fail to give a reasonable bound, the result itself would very likely have high entropy and be correctly estimated as high entropy by both model and knn, and the knn's estimation error doesn' 't matter too much. The actual algorithm reliance on this linearity is relaxed in bad cases due the algorithm design. And for regions with good predictions, we empirically evaluate the value with the local gradient.
B. Additional Experiment Details
We follow the experimental settings in Semi-ViT. Our results are trained on a single node of 8 NVIDIA A100-SXM4-80G. For supervised finetuning, we train the model with batchsize 512, learning rate 0.001 for 50 epochs with all augmentations same as in Semi-ViT.
C. Semi-supervised Data Selection
For selecting the semi-supervised data, we further add another term based on each sample's average distance to nearest k neighbours in high-confidence samples and already selected samples.
Semi Gain(z)=
Σ
cosine cosine_dist(z,, x)
(14)
*EKNN
12

---

Info-Coevolution: An Efficient Framework for Data Model Coevolution
This term is to encourage a more uniform distribution of unlabeled samples in the space between the labeled samples, so that the semi-supervised training progressively learn the pseudo target.
13


Reviews:
[{"review_id": "fALFujrHgc", "reviewer": "Reviewer_X845", "summary": "The paper presents **Info-Coevolution**, a framework aimed at enhancing the co-evolution of data and models through **online selective annotation**. The primary goal is to minimize annotation costs while preserving model performance by utilizing **Bayesian Prediction Fusion** and **data locality analysis** to assess the information gain of samples. This approach selectively annotates data, facilitating efficient dataset construction and model training.\n\nKey findings include:\n- **Reduced Annotation Costs**: Info-Coevolution achieves **lossless performance** on ImageNet-1K with only **68% of the annotation cost**, further reducing to **50%** with semi-supervised learning.\n- **Efficiency**: The framework incurs minimal computational overhead, completing the selection process in **1 minute** for million-scale datasets.\n- **Compatibility**: It is compatible with both supervised and semi-supervised learning, avoiding distribution shifts during continual training.\n\nThe paper also investigates **retrieval-based dataset enhancement** using unlabeled open-source data, showing improved performance with additional unlabeled data.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer X845 for pointing out the missing references as well as the potential improvements. We make responses as follows.\n\n**Q1: Dataset Distillation references not discussed.**\n\n**A1**: Thanks for the advice. In general, dataset distillation and active learning have a main overlapping research area which is coreset selection (if not requiring full annotation in advance, then the coreset selection can also serve as active learning), and is introduced in related works. Dataset distillation methods also include synthetic data methods, which is another setting not comparable in this work. We will add a discussion of dataset distillation to the related works to make it more clear.\n\n**Q2: Task Scope beyond image classification such as object detection and segmentation, is not investigated.**\n\n**A2**: Thanks. We here extend our algorithm to semantic segmentation and attach results on ADE20k with UperNet(BEiT-v2 backbone). The UperNet has a backbone (BEiT-v2-large) and a UperHead. We use the backbone feature from the last layer (dim 1024\\*16\\*16) and mean pooling it to dim 1024. We adapt Info-Coevolution accordingly, using pixel-wise confidence avg as model confidence, class-wise similarity weighted confidence as knn confidence, and $gain=1-avg(confidence_{model},confidence_{knn})$. The result for using 1% data trained model to select 10% data is as follows:\n\n**ADE20K 10% random**\n|aAcc|mIoU|mAcc|\n|---|---|---|\n| 82.19 | 46.89 | 58.72 |\n\n**ADE20K 10% our selection (using BEiT feature)**\n|aAcc|mIoU|mAcc|\n|---|---|---|\n| 82.84 **$\\uparrow$0.65** | 48.39 **$\\uparrow$1.50** |60.81 **$\\uparrow$2.09**|\n\n**Analysis:**\n- It can be seen that the **mAcc, mIoU, and mAcc are improved by 0.65, 1.50, and 2.09 respectively**, which is significant at this data amount. (For larger data ratio we are still running experiments and will update the results in the later updates. We will add thorough experiments in the revision. The code of this part will also be published.)\n\n**Conclusion:**\n- Info-Coevolution is generalizable to more tasks such as segmentation as analyzed.\n\n**Q3: Theoretical Depth: requires further expansion, particularly regarding limitations and assumptions.**\n\n**A3**: Thanks for the advice. The current theoretical part supports us to introduce locality into the algorithm, which could greatly improve the efficiency of distribution-based entropy estimation methods, and benefit efficient data re-balance (dynamic data rechecking). \n\nWe agree an expansion should be in the assumption of Lipschitz-continuous, please refer to the discussion in the reply A1 to reviewer aJ9B Q1 (due to char limit this year). We are open to further discussion.\n\nOn the other hand, the current assumption is that the target distribution should be between IID and uniform, so any sample predicted as redundant by the framework doesn't affect the final classification result. We will attach a theoretical discussion in the next update for this argument. So a limitation is that it currently doesn't handle a target distribution beyond the range of IID to uniform. A solution could be adjusting the sampling frequency based on target distribution. \n\nAll the discussed parts will be added to the revision, and we are **open to further discussion** for the theoretical part.\n\n**Q4: Include a broader discussion on Limitations and Future Work (applicability to weakly supervised and non-classification tasks)**\n\n**A4**: Thanks. The semantic segmentation experiment is now added in A2. We will add a more detailed discussion in the revision. \n\nIn general, the framework could be applied to models with an $f$, which produces a feature for each sample (and could be used for a retrieval task). If mutual information could be approximated as in classification, it could be a good usage case; even if not a direct approximation, the ANN could also provide distribution-based estimation (with a weighted mean of uncertainty).\n\nClassification can be interpreted as a kind of coarse-grained retrieval. \n\nFor weakly supervised training, as for CLIP/BLIP schema, they are retrieval tasks directly training the $f$ with contrastive training. The locality is also taking effect, and mutual information could be estimated in the feature space. So theoretically our framework is also applicable to weakly supervised training, and the reannotation gain could be estimated for human/ChatGPT annotation to improve data quality.\n\n**Q5: Real-World Validation**\n\n**A5**: We evaluate the method with some in-house data pipeline, and it achieves promising results.", "claims_and_evidence": "The claims are generally well-supported:\n- **Reduced Annotation Costs**: Supported by experiments on ImageNet-1K, CIFAR-10/100, and other datasets, demonstrating comparable or superior performance with fewer annotations.\n- **Efficiency**: Backed by computational overhead analysis, indicating logarithmic scaling and rapid completion for large datasets.\n- **Compatibility with Semi-Supervised Learning**: Validated through experiments with Semi-ViT and Fixmatch, showing enhanced data efficiency.\n\nHowever, the claim of **generalizability to other tasks** is not fully substantiated, as experiments are confined to classification tasks.", "methods_and_evaluation_criteria": "The methods and evaluation criteria are suitable for the problem:\n- **Bayesian Prediction Fusion** and **data locality analysis** are well-justified for estimating information gain and enhancing sample selection efficiency.\n- The use of **Approximate Nearest Neighbor (ANN)** structures like HNSW ensures scalability for large datasets.\n- Evaluation criteria (e.g., accuracy on ImageNet-1K, CIFAR-10/100) are standard benchmarks, facilitating comparison with prior work.\n\nThe evaluation could be improved by including **additional tasks** and **real-world datasets** to demonstrate broader applicability.", "theoretical_claims": "The paper presents several theoretical claims:\n- **Theorem 3.1**: Asserts that predictions for nearby samples in feature space are similar under certain distance thresholds. The proof, provided in the appendix, appears correct but relies on assumptions (e.g., Lipschitz continuity) that warrant further discussion.\n- **Information Gain Estimation**: The extension to broader tasks is theoretically sound, but the derivation could benefit from more rigorous analysis, especially in multi-class settings.\n\nIn general, while the theoretical claims are credible, they also appear somewhat trivial.", "experimental_designs_or_analyses": "The experimental designs are robust and validate key claims:\n- **ImageNet-1K Experiments**: Demonstrate lossless performance with 68% annotation cost, compatible with semi-supervised learning.\n- **Comparison with Coreset Selection**: Shows competitive performance with state-of-the-art methods.\n- **Generalization Across Datasets**: Consistent improvements in annotation efficiency on CIFAR-10/100, StanfordCars, and other datasets.\n\nHowever, experiments are limited to **image classification tasks**, lacking validation on **real-world industrial data** or **larger datasets**. The computational overhead, while low, remains significant for resource-limited settings.", "relation_to_broader_scientific_literature": "The paper builds on and extends prior work in:\n- **Active Learning**: Addresses limitations of traditional methods by integrating model-specific estimation with distribution awareness.\n- **Coreset Selection**: Improves upon existing methods by leveraging model-specific information without requiring fully annotated data.\n- **Semi-Supervised Learning**: Bridges the gap between fully supervised and weakly supervised approaches.\n\nThe paper contributes to the literature by proposing a **more efficient and scalable approach** to data annotation and model training, with potential applications in resource-constrained settings.", "essential_references_not_discussed": "The paper does not discuss **dataset distillation**, a highly relevant field that shares similarities with the goals of Info-Coevolution. Dataset distillation focuses on synthesizing a small, informative dataset that can be used to train models with performance comparable to training on the full dataset. This is conceptually aligned with Info-Coevolution's goal of reducing annotation costs while maintaining model performance.  \n\nA key work in this area is:  \n- **Wang et al. (2018)**: \"Dataset Distillation\" (arXiv:1811.10959). This paper introduces the concept of dataset distillation, where a small synthetic dataset is created to mimic the performance of a larger dataset. The methods and insights from this work could provide valuable context for Info-Coevolution, particularly in terms of reducing data requirements while preserving model performance.  \n\nThe omission of this reference is notable, as dataset distillation represents a complementary approach to the problem of efficient dataset construction and could enrich the discussion of related work in the paper. Including this reference would help situate Info-Coevolution within the broader landscape of data-efficient machine learning techniques.", "other_strengths_and_weaknesses": "**Strengths:**\n\n- **Originality**: The integration of Bayesian Prediction Fusion, data locality analysis, and online selective annotation presents a novel and creative approach.\n- **Significance**: This framework addresses the critical challenge of reducing annotation costs in machine learning while maintaining performance.\n- **Clarity**: The paper is well-written with clear explanations of the methodology and results.\n\n**Weaknesses:**\n\n- **Limited Task Scope**: The experiments focus solely on image classification, and the method's potential application to other tasks, such as object detection and segmentation, is not investigated.\n- **Theoretical Depth**: The theoretical derivations require further expansion, particularly regarding limitations and assumptions.\n- **Omission of Dataset Distillation**: There is no discussion of dataset distillation, which is closely related to the goals of Info-Coevolution, aiming to synthesize a small, informative dataset that achieves performance close to training on the full dataset. This aligns with the goal of reducing annotation costs while maintaining model performance in Info-Coevolution.", "other_comments_or_suggestions": "- **Experiments**: Expand to encompass other tasks, such as object detection and segmentation, and incorporate real-world datasets.\n- **Limitations and Future Work**: Include a broader discussion on limitations and future directions, specifically regarding applicability to weakly supervised and non-classification tasks.\n- **Dataset Distillation**: Discuss its relationship with Info-Coevolution to strengthen contextual grounding and demonstrate awareness of related approaches.", "questions_for_authors": "1. **Generalizability**: Can Info-Coevolution be extended to tasks beyond image classification, such as object detection or segmentation?\n\n2. **Theoretical Limitations**: While the theoretical claims are plausible, they seem somewhat elementary. Could the authors provide a more comprehensive theoretical analysis concerning deep neural networks, including aspects of optimization and generalization theory?\n\n3. **Real-World Validation**: Has Info-Coevolution been evaluated on real-world industrial datasets or in online annotation environments? \n\n4. **Dataset Distillation**: How does Info-Coevolution compare to dataset distillation methods, such as those proposed by Wang et al. (2018) and subsequent works?\n\n[1] Wang et al. (2018). \"Dataset Distillation\" (arXiv:1811.10959).", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "EjsAvo67Lf", "reviewer": "Reviewer_aJ9B", "summary": "The paper introduces Info-Coevolution, a framework for selective data collection which aims to improve data annotation efficiency. It proposes strategies to estimate information gain by leveraging Bayesian principles, and also uses ANN structures to help achieve efficient data selection with minimal computational overhead, which reduces the need for frequent model updates over the selection process. The framework was benchmarked against conventional active learning and coreset selection methods, and was shown to reduce annotation costs while maintaining model performance.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer aJ9B for the recognition and appreciation of our work, and for the valuable questions as well as comments. For the comments and questions, here are our responses:\n\n**Q1: The Lipschitz constant may be difficult to compute rigorously and large in practice.**\n\n**A1**:Thanks for the good question. Previously we use the theorem 3.1 to support our introduction of locality, but it is true that the Lipschitz constant is not tight enough to be used in many real settings. We here discuss a tighter bound and the overall reasonability.\n\nFor a single linear layer ($g$) with W ($\\mathbb{R}^{d\\times C}$) and b, its Lipschitz constant is bounded by the spectral norm of W, which is $||W||_2$. However, for a Gaussian weight with $N(0,\\sigma^2)$, it could still have a bound of $O(\\sigma(\\sqrt d + \\sqrt C))$. In practice, this is still quite big, and it only serves as a worst-case bound.\n\nThere is a tighter bound using the local gradient assuming smoothness (which is true for most type of $g$):\n$$\\lVert g(z_1)-g(z_2)\\rVert_2 \\leq sup_{\\lambda \\in [0,1]} \\lVert\\nabla g(z_\\lambda)\\rVert_2 \\leq sup_{\\lambda \\in [0,1]} \\lVert \\nabla g(z_\\lambda)\\rVert_2 \\cdot\\epsilon ,$$ Where $z_\\lambda = (1-\\lambda)z_1 + \\lambda z_2$ .\n\nEmpirically, the linearity mainly takes effect when the predictions are reasonably good. It is because \n- On one hand, when the prediction is poor, the neighbour could be more random and the predicted knn-prediction will give a high entropy itself as well as the model, which will not contradict our purpose of the algorithm (the corresponding sample would very likely have a high entropy and gain, and the theoretical guarantee of knn pred is from the k value); \n- On the other hand, when the prediction is good (e.g. p>0.7 or higher) in the region, the local gradient will be smaller, and the softmax backward propagation further suppresses the bound with $||p-y_t||<=1$. \n\nSo in the cases where the local gradient might fail to give a reasonable bound, the result itself would very likely have high entropy and be correctly estimated as high entropy by both model and knn, and the knn's estimation error doesn't matter too much. The actual algorithm reliance on this linearity is relaxed in bad cases due to the algorithm design. And for regions with good predictions we can empirically evaluate the value.\n\n**Q2: The method claims to be able to address data distribution shifts however it was not shown in the benchmarks**\n\n**A2**: The data distribution shifts in the article refer to the problem of model-based active learning: it only emphasizes on samples learned poorly by the model, and the selection causes a distribution problem and worse performance. This is shown in the ablation as dynamic-rechecking/KNN-only are all without this performance drop. We will revise the corresponding parts to make this clear.\n\n**Q3: The set of experiments does not demonstrate the generalizability of the method to other data modalities or models, and expanding that evaluation would make the analysis stronger.**\n\n**A3**: Thanks for the advice. We further add semantic segmentation experiment on ADE20k with UperNet(BEiT-V2 backbone). See A2 to Reviewer X845. \nFor modalities other than vision, if there is a sample-level feature, it is also possible to extend the framework to them. Will add more discussion in the revision of future work.\n\n**Q4: Typo in the abstract**\n\n**A4**: Thanks, we have fixed it in local revision. Should be \"… Info-Coevolution reduces annotation and training costs by 32% without performance loss\".\n\n**Q5: Hyperparameters used in the framework (such as the distance and similarity thresholds). Could the authors give more insight into finding the optimal values, and how adjusting them might affect the method’s performance.**\n\n**A5**: The hyperparameters should depend on the feature space itself (how dense is the data in the space), and can use empirical methods to evaluate(retrieve some samples to estimate the knn distance distribution and knn-prediction correlation, and an adequate k). \n\nGenerally, k = 8, cosine similarity 0.9 is good enough, and cosine similarity 0.85 also works in the setting; 0.95 will be too big as there will be very few near neighbour pairs. \n\nToo small distance (too large similarity threshold) may fail to reduce cost and become a model-based method (for dynamic rechecking. Actually using 0.9 for knn and 0.85 for dynamic rechecking is also a good choice), while too large distance threshold may affect sparse-region samples' knn-prediction (their k-near-nighbour could be further and less linear correlated, and should be kept for the sake of generalization). \n\nIn our actual use cases, the threshold value is not sensitive, cosine similarity 0.9 directly works without tuning, and cosine similarity 0.85 doesn't make a statistically significant difference.", "claims_and_evidence": "The claim that Info-Coevolution can reduce annotation and training costs without compromising model performance was also support by benchmarks on various datasets, where it was compared with other baseline methods. The ablation experiments also further validated that each component of the methodology contributes to the performance of the framework.", "methods_and_evaluation_criteria": "Yes, the methodology introduced tackles the task of improving annotation efficiency while maintaining the model performance. Benchmarks were performed on well known datasets such as ImageNet-1k, CIFAR-10/100 etc to showcase the method’s effectiveness compared to other baseline methods. The authors also evaluated both the supervised and semi-supervised learning settings, covering different training paradigms.", "theoretical_claims": "Yes, the proof for Theorem 3.1 is correct, however it should be noted that it is made under the assumption that the function g is Lipschitz continuous. While most common used neural networks exhibit Lipschitz continuity, the Lipschitz constant may be difficult to compute rigorously and large in practice. Hence the bound may not be tight in real world settings, a limitation that should be addressed.", "experimental_designs_or_analyses": "The experimental designs and analyses were sound, they were performed on well known benchmark datasets, and compared with established baselines under supervised and semi-supervised settings, with clearly defined annotation metrics such as accuracy improvements across different annotation ratios and annotation efficiency. The ablation experiments also served to demonstrate that each component of the framework contributes to the performance improvements.\n\nHowever some limitations would include that the benchmarks only contained image/computer vision datasets and the models used were mostly limited to ViTs and ResNets. The set of experiments do not demonstrate the generalizability of the method to other data modalities or models, and expanding that evaluation would make the analysis stronger.", "relation_to_broader_scientific_literature": "The contributions are related to several areas in ML, notably active learning and coreset selection. Traditional active learning methods focuses on selecting the most informative samples for annotation, usually based on model uncertainty. Info-Coevolution additionally integrates distribution awareness of the data for selection, which aides in addressing common issues in active learning such as bias and distribution shifts. It also builds upon coreset selection ideas where representative subsets are chosen to improve training. But unlike traditional coreset methods which are often static, Info-Coevolution filters data dynamically and in an online manner, making the process more efficient. Lastly, the methodology extends ideas from information theory, using information gain based selection, but incorporates additional information such as data similarity and using a bayesian prediction function to estimate the information gain for each datapoint more accurately.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "Strengths:\n- The methodology is novel by integrating several ideas in a unified framework\n- The benchmarks have demonstrated the effectiveness of the method and the savings it can achieve without compromising model performance, which is key for real world applications\n- The experiments were clear and ablation study helped to break down the components of the approach\n\nWeaknesses:\n- The authors mention in the limitations section that their experiments mainly considers cases where the training and target distributions are the same, the method claims to be able to address data distribution shifts however it was not shown in the benchmarks\n- Other concerns/weaknesses have been pointed out in the comments above", "other_comments_or_suggestions": "There is a typo in the abstract where a sentence is incomplete: \n\n> … Info-Coevolution reduces annotation and training costs by 32% without performance.", "questions_for_authors": "The paper does not provide a detailed analysis of the hyperparameters used in the framework (such as the distance and similarity thresholds). Could the authors give more insight into finding the optimal values, and how adjusting them might affect the method’s performance.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "b58t7cWile", "reviewer": "Reviewer_9pQu", "summary": "This paper points out the current issue of active learning. Traditional active learning methods select informative data points for annotation but suffer from high computational costs, frequent model retraining, and bias in uncertainty-based selection. Considering these, they propose a novel framework called Info-Coevolution, a model-data fusion coevolution model that integrates: (1) Bayesian information gain estimation to evaluate how much information a sample contributes to model improvement; (2) kNN approximation with HNSW, to measure entropy and confidence without model retraining and (3) Bayesian fusion, to model confidence and data-driven uncertainty for more robust sample selection. The authors then test their model on CIFAR-10, CIFAR-100, ImageNet-1K, and the Info-Coevolution got 68% annotation cost while maintaining full model performance and 50% annotation cost with semi-supervised learning.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer 9pQu for the suggestions as well as the potential improvements. We make responses as follows.\n\n**Q1: Adding references and baselines**\n\n**A1**:Thanks for the suggestions. We attach more baselines here, and will add the corresponding references.\n\nImageNet-1k with ViT supervised training from 1% data to 10% data, \n|Method|step|Acc|Cost|\n|---|---|---|---|\n|MASE|9%|80.0|1 training + selection|\n|MASE|1w|80.2|12 training + selection|\n|BASE|9%|79.7|1 training + selection|\n|BASE|1w|80.2|12 training + selection|\n|Partial BADGE|9%|78.9|1 training + selection|\n|Partial BADGE|1w|80.4|12 training + selection|\n|Info-Coevolution|9%|**80.2%**|**1 training** + selection|\n|Info-Coevolution|~2.25%|**80.5%**|**4 training** + selection|\n\n**Analysis:**\n\n- Previous active learning methods require training for each step. At the same step number, Info-Coevolution has the best performance with better time cost. Given the full budget, other baseline methods have much higher costs ($O(n^2)$ for these step-wise methods) and still have lower performance than Info-Coevolution. \n\n- What's more, these methods are also hard to further scale (due to $O(n^2)$ training complexity) while Info-Coevolution can directly scale from 12w samples to 87w samples without a in-middle training step. \n\n- Additionally, Info-Coevolution doesn't introduce any data distribution problems, so we are able to do continual training. We have additionally verified that using continual training on previous checkpoints has no performance loss. (1% data with 50 epoch, continual training with selected10% data with 45% epoch, then continual training on 68% data with 40 epoch) The training cost of our algorithm could be O(n) in total for selected data.\n\n**Conclusion:** \n\n- Info-Coevolution has better performance and much better scalability than previous baselines.\n\n**Q2: Ablation study to show the effectiveness of (1) Bayesian fusion, (2) kNN approximation**\n\n**A2**: Thanks for the question. We make a clarification here, and ask for the reviewers' clarification on some question points to further clarify/discuss in the next response.\n\nWe have conducted part of the ablation in sec 4.3 ablation, including kNN only, model-only, fused, and their combination of dynamic rechecking. The Data column is referring to kNN-only. It can be seen dynamic rechecking is the one guaranteed non-biased distribution when incorporating model-based gain estimation (model estimation without dynamic rechecking will lower the performance), while kNN-only without dynamic rechecking could improve performance already.\n\nFor the calibration, which calibration (e.g., temperature scaling) is referred to? Could you give a reference so we can be more clear and add a comparison?\n\nMonte Carlo Dropout-based uncertainty estimation introduces random noise in the feature space, which is somehow similar to our aggregating information from nearby samples to estimate. But the differences are: \n1. our method does not have to do inference multiple times. \n2. As a model-based method, Monte Carlo Dropout-based uncertainty estimation is estimating the model optimization space. It could still suffer distribution problems, as the dropout is not considering data distribution (it is more about the local curvature of the model).\n\nFor kNN approximation vs. true model entropy estimation, our weighted KNN-based method doesn't introduce much distribution bias. Incorporating model-based estimation usually causes this (as observed with lower performance than randomly selected data), and needs to be fixed by dynamic rechecking or other distribution trimming methods.\n\n**Q3: For the Bayesian information gain estimation, no proof shows how kNN-based entropy estimation approximates the true situations or the model-based entropy estimates (approximation error).**\n\n**A3**: We would like to first clarify on the point: is the reviewer referring to the problem that kNN-based entropy estimation or model-based entropy estimation could differ to real entropy?\n\nWe add some preliminary discussions here and will update it in the next reply based on the reviewer's feedback. Empirically, the model confidence (defined in section 3) is linear correlated (more than 0.95) to sample prediction accuracy on both training data and validation data. We are using this probability proxy instead of real entropy (mentioned in sec 3), because it keeps this observed linearity instead of introducing log function with an unstable bound, and thus when linearity does hold, the knn prediction can help to decide the sample priority. \n\nWe also added some discussion in A1 to Reviewer aJ9B about our Lipschitz constant assumption, which is somehow related. \n\nWe are open to further discussions.", "claims_and_evidence": "Yes.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "For the Bayesian information gain estimation, no proof shows how kNN-based entropy estimation approximates the true situations or the model-based entropy estimates (approximation error).", "experimental_designs_or_analyses": "The baselines selection is too limited, just include very normal baselines like random, coreset. Can consider stronger baselines like BADGE (incorporating the uncertainty and diversity based measures, it is comparable).", "relation_to_broader_scientific_literature": "There is no discussion about: Bayesian fusion vs. standard active learning work that combines the model- and data-driven approaches like BADGE and Batch Active Learning at Scale.", "essential_references_not_discussed": "See previous sections.\n\n[r1] Ash, Jordan T., et al. \"Deep batch active learning by diverse, uncertain gradient lower bounds.\" arXiv preprint arXiv:1906.03671 (2019).\n\n[r2] Citovsky, Gui, et al. \"Batch active learning at scale.\" Advances in Neural Information Processing Systems 34 (2021): 11933-11944.", "other_strengths_and_weaknesses": "Strengths:\n1. It reduces the re-training cost;\n2. It reduces annotation costs by 30-50% while maintaining performance.\n\nWeaknesses:\n1. Should add stronger baselines;\n2. No ablation study to show the effectiveness of (1) Bayesian fusion, like Bayesian vs. direct calibration (e.g., temperature scaling); Bayesian fusion vs. Monte Carlo Dropout-based uncertainty estimation and remove this part; (2) kNN approximation vs. true model entropy estimation (can be replaced by computing from full retraining).", "other_comments_or_suggestions": "Should use more space to describe Bayesian fusion.\n\nLine 025-028: \"For real-world datasets like ImageNet-1K, Info-Coevolution reduces annotation and training costs by 32% without performance.\" is not a completed sentence.", "questions_for_authors": "See Section \"Other Strengths And Weaknesses\".", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kZ87yianHq", "reviewer": "Reviewer_i2qw", "summary": "The paper addresses the challenge of high annotation costs and inefficiency in training models on growing datasets by proposing a framework for online selective annotation that co-evolves data and models. The method combines information gain estimation (model uncertainty and dataset locality via nearest-neighbor analysis) with Bayesian prediction fusion to merge model and data-derived predictions, reducing bias, and dynamic rechecking to update sample priorities post-annotation for class balance. Using efficient approximate nearest-neighbor search (HNSW), it scales logarithmically, enabling million-scale dataset handling. Key results include achieving lossless ImageNet-1K performance with 32% fewer annotations (68% total) and 50% with semi-supervised learning, while generalizing across datasets (CIFAR, SVHN) and architectures (ViT, ResNet). The framework integrates public data (e.g., LAION-400M) to enhance performance with minimal unlabeled data and incurs low overhead (~10 GPU hours on ImageNet). It outperforms coreset selection and active learning by avoiding distribution bias and automates annotation halting when gains plateau.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer i2qw for the suggestions as well as the potential improvements. We make responses as follows.\n\n**Q1: The paper only compares with one AL method, DQ (coreset-based), in Fig. 6. More AL methods could be compared, from the early entropy-based methods to recent AL methods. I understand the proposed method is both data-agnostic and model-agnostic, but it'd be better to include more AL methods to comprehensively evaluate its effectiveness. More experiments on different semantic understanding tasks could be considered.**\n\n\nThanks for the suggestions. We attach more baselines here and extend our algorithm to the semantic segmentation task.\n\n### **More baselines**\n\nImageNet-1k with ViT supervised training from 1% data to 10% data, \n|Method|step|Acc|Cost|\n|---|---|---|---|\n|MASE|9%|80.0|1 training + selection|\n|MASE|1w|80.2|12 training + selection|\n|BASE|9%|79.7|1 training + selection|\n|BASE|1w|80.2|12 training + selection|\n|Partial BADGE|9%|78.9|1 training + selection|\n|Partial BADGE|1w|80.4|12 training + selection|\n|Info-Coevolution|9%|**80.2%**|**1 training** + selection|\n|Info-Coevolution|~2.25%|**80.5%**|**4 training** + selection|\n\n**Analysis:**\n\n- Previous active learning methods require training for each step. At the same step number, Info-Coevolution has the best performance with better time cost. Given the full budget, other baseline methods have much higher costs ($O(n^2)$ for these step-wise methods) and still have lower performance than Info-Coevolution. \n\n- What's more, these methods are also hard to further scale (due to $O(n^2)$ training complexity) while Info-Coevolution can directly scale from 12w samples to 87w samples without a in-middle training step. \n\n- Additionaly, Info-Coevolution doesn't introduce any data distribution problems, so we are able to do continual training. We have additionally verified that using continual training on previous checkpoints has no performance loss. (1% data with 50 epoch, continual training with selected10% data with 45% epoch, then continual training on 68% data with 40 epoch) The training cost of our algorithm could be O(n) in total for selected data.\n\n**Conclusion:** \n\n- Info-Coevolution has better performance and much better scalability than previous baselines.\n\n### **Semantic segmentation**\n\nWe here extend our algorithm to semantic segmentation and attach results on ADE20k with UperNet(BEiT-v2 backbone). The UperNet has a backbone (BEiT-v2-large) and a UperHead. We use the backbone feature from the last layer (dim 1024\\*16\\*16) and mean pooling it to dim 1024. We adapt Info-Coevolution accordingly, using pixel-wise confidence avg as model confidence, class-wise similarity weighted confidence as knn confidence, and $gain=1-avg(confidence_{model},confidence_{knn})$. The result for using 1% data trained model to select 10% data is as follows:\n\n**ADE20K 10% random**\n|aAcc|mIoU|mAcc|\n|---|---|---|\n| 82.19 | 46.89 | 58.72 |\n\n**ADE20K 10% our selection (using BEiT feature)**\n|aAcc|mIoU|mAcc|\n|---|---|---|\n| 82.84 **$\\uparrow$0.65** | 48.39 **$\\uparrow$1.50** |60.81 **$\\uparrow$2.09**|\n\n**Analysis:**\n- It can be seen that the **mAcc, mIoU, and mAcc are improved by 0.65, 1.50, and 2.09 respectively**, which is significant at this data amount. (For larger data ratio we are still running experiments and will update the results in the later updates. We will add thorough experiments in the revision. The code of this part will also be published.)\n\n**Conclusion:**\n- Info-Coevolution is generalizable to more tasks as analyzed.", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "To me, the proposed method is more like a hybrid active learning (AL) methods combining uncertainty-based AL (in this method, based on the decision boundary of a pretrained backbone) and diversity-based AL (in this method, based on the nearest neighbor data similarity) for information gain estimation. Therefore, I cannot read this work as a new paradigm for AL. It's more like an improvement work for hybrid AL. \n\nRegarding the method design, information gain has been widely recognized as an effective criterion for AL. The Bayesian Prediction Fusion of data similarity and model prediction is reasonable to me. It provides to some extent new insights for AL community towards the data-agnostic and model-agnostic AL. \n\nExperiments on ImageNet, CIFAR-10, CIFAR-100, Standfordcars, fool101 and SVHN are conducted. These are commonly used active learning datasets, so the classification-based AL experiments are generally adequate. \n\nHowever, only image classification task is evaluated. Showing results on segmentation/detection tasks would be more convincing.", "theoretical_claims": "I checked Theorem 3.1, and it looks correct.", "experimental_designs_or_analyses": "The paper only compares with one AL method, DQ (coreset-based), in Fig. 6. More AL methods could be compared, from the early entropy-based methods to recent AL methods. I understand the proposed method is both data-agnostic and model-agnostic, but it'd be better to include more AL methods to comprehensively evaluate its effectiveness.\n\nMore experiments on different semantic understanding tasks could be considered.", "relation_to_broader_scientific_literature": "The proposed method would benefit ML training by reducing the annotation and training costs on large datasets.", "essential_references_not_discussed": "No", "other_strengths_and_weaknesses": "None", "other_comments_or_suggestions": "None", "questions_for_authors": "None", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper introduces Info-Coevolution, a framework for efficient dataset construction and training through online selective annotation. Info-Coevolution enables coevolution of the model and dataset using task-specific models. The method selectively annotates streaming or web data, significantly reducing annotation and training costs without degrading performance. 

The paper presents an important contribution to efficient learning at scale. While one reviewer raised concerns about the theoretical underpinnings, these were satisfactorily addressed in the rebuttal and discussion. The approach is novel, the results seem strong and relevant, and the broader community would benefit. I recommend acceptance and encourage the authors to incorporate reviewer feedback in the final version.
Decision: Accept (poster)