A comparison of dataset distillation and active learning in text classification
Authors: Zhang Zeyu
Zhang Yuanchun
Venue: Submitted to ICLR 2023 (2023)
Keywords: knowledge distillation
dataset distillation
active learning
text classification

Abstract: Deep learning has achieved great success over the past few years in different aspects ranging from computer vision to natural language process. However, the huge size of data in deep learning has always been a thorny problem in learning the underlying distribution and tackling various human tasks. To alleviate this problem, knowledge distillation has been proposed to simplify the model, and later dataset distillation as a new method of reducing dataset sizes has been proposed, which aims to synthesize a small number of samples that contain all the information of a very large dataset. Meanwhile, active learning is also an effective method to reduce dataset sizes by only selecting the most significant labeling samples from the original dataset. In this paper, we explore the discrepancies in the principles of dataset distillation and active learning, and evaluate two algorithms on NLP classification dataset: Stanford Sentiment Treebank. The result of the experiment is that the distilled data with the size of 0.1% of the original text data achieves approximately 88% accuracy, while the selected data achieves 52% performance of the original data.


Under review as a conference paper at ICLR 2023
A comparison of dataset distillation and active learning in text classification
Anonymous authors Paper under double-blind review
Abstract
Deep learning has achieved great success over the past few years in dif- ferent aspects ranging from computer vision to natural language process. However, the huge size of data in deep learning has always been thorny problem learning the underlying distribution and tackling various human tasks. To alleviate this problem, knowledge distillation has been proposed to simplify the model, and later dataset distillation as a new method reducing dataset sizes has been proposed, which aims to synthesize small number of samples that contain all the information of a very large dataset. Mcanwhile, active learning is also effective method to reduce dataset sizes by only selecting the most significant labeling samples from the orig- inal dataset. In this paper, we explore the discrepancies in the principles of dataset distillation and active learning, and evaluate two algorithms on NLP classification dataset: Stanford Sentiment Treebank. The result of the experiment is that the distilled data with the size of 0.1% of the original text data achieves approximately 88% accuracy, while the selected data achieves 52% performance of the original data.
Keywords: knowledge distillation dataset distillation active learning text classification
Introduction
Deep learning whose neural network has large number of layers in order to solve complicate human tasks has achieved great success over the past few years in many various applications ranging from computer vision to natural language process. These networks have to be trained on a large number of data so as to learn the underlying distribution of the task. However, the huge computational complexity and the massive storage which deep learning requires are two main setbacks in the study. Therefore, data reduction has become a popular and feasible method to help people to learn the model and solve the problem by a small number of data.
Nowadays to improve the efficiency of deep neural networks, knowledge distillation HHinoon et al., 2015) has been proposed to compress the complex model into the simpler model and remain the approximately same accuracy as the original model. In knowledge distillation, the teacher model transfers the knowledge to the student model which has the less layers and storage. It asks the student model to achieve two goals: minimizzng the loss function between the teacher model's soft prediction (softmax(T=t) i.c. T is the distillation temperature) and student model's soft prediction which means the distilled model need achieve as much performances as the original model; minimizing the loss function between the student' 's hard prediction (softmax(T=1)) and the ground truth. Tt. actually plays a role of model compression in deep learning which reduce over-fitting and is widely used in data mining of infinitely large, unsupervised datasets; less or zero sample learning; migration learning and etc.
Different from knowledge-disteee distillation which transfers the complex model to the simpler model, data distillation (Wang et al), 2018) is targeted to encapsulate all the knowledge from a large number of data to a small number of data. It an alternative formulation to reduce the data sizes by synthesizing small number of data points containing all the

---

Under review as a conference paper at ICLR 2023
information of large dataset that do not need to come from the correct true dataset dis- tribution. Compared withhtther methods which aim to reduce the size of dataset such as prototype (Garcia et all, 2012) selection in nearest-neighbor classification, dataset distilla- tion aims to generate a small number of datasets from the original data rather than selecting samples from the true distribution. It depends on backpropagated gradient to compress dataset into a small set of synthetic data samples which is good for a complex neural net- work to be trained on a small number of data rather than the original massive dataset with the almost similar accuracy.
Sometimes, we can regard deep learning as situation that students always learn everything in detail from teachers in class while the amount of knowledge is very huge, however active learning is an effective way to learn parts of knowledge actively which is selected deliberately to master the knowledge to the maximum extent. Tt is a cyclic procedure bet ween a teacher or an oracle (usually a human annotator) and an active learner (Schröder & Nickler, 2020). Compared with passive learning which is only to feed the data to the model, active learning choose which samples to be labeled by a human expert, the so-called human in the loop.
In this paper, we discuss various discrepancies wetween two methods of reducing dataset sizes and use two algorithms to solve text classification on NLP classification dataset: Stanford Sentiment Treebank (SST) which consists of 21,5154 phrases from movies with fine- -ra-gga sentiment labels in the range of 0 to 1. We use the classification accuracy to compare the performances of active learning and dataset distillation on text classification.
2 Related Work
2.1 Knowledge Distillation
Hinton et al. have proposed knowledge distillation as a method for imbuing smaller, more efficient networks with all the knowledge of their larger counterparts (Hinton et al. 2015). And then knowledge distillation has been used in various aspects. Linfeng Zhang et al. have proposed self distillation, which notably enhances the accuracy of convolutional neural networks through shrinking the size of the network rather than aggrandizing it (Zhang et al, 2019) .o mitigate the massive number of parameters in deep neural networks, Sukmin Yun el al. have proposed a new regularization method that penalizes the predictive distribution between similar samples and distill the predictive distribution between different samples of the same label during training (Yun et al. 2020) For casual distillation in language models, Zhengxuan Wu et al. have proposed to augment distillation with a third objective that encourages the student to imitate the cansal
cansal computation process of the teacher through interchange intervention training (Wu et al. 2021
2.2 Data Distillation
Data distillation has been proposed firstly by Tongzhou Wang alternative formulation to synthesize a small number of data from the massive original datasct (Wang et al,, 2018). It has been used in various industries. Tian Dong et al. have explored that datasct distilla- tion is a better solution to replace the traditional data generators for private data generation because the-distilles data set of synthetic data which will not disclose the original in- formation (Dong et all. 2022). Shengyuan Hu et al. have proposed a new scheme which includes gradient compression via dataset distillation in federated learning for upstream communication where every client learners use a light- distilled dataset as the train- ing data instead of transmitting the model update (Hu al,, 2022). To reduce catastrophic forgetting in deep neural networks, Wojciech Masarczyk et al. have investigated the use of synthetic data to generate two-step optimisation process via meta-gradients (Masarczyk & Tautkute, 2020). To alleviate the storage and time for training neural models on large-scale graphs in real world applications, Wei Jin et al. have proposed the condensation problem by imitating the GNN training trajectory on the original graph through the optimization of a gradient matching loss and design a strategy to condense node futures and structural information simultaneously (Jin et al., 2021). Yongqi Li et al. have developed a novel data
2

---

Under review as a conference paper at ICLR 2023
distillation method for text classification and evaluate their method eight benchmark
datasets [i Li 2021
2.3 Active Learning
In the era of the deep learning explosion, various tasks and applications are considered to be solved using data-driven learning approach, which places even higher demands data. In practice, it is not possible to abandon either labelling or unlabeled data altogether, and active learning labeling it all, selectively. For low-data regimes and on self- self-trained vision transformers, Amin Parvanch et al. have proposed novel method for batch active learning whose performances are espocially significant (Parvanch et al, 2022). To remove label noise by fully re-annotating al. have
provide
reasonable expedient to label valuable data without
large datasets is infeasible in cooource-coussrainneed settings, Melanie Bernhardt proposed to use active label cleaning Lo rank inst ances and introduce simulation framework to evaluate relabeling efficacy (Bernhardt. et al, 2022). Yuhang Zhang et al. have presenled new active learning framework, which for the first time incorporates contrastive learning into recently proposed one-bit supervision, which is more efficient than previous active learning methods requiring assigning accurate labels to the queried samples (Zhang et al 2022).
3 Methodology
3.1 Data Distillation
Data distillation pay more attention transferring knowledge and information from the original dataset to the distilled data, rather than reconstructing samples from the true distribution. Therefore, we have to consider the connection between the distilled data and the original data, which can be linked with the gradient (Wang et al., 2018). When we use the distilled data to train the model, the weight of the network is from various functions from synthetic data, and backpropagated gradient can be used to track the original data to close the gap between the distilled data and the original data. For text. dataset, the original data and the distilled data can be settled, then the distilled data can be seperated into several batches
train the model by applying the gradient descent we find the appropriate function which imput the distilled data and output the well-trained model, which means that when we divide the dataset into batcher proper way then we can use the distilled data to train the neural network as the normla text data.
these batches. Whil While
Knowledge distillation
Big Data
training
teacher model
compressing
student model
small
Big Data
compressing
training
small network
Data distillation
Figure Illustration of knowledge distillation and data distillation
3

---

Under review as a conference paper at ICLR 2023
As for data distillation, considering training dataset {- parameterize our neural network as and denote 1(,,,)) as the loss function that represents the loss of this network data point 2z- The target of data distillation is to find the minimizer of the empirical error over entire training data:
N Σ N
0" = arg min
Uz arg min (x,
17 Applying the data distillation in text classificatio, considering text training datasct T {pi, 9i /: where pj a piece of text and is the corresponding label of the text, and M is the number of samples in original dataset. It is noticeable that the lext data is discrete, may use mathematical matrixes rather than these discrete words to form the distilled data.
The current parameters be updated with the learning rate 7 and the loss function follows:
0.t1 -, 0, (1,,
However, it takes number of update steps in the training process, so we use the data distillation. It means that we aim to learn small number of synthetic distilled training data then we can apply random gradient descent on every minibatch to find the optimal.
The distilled dataset Ī {p: q where N << M. Due the standard training usually applies minibatch stochastic gradient descent or its variants, for text classification, we randomly initialize the distilled data as To and divide them into various minibatches ( where S1 is the number of batches. Then, when inputing a minibatch of data, the parameter is renewed by the learning rate 7 and the loss function:
8 = (f.
Therefore, the basis of the parameters and the initial 0,, obtain synthetic data D. which minimizes the loss function:
D. arg min ((p, 01) = arg min(p, 0, - -j (F,
D
where we derive the new weights as function of distilled data D and learning rale using Equation and then evaluate the new weights over all the training data D. The loss function ((, 0o) is differentiable, and can thus be optimized using standard gradient-based methods.
use ( to make To more close to Ts. then T' can be used as the original data to train the neural network.
Afterwards,
3.2 Active Learning
whather as required
apply
model
modell
manual labeling by query function
Figure 2: process of active learning
4

---

Under review as a conference paper at ICLR 2023
As an effective method for reducing sizes of dataset, active learning is targeted to label the most important data from the overall dataset, which means that it is kind of like choosing the sample from true distribution and reconstructing life-like sample rather than using the part of original dataset.
As the figure2 illsutrates that the first step of active learning is gathering the data. After- wards, some data are selected to train the model while at this time, the training model's accuracy is not good enough. Therefore, query functions determine data selection choices of the model in the AL loop. They are used in manual labeling which means that human experts apply the query functions to choose the corresponding label and choose more labeled data to train the model. When the accuracy is as required, the model can be utilized in real human tasks.
We considered multiple methods of active learning in text classification follows (Jacobs ct al., 2021).
(1) Variation Ratio:
The variation ratio measures dispersion around the class. The below equation shows how the variation ratio iscomputed, where fa denotes the mode count and F the number of stochastic forward passes.
Jn F
u[a] =
Actually, variation ratio is a form of predictive uncertainty rather than capluring the un- certainty in the model.
(2) Predictive Entropy:
In mathematics, entropy can be used to measure the uncertainty of a system, with higher entropy indicating greater uncertainty and lower entropy indicating less uncertainty. There- fore, in a dichotomous multiclassifcattion scenario, those sample data with higher entropy can be selected as the pending annotation data. Entropy H(x) has been defined as:
n
H(x)
Σ p(xx))og22(a
where p(x;) gives the probability of the th possible value for the symbol. Entropy is used to quantify the information of data. We can compute the chance of the model classifying data point by averaging over the softmax probability distributions across the T stochastic forward passes.
H 22, Dtrain]= D cl dp, 1 T Σplu= -ol
where wy denotes the stochastic forward pass f, and e the number associated the class- label.
Bayysiin Active Learning by Disagroement:
This is a form of conditional mutual information, the condition or the third variable being the training data Dtruin- Houlsby et al. used this form of mutual information in an-active learning setting and dubbed it Bayesian active learning by disagreement (BALD) (Houlsby et al. 2011).
Tyr w|a, Devain) = =- ---( Σ p(u c/s, Noy)
F Σ -rv-= c(,,W))
1
Σ plu c/., (l wiygggy cl,,
F
5

---

Under review as a conference paper at ICLR 2023
4 Experiment
4.1
Data
SST consists 11,850 sentences and 5 classes of data in the following way:
Table 1: An overview of SST in the experiment
Datset
Classes
Train Samples
Test Samples
SST
10534
1316
The reason why choose SST the experiment dataset is that its size is massive and can be a benchmark for language models. It allows for the evaluation of AL and data distillation for larger dataset and for comparison with results found in related work.
To show the efficiency of the distilled data in text classification, we compare the accuracy between the original data and the distilled data on the same test set. Here, we choose 0.1% of the dataset size applying in the distilled data and random samples.
As is known to us all, the most important factor in active learning to choose appropriate query function to solve the task in required accuracy. In this paper, we use various query functions active learning text classification and compare the performance with random sampling using 1% of the dataset size (Jacobs et al. 2021
4.2 Experiment Results
Table 2: The result of data distillation
Method
SST
Original Data
0.9859
Distilled Data
0.8799
Random Data
0.6290
0.9
Distilled Data
Random
0.8
0.6
Accuracy
0.5
0.3
0.2
2
Epoch
Figure 3: lcarning rate of data distillation
In this paper, 0.1% of the original training data the number of samples in the random data and the distilled data. The result of data distillation in this experiment is shown as table2 and learning rate is shown figure3.

---

Under review as a conference paper at ICLR 2023
The distilled data achieve about 88% accuracy and random data only achieve 63% accuracy. It is noticeable that although the distilled data cannot perform as well as the full data, the size of distilled data is much smaller than the full data with only 0.1% of the original data.
From the figure3 above, it observed that training neural network on the distilled data is more casily than that the random data. Looking at the red line which denotes epoch of distilled data rises fast to reach its highest level, while the blue line denoting the random data has slowly increase. It means that data distillation an effective way train deep neural network in text classification.
Jacobs has discussed about yarious query functions in active learning in SST and the result is shown below (Jacobs et al, 2021h. It is clear that on this dataset, active learning does not perform much better than random sampling. During 15 epochs, the gap between random sampling and active learning is really small. Therefore, active learning does not seem to have a significant performance text classification.
SST: Active Learning Query Functions vs. Random Sampling
54
52
50
48
Accuracy
Random Sampling Variation Ratio Predictive Entropy BALD
1000
2000
3000
4000 Size of
5000
6000
7000
8000
Figure Active Learning
4.3 Data Analysis
It is noticeable that data distillation outperforms active learning in text classification with 88% accuracy of the original data. Here, we demonstrate the differences between active learning and data distillation.
(l)s
The basic algorithm of data distillation is similar with knowledge distillation which is more like data compression using gradient descent. Thus, data distillation is able to maintain all information and knowledge from the original data. In addition, backpropagated gradient is the most effective method on data distillation. When training neural networks on a small number of data among a large number of data can achieve the optimal result, it is effective to use backpropagated gradient.
For active learning, it is necessary to compare various performances from different query functions. One of the most noticeable defects which compare with data distillation is that manual label leads to human disturb. Human labeled datasct in active learning is from true datasct distribution but not maintaining all information and knowlodge from the true dataset. In addition, noise and outliers in large amounts of data. will always be picked as the labeled dataset, which bad for model training
Hard to transfer:

---

Under review as a conference paper at ICLR 2023
As for active learning which is data selecting strategy has difficulties in generality, It is not able to be generic among different tasks with various dataset distribution and labels, and in many occasions people need to spend more time in designing the specific active learning strategies for various human tasks.
Speaking of data distillation, it is general method to reduce the dataset size. Among varions industrics, applying data distillation bases on a general way including selecting the distilled size of data and gradient descent. Therefore, when facing different dataset distribution and labels, data distillation play a significant role in generality.
(3)Unstable performance:
Data distillation focuses on transferring all knowledge and information from the original dataset to the distilled data using gradient descent to update parameters and find the most optimal distilled dataset which aims to minimize the loss function. Although the distilled data performs worse than the full original data, it does outdo the performance of active learning and shows the stable performance.
Compared with data distillation, active learning is used by its specified selection strategy, thus query function and the nature of the dataset are two main factors during the process. In practice, people need first select and label data based on active learning, and if the strategy is not as good as random sampling at this point, they are not able to change or stop in time because the data has already been labeled and the sunk costs have already been incurred.
Active learning shows unstable performance among different datasets which own different distribution and labels, therefore when people want to solve human tasks in an efficient way, active learning is not as steady as data distillation.
Conclusion
In this paper, we explore data distillation which aims to distill all information and knowl- edge from the original data to small number of synthetic dataset. We compare different performances between data distillation and active learning in text classification and discuss about discrepancies between two algorithms. The result is that the distilled data with only 0.1% of the original data achieve 88% accuracy, while active learning only performs 52% accuracy of the original data.
In conclusion, although data distillation has not been used in many industries, many surveys show the efficiency of data distillation in machine learning. We believe data distillation will play a significant role in machine learning and deep neural network in the future.
References
Mélanic Bernhardt, Daniel C. Castro, Ryutaro Tanno, Anton Schwaighofer, Kerem C. Tezcan, Miguel Monteiro, Shruthi Bannur, Matthew P. Lungren, Aditya Nori, Ben Glocker, Javier Alvarez-Valle Valle, and Ozan Oktay. Active label cleaning for improved datasel quality under resource constrainis. Nature Communications 13(1). mar 2022. doi: 10. 1011100414667---00 27770-82--88-333 URL htlps://doi.onr http://...033//222514416--000000
Tian Dong, Bo Zhao, and Lingjuan Lyn, Privacy for free: How does dataset condensation help privacy?, 2022. URL https:/ /ttpss//axxvoooae www.ariwwioggaaaaaa 222222200020000
Salvador Garcia, Joaquin Derrac, Jose Cano, and Francisco Herrera. Prototype selection for nearest neighbor classification: Taxonomy and empirical study. IEEE Transactions Pattern Analysis and Machine Intelligence, 34(3):417-4355 2012. doi: 10.1109/TPAMT. 2011.142.
Geoffrey Hinton, Oniol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network, 2015. URL. https://arxiv.org/abs/////
8

---

Under review as a conference paper at ICLR 2023
Neil Houlsby, Ferenc Huszár, Zoubin Ghahramani, and Mäté Lengyel Bayrsian active learning for classification and preference learning, 2011. URL https: /aap/aaxxioorrrrr..e / 1112.5745.
Shengyuan Hu, Jack Goetz, Kshitiz Malik, Hongyuan Zhan, Zhe Liu, and Yue Lin Fedsynth Gradient compression via synthetic data in federated learning, 2022. URL https:/ /arxiv. prg/abs/200..0122 2204.01227
Picter Floris Jacobs, Gidcon Maillette de Buy Wenniger, Marco Wiering, and Lambert Schomaker Active learning for_rrrdreeeeeeee reducing labeling effort in text classification tasks, 2021. URL hutps://axii..or//as////00..000
Wei Jin, Lingxiao Zhao, Shichang Zhang, Yozen Liu, Jiliang Tang and Neil Shah Graph condensation for graph neural networks, 2021. URL httpss//arri..orrgabbb///////// abs ottp 21100075880
Yongqi Li and Wenjie Li, 2021. URL https:/ http:p//aaaa..orrr/bb/b20/00444
Wojeiech Masarezyk and Ivona Tantkute Reducingg colastoophc forgetting with learning on synthetic data, 2020. URL https://arrii.orr/as///000....
Amin Parvanch, Ehsan Abbasnejad, Damien Teney, Reza Haffari, Anton van den Hengel and Javen Qinfeng Shi. Active learning by feature mixing, 2022. URL https: /aaxiaxxiivoorr/.. abs/2203.0703.
Christopher Schröder and Andreas Niekler A of learning for text classification using deep neural networks, 2020. URL https://arxiv..r www.:oor///bbboa hatps///0000020500
Tongzhou Wang Jun-Van Zhn Antonio Torralba, and Alexci A. Efros. Dataset distillation, 2018. URL httpp:/ Attps://arxiv.oou /ar//aaxvvvrrrggrr orgg /ass///a////// 1811.
Zhengxuan Wu, Atticus Geiger, Josh Rozner, Elisa Kreiss, Hanson Lu, Thomas Icard, Christopher Potts, and Noah D. Goodman Causal distillation for language models. 2021. doi: 10.48550/ARD AA/// TTXXTV2112.025511 URL htttps///ariitn httpps//aaxxv..or//aaaaaaa wwtp:///////bbb//.//22222
Sukmin Yun, Jongjin Park, Kimin Lee, and Jinwoo Shin Regularizing class predictions via self- eelflknoweedge distillation, 2020. URL https:: https://arxiv.org.. abs /btp//abbb/003333944
Linfeng Zhang, Jiebo Song, Anni Gao, Jingwei Chen, Chenglong Bao, and Kaisheng Ma. Be your own leacher: Imprrve the performance of consolutional neural networks via self distillation, 2019. URL https://arxiv.org//aae 19t.l////900000009444
Yuhang Zhang, Xiaopeng Zhang, Lingxi Xic, Jie Li, Robert C. Qiu, Hongtong Hu, and Qi Tian. One-bit active query with contrastive pairs. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 9697 9705, June 2022.


Reviews:
[{"review_id": "f6hsWRyibUG", "reviewer": "Reviewer_8K6w", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper introduces and compares discrepancies of Data Distillation and Active Learning and evaluate them on NLP classification dataset: Stanford Sentiment Treebank.", "summary_of_the_review": "This is a good introductory article to compare Data Distillation and Active Learning. But it did not propose any new methodology or dataset. And there are no new dataset proposed nor enough experiments to serve as benchmark.", "strength_and_weaknesses": "Strength:\n1. This is a good introductory article to compare Data Distillation and Active Learning.\n\nWeakness:\n1. This paper did not propose any new methodology or dataset.\n2. There is only one evaluation dataset and two sets of experiments.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "Op9y-I40vW", "reviewer": "Reviewer_wkGv", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper provides a comparison of data distillation and active learning for text classification. The models used in the paper are the most common types of distillation and active learning models. Experiments are performed on the ST dataset. Discussions are offered on the results.", "summary_of_the_review": "No novelty. Strightforward comparison between distillation and active learning. Limited results and discussions.", "strength_and_weaknesses": "Strengths:\n1. Good references to past papers that propose the ideas.\n2. Clear introduction on the technical details.\n\nWeaknesses:\n1. No novel ideas are proposed in the paper.\n2. Only one small dataset is used for experiments.\n3. Discussions lack depth.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "8lRZ93fmaT8", "reviewer": "Reviewer_WRBC", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper compares dataset distillation and active learning in text classification. Authors introduce basic methods and conduct experiments on the SST dataset. ", "summary_of_the_review": "This paper is below the standard of ICLR. ", "strength_and_weaknesses": "The research objective of this paper is neither clear nor reasonable. Experimental results of active learning are not reliable. The conclusion is not reasonable based on the simple experiments. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "1: The main claims of the paper are incorrect or not at all supported by theory or empirical results.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject