Self-Guided Noise-Free Data Generation for Efficient Zero-Shot Learning
Authors: Jiahui Gao
Renjie Pi
LIN Yong
Hang Xu
Jiacheng Ye
Zhiyong Wu
WEIZHONG ZHANG
Xiaodan Liang
Zhenguo Li
Lingpeng Kong
Venue: ICLR 2023 notable top 25% (2023)
Keywords: Pre-Trained Language Model
Prompt-Based Learning
Efficient Zero-Shot Learning

Abstract: There is a rising interest in further exploring the zero-shot learning potential of large pre-trained language models (PLMs). A new paradigm called data-generation-based zero-shot learning has achieved impressive success. In this paradigm, the synthesized data from the PLM acts as the carrier of knowledge, which is used to train a task-specific model with orders of magnitude fewer parameters than the PLM, achieving both higher performance and efficiency than prompt-based zero-shot learning methods on PLMs. The main hurdle of this approach is that the synthesized data from PLM usually contains a significant portion of low-quality samples. Fitting on such data will greatly hamper the performance of the task-specific model, making it unreliable for deployment. Previous methods remedy this issue mainly by filtering synthetic data using heuristic metrics(e.g., output confidence), or refining the data with the help of a human expert, which comes with excessive manual tuning or expensive costs. In this paper, we propose a novel noise-robust re-weighting framework SunGen to automatically construct high-quality data for zero-shot classification problems. Our framework features the ability to learn the sample weights indicating data quality without requiring any human annotation. We theoretically and empirically verify the ability of our method to help construct good-quality synthetic datasets. Notably, SunGen-LSTM yields a 9.8% relative improvement than the baseline on average accuracy across eight different established text classification tasks.

Published as a conference paper at ICLR 2023
SELF-GUIDED NOISE-FREE DATA GENERATION FOR EFFICIENT ZERO-SHOT LEARNING
Jiahui Gao'; : Př Yong Lin', Hang Xu', Jiacheng Ye'.
Zhiyong Wu', Weizhong Zhang,, Xiaodan Liang", Zhenguo Li', Lingpeng Kong' The University of Hong Kong HHong Kong University of Science and Technology Huawei Noah's Ark Lab *Shanghai AI Lab Fudan University Sun Yat-sen University {sumiler, carsonye] ) @connect. lpk0cs hku hk, yonghi org. en, (xu. hang, li, zhenguo H @huawei...... com, Shangweizhonger xdliangg2e 328 @egmall....o com
(rpi, ylindf ) &connect. ust.hk,
ABSTRACT
There is rising interest in further exploring the zero-shot learning potential of large pre-trained language models (PLMs). A new paradigm called data- setiaaaeenration-- based zero-shot learning has achieved impressive success. In this paradigm, the synthesized data from the PLM acts as the carrier of knowledge, which is used to train task-specific model with orders of magnitude fewer parameters than the PLM, achieving both higher performance and efficiency than prompt-based zero-shot learning methods on PLMs. The main hurdle of this approach is that the synthesized data from PLM usually contains a significant portion of low-quality samples. Fitting such data will greatly hamper the performance of the task- specific model, making it unreliable for deployment. Previous methods remedy this issue mainly by filtering synthetic data using heuristic metrics(e.g. output confidence), or refining the data with the help of a human expert, which comes with excessive manual tuning or expensive costs. In this paper, we propose a novel noise-robust re-weighting framework SUNGEN to automatically construct hugh-quality data for zero-shot classification problems. Our framework features the ability to learn the sample weights indicating data quality without requiring any human annotation. We theoretically and empirically verify the ability of our method to help construct good-quality synthetic datasets. Notably, SUNGEN-LSTM yields a 9.8% relative improvement than the baseline on average accuracy across eight different established text classification tasks.
INTRODUCTION
Owing to the superior generative capacity of large-scale pre-trained language models (PLMs), there has been an emerging trend of using these powerful models (c.g., GPT) to generate training data for downstream tasks (Anaby-Tavor et al. 2020 Puri et al., 2020 Kumar et al.) [2200] Lee et al. 2021
inter alia). Among them, new line of generation-bused zero learning using the unfinetuned
PLM pushes the envelope further (Schick & Schütze 2021 Ye et al. 2022a Meng et al.) 2022).
featuring total annotation-free training for downstream tasks. Ye et al. (2022a) (ZEROGEN) further
boosts the efficiency by using the generated data to train tiny task models (TAM), which have orders- of-magnitude fewer parameters than the PLM. Specifically, they first design prompts incorporating the task description and label information, then use them to guide the data generation from the PLM. Subsequently, the synthesized dataset is used to train the tiny task specific models. Compared with the classic prompt- based zero-shot learning on PLM, this new paradigm enjoys two favorable properties: (1) since the task model has orders-of- -ow--n-ituaaaeeeeee-e fewer parameters than the PI.M, demonstrates much lower inference latency; (2) with the large amount of -egeerree training data, the task model often shows better performance than prompt- zero-shot PLM counterparts.
In the above paradigm, the amount and the quality of the generated data are crucial factors for the task model's performance. Unfortunately, despite the unlimited training data that one can generate
"Equal Contribution. Code is available this link

---

Published as a conference paper at ICLR 2023
in theory, the data quality is not always guaranteed. Our experimental observation across many downstream tasks verifies the existence of this issue: in ZEROGEN, after a few training epochs on the PLM-generated dataset, although the training accuracy steadily improves, the actual test accuracy of the model starts declining rapidly (e.g., IMDb in Figure
a clear indication of the model overfitting to low-quality data (noisy data) (Arpit et al. 2017 More specifically, we identify two major cases of noisy samples in the synthetic dataset: corrupted labels and task-irrelevant samples (Table Appendix). Without any task-related fine-tuning, it is challenging for PLM follow user's instruction (task-specific prompt including label information) to generate accurate samples in the target domain (Ouyang et al., (022). To alleviate the data quality issue, recent work adopts human-active labeling to correct the corrupted label or revise the example (Wang et al.. 202 ct al. 2022). However, such methods introduce considerable costs and may be unrealistic.
Liu
To avoid human intervention, the classic ap- proach to eliminate the effect of noisy data is to re- re-weight the samples. The core idea is to design a weighting function w, such that the cor-
Synthetic Data for IMD6 (100k)
Train Acspt)
SunGen
rect samples associated with larger weights and the noisy ones with smaller weights. Com- pared with heuristic design of (e.g.. according to output confidence, loss value) (Liu & Tao 2015 Wang et al., 2021b), which requires task- specific knowledge and excessive manual tun-
8
Test Aecrr)
ZeroCien
SunGen
ing, the adaptive methods that learn the sample weights in an end-to-end manner demonstrate better performances in practice (Ren et al. [018 Shu et al.] 2019;; Zheng et al. [2021). Those methods typically formulate the learning sam- ple weights into a bi-level optimization problem. with a clean validation set in the outer loop to guide the learning of w. Despite remarkable success was achieved, their dependence validation set becomes a major limitation, which is especially impractical in zero-shot setting.
Epoch
Figure 1: Training and testing accuracy of LSTM model trained on synthetic dataset. After training for more epochs, the testing performance of ZERO- GEN starts deteriorate significantly, indicating that the model starts to fit the erroneous data. clean
Our solution comes from rethinking the choice of the outer objective in the bi-level framework: can we design an objective such that the sample weights can be optimized with only access to the noisy synthetic data? To this end, we resort to a family of noise-robust loss functions (Erbust (Ghosh 2017: ZZhag & Sabuncu 2018). These functions were adopted by previous work to train the neural network under label noise due to their theoretically noise-tolerant property (Ghosh et al.) 2017 Zhang & Sabuncu 2018 Wang et al. 2019). However, from the optimization point of view, such loss functions suffer from instability and difficulty when training the neural networks (Zhang & Sabuneu 2018), which limits their effectiveness. Remarkably, our approach leverages the noise-tolerant property of these losses, while avoiding their pathology. We propose a novel bi-level re-weighting framework SUNGEN: in the inner loop, we train the task model using weighted training loss based on current sample weights; in the outer loop, the noise-robust loss is adopted to guide the learning of the sample weights. The two procedures are performed alternatively to generate a set of weights indicating the importance of samples. Notably, our method focuses on enhancing the quality of generated data, while improving the generator (e.g., modify PLM parameter, prompt engineering) is an orthogonal direction and can be applied jointly with our method.
cet al.
Our main contributions threefold. First, we propose a novel end-to-end framework construct high-quality noise-free synthetic dataset, without the aid of any human annotation ($3). Second, we offer theoretical justification ($4) and empirical verification ($5.2) for SUNGEN'S ability of recovering a noise-free dataset reliably with synthetic data only. Third, we conduct experiments on eight text classification datasets and show our method outperforms the current baseline by large margins ($5.2).
2 BACKGROUND
2.1 PROMPT-BASED ZERO-SHOT LEARNING
We first introduce crompt-based zero-shot prediction (named PROMPTING). Given a manually- designed prompt T() and query example x; E X, PROMPTING constructs a sentence T(x.) (e.g..
2

---

Published as a conference paper at ICLR 2023
Update 0
Inner Loop
Prompt
positive
The movie review
leg. GPT-3)
Synthetic Training Data
TAM LSTM)
Weighted CB Loss
S
Examples of Synthetic Data(with noise)
Parameter Sharing
Sample Weights
(One of the best films have seen. Pos.)
(00 MPp)
(The film poarly constructed. Neg.)
(2) Outer Loop
(Nothing much the movie me. Pos.)
Update
(Ray's performance is an absolute delight. Neg.
Ingut:
In Train
Validation Set
Synthetic
TAM LSTM)
Noise-Robust Loss
Generate data::
Skye
Update puramoter:
Validation
Figure 2: The framework of SUNGEN. Our bi-level framework learns sample weights w measuring data quality without relying on any human-annotated data. the inner loop, we train tiny task model (TAM) with weighted CE loss based on current sample weights, and produce trained TAM parameters 0( (w); in the outer loop, adopt noise-robust loss to guide the learning of by evaluating ((w) on synthetic validation set.
"The movie review <y;> sentiment is: <x,> "). The PLM P is expected model the probability distribution of a set of label words Yi E ) (e.g.. "positive". "negative") and select the class with highest probability for x, Even though PROMPTING has achieved remarkable success in zero-shot learning (Radford et al., [2019] Brown et al. 2020), it is difficult for PROMPTING to fully leverage the task-specific knowledge from PLMs. Besides, PRoMPTING still needs to conduct inference on a cumbersome PLM.
2.2 EFFICIENT ZERO-SHOOT LEARNING VIA DATA GENERATION
new line research (Ye et al. 2022a Meng et al. 2022 Ye al. 2022b endeavors to make zero- shot learning practical and efficient, among which the generative efficient zero-shot learning paradigm proposed by ZEROGEN (Ye et al. 2022a) is as follows:
Synthetic Data Generation. Given task, the paradigm first generates a synthetic dataset Syy (Nayn: Vsyn) with the help of large-scale PLM and task-related prompts. The idea is to use model P to generate the input X.yn based on pseudo label M/yn- For example, in text classification task, class label Bsyn uniformly sampled: ylyn U(y ; 1/.............................. .k), where K is the number of classes. The pseudo label Ysyn is transformed into a label-descriptive prompt (14ay) to generate Xsyn: Xsyn ~ (K-IT(Us))) The generated Xsyn and pseudo label ysyn are paired to construct a pseudo training dataset S-yn-
Efficient Training and Inference. achieve efficiently training and inference, the paradigm then trains tiny task model (TAM) (e.g., 1-layer Bi-LSTM) on the synthetic dataset Ssyn- TAMs typically have much fewer parameters, which is thus easy to train and more efficient during inference.
Although ZEROGEN achieved promising results by training TAM with Ssyn we find that model's performance rapidly declines after several training epochs, indicating overfitting to noisy samples (Figure D A classic approach to improve the quality of a dataset containing low-quality data is to re- weight the samples using some function w. Intuitively, if we assign large weights to correct samples and small weights to noisy can he reduced. However, the heuristic design of w (e.g., according to output confidence, loss value) suffers unstable performances and requires task-specific knowledge (Shu al. 2019). Therefore, paper seeks to optimize w automatically.
the negative influence of noisy samples during training
3 METHOD
To automatically lear sample weights, the bi-level optimization approaches (Ren al. [018 Shu (et al.; 2019) have been proven to effective. However, such methods are impractical in our zero-shot scenario, since they depend clean validation set guide the optimization of sample weights To
3

---

Published as a conference paper at ICLR 2023
cireumvent the absence of human-labeledd data, we propose Self-gUided Noise-free data GENeration framework, named SUNGEN (Figure (. Concretely, we propose a sample reweighting framework via bilevel optimization using a noise-robust loss (Ghosh et al. 2017 [aang et al. 2019: Zhang & Sabuncu 2018) as the outer objective. Due to the appealing property of such loss functions, our method is able to learn meaningful sample weights with only synthetic (noisy) validation dataset.
/aag
Notations. As elaborated in Section we first generate a synthetic dataset using a left-to-right PLM and task-related prompts for the given task. Let Ssyn Solean C. X 3 Rd x {1,..., K} denote distribution of synthetic (noisy) and clean (gold) data, respective. Here is the dimension of the input space and K is the number of classes. We draw training dataset Syy == {(x, () V and a validation dataset Sin == {(.r, 1 from Soyn- Denote [(x, 0) as the classifier (TAM) with as its parameters. w E W : (.(()))) xxx x - [o, 1]} is a re-weighting function that assign a weight to each sample. The bold w is sample weight vector of length N indicating training samples'
importance, which contains per-sample weight W/ == w(x, yr).
Overall Framework. Without the presence of clean validation set, our proposed bi-level optimiza- tion framework SUNGEN (as shown in Figure 2 can be outlined as follows:
w E arg min Cacma (0(w), Sign) = arg mim
1
Σ
Eratus (a ((w))).)
(1)
M
(
s.t. ((w) E arg min (8, Shyn(w)) = arg min
Σ
w(x
(2)
where w* denotes the optimal sample weights, which is obtained from the outer loop (Eqn. D ((r) denotes the classifier's parameters after weighted training with w, which is obtained from the
inner loop (Eqn. 2) frobust denotes the noise-robust loss calculated on validation set; foe denotes the cross-entropy
loss calculated on training set. The whole process is: (I) in the inner loop 2
we fix and optimize the classifier f(x 0) using the weighted loss Log over synthetic training set Siyn and derive the trained classifier (ww); (2) in the outer loop(Eqn. L we calculate noise-robust
loss Lrubust on the synthetic validation set Sym at the optimized ((r). The outer loss is minimized to guide the optimization of w, and the outer gradient Crouust is calculated via truncated back- propagation(Appendix) H.. The two procedures are performed alternatively for ' iterations.
Notably, our framework prevents the need for clean validation set, which the main ob- stacle for the previous bi-level re-weighting approaches under zero-shot scenario (Ren et al. 2018; Shu et al.] (211).. The magic this appealing feature lies in the choice of the outer objective Lmshust in Equ. (I). Recall that the goal of sample reweighting is to find w", with which the model f(x; 0) can be trained using weighted loss over the synthetic training set, such that the model performs well the clean data (from the same distribution as test data). In Sec. 4 under the condition that the majority of data are correctly labelled (Ghosh et al. 2017 Wang et al. 2019), we theoretically show that using Crobust as the outer objective, our method can find a set of sample weights w" with just the synthetic validation set, such that w² maximizes the model performance on the clean data. The whole procedure is illustrated in Algorithm [
Algorithm 1 Bilevel Robust Sample Reweighting
Require: a TAM with parameters 9, synthetic training dataset S and validation dataset Syy, outer step size X, outer iterations T.
1: Initialize sample weightings w with each w;
to be 0.5.
2: for training iteration 1,2... ....TTo do
3: Conduct weighted training (Lov) on TAM using w',St and obtain 0 (ww"
Noise-robust Loss Functions. In the noisy label learning literature, there is a family of loss functions that possess the following property:
4: Evaluate O(w²) on Shy then obtain meta gradient V Crobus via Eqn. (II..
Reorsal (f fxx 0). C, 60,
5: Update u using V Crohuue by gradient de- scent w 4- w² Lm
where [(, 0) denotes classifier, is the input, K is the number of classes, and C is a con- stant. Previous work (Wang et al.) 2019 Zhang & Sabuncu, 2018: Ghosh et al. 2017) has shown that these loss functions have consistent global minimum under label noise. More formally, when the majority of the training samples are correctly
6: end forOutput: Optimized sample weights "
4

---

Published as a conference paper at ICLR 2023
labelled, the global minimizer (0*) of Lobust is the same regardless of whether the training data is clean or noisy. This noise-robust property enables these losses to optimize the sample weights given only noisy validation dataset. Detailed proof will be given in Sec.
In particular, we consider the reversed crosspyntropy loss foe with the following form for sample (xr, y):
K Σ (a log ((k)))).
Lae (f
where fk( =
cf Em th denotes the predicted probability for each label i E {1, K}. and 2;
are the logits. We denote the ground-truth distribution over labels by (kkaar), and Σ (k<((z)) (z) 1. Given the ground-truth label is y/ then ((y(æ)) = and ((k) x)) = 0 for * y. log is approximated to constant A. One can easily check that fee satisfies Property and C
-(K- A) in this case.
Remark. Even though foobust is theoretically noise-tolerant, it has been shown that using them to train the network parameters leads difficulty in optimization and hampers the network's performance (Wang et al.] 2019 Zhang & Sabuncul 2018). Remarkably, adopting robust as the objective in the outer loop Eqn. (1 implicitly overcomes the side effects of robust- Since fobust is now used to optimize the sample weights w, which is in much smaller search space and has simpler structure than O, thus is easier to optimize. This "decouples" noise removal from network training, and thereby prevents hampering the TAM's performance.
Clean Subset Sampling. Our framework enables us derive a of continuous weights w, which encodes the data quality and can well separate the noisy samples from clean ones, as shown in Figure 3 With those weights, we can sample subsets with arbitrary budgets and use them to train the task models with unweighted CE loss. More specifically, suppose the budget is D, we first normalize w; to w/ (i.e. Σx= w/ = D), based on which we then sample Bernoulli random variable I Ber(w,) for each sample to indicate whether they should be included. Denote the D - 1 when n - oc, meaning that Đ is close to D. Notably, because the important and frese-free data are associated with larger weights, training on the sampled subset can perform on par with weighted training over the entire dataset, while being much more efficient.
size of sampled subset as D. Because By- || = Ei= wy = D, it is clear that D
4 THEORETICAL ANALYSIS
Though we have no clean data as validation set, our method still enjoys favorable theoretical properties. Recall that Selcan and Sayo are as the clean and synthetic (noisy) distribution, respectively. We further denote L(O,S) S) = E( sss [ (x; 0), y)] given data distribution S. Let the optimal network parameters obtained with CE loss over clean dataset be 0* 0= the 0" unique, or we focus on the 0* with minimum norm.
We assume :- arg ming Lee Selean).
Property We call a loss function & a robust loss, under mild conditions in (Ghosh el al. its minimizer on the noisy dataset collides with the one on the clean dataset, i.e.,
2017
arg min Lrobuaa (0, Scleau) = arg min Crobuse (0, S,S.))
Previous work on noise-robust loss functions (Ghosh et al.; 2017 Wang et al. 2019 Zhang &
Sabuncu 2018 Xu al.) 2019) have shown that the losses satisfying Eqn. (3) have the above noise-tolerant property. We give the detailed assumptions and complete proof in Appendix A.2
Assumption 1. Let Pelem and P denote the probability density function of Sekean and Sam respec- lively. There exists a weighting, function w" such that IP elean (x. v) w² (xx yIIl syn (x. (w)y)-
seighting.uunction
Assumption I is reasonable because synthetic data generated by the PLM has wide coverage. Therefore, with proper weight function w, we may recover the clean distribution by reweighting the synthetic data.
5

---

Published as a conference paper at ICLR 2023
Assumption 2. The optimal 0' uniquely minimizes Croteer Selean), L.e, Lrotuss Sctean) Crobust 0, Sclean) for all 0 + 0"
Assumption 2 is natural since the robust losses are originally designed to train the model. Thus minimizing mihust (0, Scimm) is expected to achieve promising performance, as justified by Ghosh et al. (2017) (though the optimization difficulty mentioned in the Remark of Section poses challenges for model training). We also experimentally show that when training model with (0 St clean Crobus Stmem) also decreases, and reaches the plateau at a similar point (Appendix A.I This indicates that (0, Setean) and Lrobust (0, Sckean) have close optimal solutions. Note that even if O and 0" differ by small quantity, i.e., Crobuse (f", Seleun) < Lrobuu (0, Sclean) + E holds with a small E, the following proof holds with just minor modifications.
Theorem If Assumptiin I holds. there exists a w such that ((w)) ) = 0" Further with Assumption and Property 1 our method can uniquely return w" and the resulting 0 with only synthetic (noisy)
data S
Given Assumption holds, Lee (0, Selean) and Labbua (0, Selean) have consistent optimal solution 0" Furthermore, Property of Labuus indicates that 0 can be found with just the noisy synthetic data Sayn- Therefore, we can optimize frobust (f, Soyn) to find 0", Finally, since 0 can be parameterized by w (Theorem [b. which is achieved by the inner loop (Eqn. 2). we can optimize Crobust (0(w), Say) over in the outer loop (Eqn. (ID) find the optimal corresponding w" as well the 0" See Appendix A.3 for detailed proof.
The above analysis theoretically shows our SUNGEN is able to learn the optimal sample weights w and the resulting optimal model parameters 0 with just synthetic data. Notably, Theorem is based on population losses with infinite samples. We further characterize the generalization bound when there is finite samples:
i
Theorem 2 (Finite Sample Generalization). Suppose we have access to synthetic datasets S and Skm both with finite samples N. Let Ô" (w) be the deterministic mapping from w to defined the inner of Eqn 2 given St Assuming the output of loss function Crolust is upper bounded by M, the carnality of W is (W), the outer loop of Eqn is solved within e-approximately, and the solution in satisfies the following condition:
Crabur (1b); S. S min Croiust (6 Sw Stin) +
we then have with probubility at least
-
(ln(
Eroobuse θ- Sem) S min Lintusse ( (w)): + + K
(4)
w
M
Refer to Appendix A.4 for full proof. Theorem characterizes the generalization ability of SUNGEN. If we obtain an approximate solution of the bilevel problem Eqn. ( given finite samples, Eqn. shows that such solution has test performance close to the oracle model.
5
EXPERIMENTS
5.1 SETUP
Datasets & Baselines. evaluate SUNGEN on eight text classification tasks, including IMDb (Maas
et al. 2011). SST-2 (Socher et 2013). ). Rotten Tomatoes (Pang & Lee (005), Amazon (McAuley
Leskovec 2013). Yelp (Zhang et al. 2015), Subj (Pang & Lee| 2004), AGNews (Zhang al.
2015) and DBpedia (Zhang et al. 2015). 3. These tasks have various number of classes, ranging from 2 to 14. Other details about datasets are in Appendix |F We compare our proposed method with the following baselines: (1) PROMPTING. The prompt- based zero-shot classification method based on
PLMs (Brown et al. 2020 Gao et al. 2021b). (2) ZEROGEN. recent zero-shot learning work via
dataset generation Ye et al. 2022a
Implementation Details. We compare the baselines using GPT2-XL (Radford et al. 2019 as PLM. For text generation, use Nucleus Sampling (Holtzman et al. 2020) with 0.9 the decoding

---

Published as a conference paper at ICLR 2023
Table 1: Evaluation results for SUNGEN framework on two different scales of TAM. The scale of synthetic dataset is 200k for both ZEROGEN and SUNGEN. The scales of labeled data in supervised setting are listed under task names, "Gold Data refers to the standard dataset with human annotations.
TAM
#Param
Setting
IMDb
SST-2
Rotten
Amazon
Yelp
Subj
AGNews
DBpedia
Avg
#Gold Data
25k
6.9k
8.1k
256
560k
Ri:
720k
560%
DistiBERT
66M
87.24
89.68
83.67
92.63
95.42
95.95
94.51
99.14
92.28
SUPERVISED
LSTM
~
84.60
76.30
77.49
86.36
91.30
90.20
90.61
98.28
86.89
PROMPTING
80.64
89.22
83.63
82.72
68.00
68.81
67.88
77.85
DisilBERT
66M
ZEROGEN
84,28
87:27
83.02
87.19
87.58
80.45
76.48
79.32
83.20
SUNGEN
89.38
89,45
84.52
89.01
89.19
83.25
80.49
82,67
86,00)
ZEROGEN
71.52
74.89
73.45
80.48
84,95
69.15
73.34
67.02
74.35
LSTM
~M
SUNGEN
84.10
84.58
83.21
84.22
89.06
75.85
79.75
72.07
81.61
Table 2: Evaluation results using different vali- dation sets (S") in the outer loop. LSTM is used
Table 3: Experimental comparison with other de-noise methods using LSTM as TAM.
as TAM.
Method
IMDb
Amazon
Yelp
Method
8"
IMDb
Amazon
Yelp
Rotten
Confidence
79.97
70.44
76.91
SUPERVISED
84,60
86.36
91.30
77.49
Co-Teaching
72.64
73.53
75.50
ZEROGEN
71.52
80.48
84.95
73.45
Gold
82.34
84.71
88.83
80.05
Meta-Weight-e
71.23
79.25
83.41
SUNGEN
Syn.
84.10
84,22
89.06
83.21
SUNGEN
84.10
84.22
89.06
strategy and use GPT2-XL as the generator. To make fair comparison, we use the best prompts designed by (Ye et al. 2022a) in both PROMPTING and data-generation settings (Appendix Table TO). For task model training, we use I-layer Bi-LSTM and DistilBERT-bass the lightweight classifters. The bilevel procedure is iterated 50 times for each task. For more details(e.g., full prompts, training details), please refer Appendix
5.2 EXPERIMENTAL RESULTS
Main Experiments. We present our main experiment results in Table I We observe that our SUNGEN achieves considerable performance gain over the ZEROGEN baseline across all the tasks. Interestingly, the improvement is more prominent for LSTM compared with DistilBERT, which shows relative improvement over ZEROGEN by 9.8% on average across all tasks. We conjecture the reason is that the pre-trained models such DistiBBERT are inherently more robust to noisy training data, which is also pointed out in (Hendrycks et al. 2019). Surprisingly, on Rotten Tomatoes and Yelp, SUNGEN-LSTM even outperforms ZEROGEN- DiST which requires no pre-training and has much fewer parameters.
Effectiveness of SUNGEN on Constructing Noise-Free Data. Figure ! shows that while ZeroGen suffers from overfitting on noisy data, the problem disappears when training with the subset selected by SUNGEN. With longer training epochs, the data of SUNGEN consistently helps improve the performance of TAM on test set and significantly surpasses the result of ZeroGen, which proves that SUNGEN can effectively construct noise-free high-quality data.
Synthetic Data Gold Data Supervision. A key advantage of our proposed framework is that we do not require gold data to optimize sample weights. Here, we compare our SUNGEN to the bilevel reweighting method which uses gold data for calculating the outer objective in Table 2 Specifically, our SUNGEN calculates the robust outer objective free on the noisy synthetic data, while the counterpart calculates the normal outer objective foe on gold data. Table that our method achieves similar performances as the counterpart, which verifies that our method using the noise validation set can equivalently supervise the optimization of sample weights as the clean validation set does.
SUNGEN vs. Popular Denoise Baselines. We compare with other de-noise methods, which are (1) directly removing data with low predicted confidence (Swayamdipta et al. 2020). (2) Co- Teaching (Han et al. 2018), which distinguishes noisy data by large loss value, and (3) Meta- Weight- Net (Shu et al., 2019). which relies meta samples and loss-based proxy model to learn the sample weights. Since in zero-shot setting we have no access to gold data, we use the synthetic data as validation set for Co- Co-Teaching and Meta- Weig--WegtttNet. Results in Table prove our method's
7

---

Published as a conference paper at ICLR 2023
Table 4: Results of SUNGEN-LSTM different
Table 5: Diversity and Correctness. We mea-
data sizes. Given subsets of SUNGEN, models are trained with Lee. For the 1,000k set of SUN-
sure the diversity by Self-BLEU4 and correct- ness by accuracy of an oracle model (finetuned RoBERTa-Large) Lower Self-BLEU4 score in- dicates higher diversity. "SUNGEN-Top" and "SUNGEN-Botton" represent 10k samples with highest and lowest weights respectively.
GEN, the model is trained using weighted fee- Subsets that surpass the original full set (1,000k)
are marked by bold. Performance of original full set is marked by underline.
Method
IMDb
Yelp
Diversiyy
Size
IMDb
Amazon
Yelp
Gold
0.30
0.29
ZERO
SUN
ZERO
SUN
ZERO
SUN
ZEROGEN SUNGEN
0.15
0.12
0.14
1,000k
78.29
86.56
82.47
84.63
86.28
90.38
IÜk
62.40
72.05
74.46
75.84
75.22
80.67
Correcinesx(5)
20k
65.12
79.96
75.78
77.52
79.55
82.88
Gold
96.22
96.60
98.35
50k
68.12
81.14
78.14
81.97
80.81
85.82
ZEROGEN
75.86
93.58
94.47
100k
71.28
82.09
80.25
83.68
82.97
88.41
SUNGEN
82.27
88.87
90.78
200k
71.52
84.10
80.48
84.22
84.95
89.06
SÜNGLENTTP
86.00
84:20
8533
SUNGEN-Boutom
4.57
44.66
superiority in zero-shot setting. We further analyze why the loss-value-based methods are not able to distinguish noisy data in our situation in Appendix
5.3 ABLATION STUDY AND ANALYSIS
Performance of Different Data Sizes. Table shows that even with much less data, model trained by SUNGEN achieves better performance than ZEROGEN. For example, in IMDb, by removing low-quality data, subset with 2% data achieves better performance than the original data (20k vs. 1.000k), which shows superior data quality of SUNGEN. Besides, from Figure ((), we find the percentage of erroncous data is small. However, those erroncous data significantly degrades the model performance: without weighted training to remove the effect of noisy data, the IMDB accuracy of full set is decreased from 86.56 to 78.29.
Analysis of Data Diversity and Correctness. Table 5aleessshon shows that our selected noise-free dataset is more diverse than data generated by ZEROGEN. One interesting thing to note is that in Amazon and Yelp, average correctness of SUNGEN is slightly lower than ZEROGEN. In addition, the top samples (with highest weights) have slightly lower correctness than average. This is expected as the data with highest correctness may be redundant or too simple, while the challenging and informative samples often harder to classify and thus have lower correctness. The results further verify that SUNGEN effectively separates the informative samples from the ones that are redundant or erroneous. This not be done with the heuristic methods that manually sets threshold to separate clean and noisy data, which may keep redundant samples and remove hard ones.
Analysis of Removed and Selected Examples. We take IMDb (sentiment classification task of movie review) as the example task and find that the majority of samples associated with small weights are wrongly labeled, as shown in Table Besides, there a small part of erroneous data which contains task-irrelevant text, meaning the generated text is not movie review and has no obvious emotional tendency. For samples with large weights, we actually find them to be well-writter transitional complex sentences (bottom part of Table 6). which verifies that SUNGEN tends to select correct and important samples.
3"
16
Figure 3: Histogram of leartt weights in IMDb synthetic dataset (1,000k). The weights gradually separated as optimization proceeds, indicating SUNGEN can differentiate high-quality data from erroneous ones.
8

---

Published as a conference paper at ICLR 2023
Table 6: Examples of removed data and selected data in IMDb synthetic dataset.
Text<X>
Label<Y>
Noisy Type
Removed Data (Data with Small Weights)
The film does great job of capturing the fear and battle that so many U.S. troups have experienced during the seven-yeur war in Afghanistun.
Neg.
Noisy Y
This long, pompous, bhuin-smoking movic makes big hit of very much all. One of the worst cult films ever made. 2D CGI animations of zombies and comic-book characters. Some bad acting, technical problems, cheap gimmicks and script that is. The four hour look at family dysfunction brings forth richer character study. A fresh and visceral portrait of a man trying to make sense of his life.
Pos.
Noisy Y
Pos.
Noisy Y
Neg
Unrelated X
Neg
Unrelated X
Selected Data (Data with Large Weights)
Despite its oddball structure and topsy-turvy interactions between characters, this surprisingly zany animated film succeeds where so many animated films fail. Not worth the time for the main actors, but for the almost movie very good story that puts many sci-fi movies the past shame.
Pos.
No Noise
Pos.
No Noise
An outrageously stupid movie that has been thoroughly disproved by fact and logic. Wonder Woman is a big- budget superhero blockbuster that turns the spotlight on the potential of woman leader... but the movie is ultimately unfulfilling and laden with female stereotypes.
Neg.
No Noise
Neg.
No Noise
While satire on class and power. the film's 's dismissal of human misery is shallow and the actors' purtrayal of the weak and needy are gratingly self- self-pitying.
Neg.
No Noise
6 RELATED WORK
Zero-shot Learning via PLM. The popular prompt-based zero-shot prediction is proposed by
GPT (Radford et al. 2019). With well-designed prompts, large-scale PLMSs have shown its notable zero-shot learning ability in various tasks (Jiang et al. 20200; Shin et al. 2020 Reynolds & McDonell
202I). More recently, data-generation-basee
work via PLM has gained popularity and shown superior
ability on synthesizing task-specific data (Anaby-Tavv Tavor et al. 2020; Puri et al., 2020 Kumar et al.) 2020 Lee et al. [021]] Wang et al.] [2221b] Yoo et al. [021 Bonifacio et al. (0022). Apart from work that still relies task-relatee human- -nnoottedd data to instruct or fine-tune the generative PLM,
recent line of research explores this direction in zero-shot scenario: [Schick & Schütze (202T), ); Meng et al. (022) use PLM with task-dependent prompts to generate data, and finetune another PLM on such data for task prediction. To further investigate PLM's zero-shot ability and alleviate the computation cost of PLM. Ye al. (2022a) study an extreme scenario, which trains a tiny model from scratch using the synthetic data.
Noise Robust Learning. Previous methods of tackling data noise can be categorized into two
groups: (1) heuristic approaches based on loss values that rely on the assumption that the network
learns easy samples first, which adopt either resampling (Han et al. 20188] Jiiag et al. [0018] Yu et al.
2019) loss reweighting (Thulasidasan et al.] 2019 Konstantinov & Lampert | Ren et al. 2018
Shu et al. 2019), or label correction (Ma et al., 2018; Kremer et al. 2018 Reed et al. (2014). These methods require either manually set threshold for the loss value or clean validation set, which makes their performance questionable in zero-shot scenario. (2) methods in another line train the
network with noise-robust loss (Ghosh et al.) 2017 Ma et al. 2020 Liu & Guo 2020 Xu al.
2019 Wang et al.) 2019). Despite they learn a robust classifier in theory, they are typically difficult to train the DNNs and result require more pyyer-parameee
tuning (Zhang & Sabuneu| 20188] Wang
et al. 2019). To this end, we take advantages from both lines of research and design an end-to-end framework which can reliably filter out harmful data without requiring a clean validation set.
7 CONCLUSION
This paper focuses on high-quality data generation efficient zero-shot learning. To address the noise in data, we design an end-to-end framework to construct clean synthetic dataset without relying on any human- data or human intervention. Our method can be jointly applied to other data generation pipelines to automatically select high-quality data. We hope this paper can provide insights for improving the quality of synthetic datasets, and inspire more exploration in data-generation-baseed zero-shot learning via PLM.
9

---

Published as a conference paper at ICLR 2023
REFERENCES
Ateret Anaby- Tavor, Boaz Carmeli, Esther Goldbraich, Amir Kantor, George Kour, Segev Shlomov, Naama Tepper, and Naama Zwerdling- Do not have enough data? deep learning to the rescue! In AAAI Conference on Artificial Intelligence, 2020.
Devansh Arpit, Stanistaw Jastrzębski, Nicolas Ballas, David Krueger, Emmanuel Bengio, Maxinder Kanwal, Tegan Maharaj, Asja Fischer, Aaron Courville, Yoshua Bengio, et al. A closer look at memorization deep networks. International conference on machine learning. 2017.
Luiz Bonifacio, Hugo Abonizio, Marzich Fadace, and Rodrigo Nogueira. Inpars: Unsupervised dataset generation for information retrieval. In International Conference on Research and Devel- opment in Information Retrieval, 2022.
T. Brown, B. Mann, N. Ryder, M. Subbiah, Kaplan, P. Dhariwal, A. Neelakantan, Shyam, G. Sastry, A. Askell, et al. Language models are few-shot learners. In Neural Information Processing Systems, 2020.
George Cazenavette, Tongzhou Wang, Antonio Torralba, Alexei A Efros, and Jun- Zhu. Dataset distillation by matching training trajectories. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 4750-4759, 2022.
Hao Cheng, Zhaowei Zhu, Xingyu Li, Yifei Gong, Xing Sun, and Yang Liu. Learning with instance- dependent label noise: sample sieve approach. arXiv preprint arXiv:2010.02347, 2020.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. International Conference on Machine Learning, pp. 1126-1135. PMLR, 2017.
Luca Franceschi, Michele Donini, Paolo Frasconi, and Massimiliano Pontil. Forward and reverse gradient- based hyperparameter optimization. In International Conference on Machine Learning, pp. 1165-1173, PMLR, 2017.
Jiahui Gao, Hang Xu, Xiaozhe Ren, Philip LH Yu, Xiaodan Liang, Xin Jiang, Zhenguo Li, al. Autobert-zero: Evolving bert backbone from scratch. arXiv preprint arXiv:2107.07445, 2021a.
Jiahui Gao, Yi Zhou, Philip LH Yu, Shafiq Joty, and Jiuxiang Gu. Unison: Unpaired cross-lingual image captioning. 2022.
Tianyu Gao, Adam Fisch, and Danqi Chen. Making pre-trained language models better few-shot learners. In Annual Meeting of the Association for Computational Linguistics, 2021b.
Aritra Ghosh, Himanshu Kumar, and P Shanti Sastry. Robust loss functions under label noise for deep neural networks. AAAI conference on artificial intelligence, 2017. 2017.
Bo Han, Quanming Yao, Xingrui Yu, Gang Niu, Miao Xu, Weihua Hu, Ivor Tsang, and Masashi Sugiyama. Co-teaching: Robust training of deep neural networks with extremely noisy labels. Advances in neural information processing systems, 2018.
Dan Hendrycks, Kimin Lee, and Mantas Mazeika. Using pre-training can improve model robustness and uncertainty. In International Conference Machine Learning, 2019.
Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The curious case of neural text degeneration. In International Conference on Learning Representations, 2020.
Lu Jiang, Zhengyuan Zhou, Thomas Leung, Li-Jia Li, and Li Fei-Fei. Mentornet: Learning data- driven curriculum for very deep neural networks on corrupted labels. In International Conference on Machine Learning. 2018.
Zhengbao Jiang, Jun Araki, Haibo Ding, and Graham Neubig. How can we know when language models know? CoRR, 2020.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference Learning Representations, 2015.
10

---

Published as a conference paper at ICLR 2023
Nikola Konstantinov and Christoph Lampert. Robust learning from untrusted sources. In International conference on machine learning, 2019.
Jan Kremer, Fei Sha, and Christian lgel. Robust active label correction. In International conference on artificial intelligence and statistics, 2018.
Varun Kumar, Ashutosh Choudhary, and Eunah Cho. Data augmentation using pre-trained transformer models. CoRR, abs/2003.02245, 2020.
Kenton Lee, Kelvin Guu, Luheng He, Tim Dozat, and Hyung Won Chung. Neural data augmentation via example extrapolation. CoRR, 2021.
Hao Li, Zheng Xu, Gavin Taylor, Christoph Studer, and Tom Goldstein. Visualizing the loss land- scape of neural nets. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett (eds.). Advances in Neural Information Processing Systems, volume 31. Curran As- sociates, Inc., 2018. URL tRLpstttss//// wtwp:eeeeeiinn..e.eiee.eeeee neurips cc /ttpaa/aappp.cppppa//0////////// pttpppppp///////1elee pdf
Alisa Liu, Swabha Swayamdipta, Noah A Smith, and Yejin Choi. Wanli: Worker and ai collaboration for natural language inference dataset creation. arXiv preprint arXiv:2201 055955, 2022.
Hanxiao Liu, Karen Simonyan, and Yiming Yang. Darts: Differentiable architecture search. arXiv preprint arXiv: /806. 09055,, 2018.
Tongliang Liu and Dacheng Tao. Classification with noisy labels by importance reweighting. IEEE Transactions on pattern analysis and machine intelligence, 38(3):447-461, 2015.
Yang Liu and Hongyi Guo. Peer loss functions: Learning from noisy labels without knowing noise rates. In International Conference on Machine Learning, 2020.
Jonathan Lorraine, Paul Vicol, and David Duvenaud. Optimizing millions of hyperparameters by implicit differentiation. In International Conference on Artificial Intelligence and Statistics, pp. 1540 1552.. PMLR, 2020.
Xingjun Ma, Yisen Wang, Michael E Houle, Shuo Zhou, Sarah Erfani, Shutao Xia, Sudanthi Wijew- ickrema, and James Bailey. fimensionallty-drivve learning with noisy labels. In International Conference on Machine Learning, 2018.
Xingjun Ma, Hanxun Huang, Yisen Wang, Simone Romano, Sarah Erfani, and James Bailey, Normal- ized loss functions for deep learning with noisy labels. In International Conference on Machine Learning, 2020.
Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts. Learning word vectors for sentiment analysis. In Annual Meeting of the Association for Computational Linguistics, 2011.
Matthew MacKay, Paul Vicol, Jon Lorraine, David Duvenaud, and Roger Grosse. Self-tuning networks: Bilevel optimization of hyperparameters using structured best-response functions. arXiv preprint arXiv: 0000888 2019.
Dougal Maclaurin, David Duvenaud, and Ryan Adams. Gradient-basee hyperparameter optimization through reversible learning. In International conference on machine learning, pp. 21 2112-1122. PMLR, 2015.
Julian McAuley and Jure Leskovec. Hidden factors and hidden topics: Understanding rating dimensions with review text. In ACM Conference on Recommender Systems, 2013.
Yu Meng. Jiaxin Huang, Yu Zhang, and Jiawei Han. Generating training data with language models: Towards zero-shot language understanding. arXiv preprint 022452200..04538, 2022.
Alex Nichol and John Schulman. Reptile: a scalable metalearning algorithm. 1003v1803..2000 2(3):4, 2018.
arXiv preprint
11

---

Published as a conference paper at ICLR 2023
Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback, 2022.
Bo Pang and Lillian Lee. A sentimental education: Sentiment analysis using subjectivity summariza- tion based on minimum cuts. In Annual Meeting the Association for Compulational Linguistics, 2004.
Bo Pang and Lillian Lee. Seeing stars: Exploiting class relationships for sentiment categorization with respect to rating scales. In Annual Meeting of the Association for Computational Linguistics, 2005.
Hieu Pham, Melody Guan, Barret Zoph, Quoc Le, and Jeff Dean. Efficient neural architecture search via parameters sharing. In International conference on machine learning, pp. 4095 4104. PMLR, 2018.
Renjie Pi, Weizhong Zhang, Yueqi Xie, Jiahui Gao, Xiaoyu Wang. Sunghun Kim, and Qifeng Chen. Dynafed: Tackling client data heterogeneity with global dynamics. arXiv preprint arXiv:2211.18873 2022.
Raul Puri, Ryan Spring, Mohammad Shocybi, Mostofa Patwary, and Bryan Catanzaro. Training question answering models from synthetic data. In Empirical Methods in Natural Language Processing, 2020.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. 2019.
Scott Reed, Honglak Lee, Dragomir Anguelov, Christian Szegedy, Dumitru Erhan, and Andrew Rabinovich. Training deep neural networks on noisy labels with bootstrapping. arXiv preprint arXiv:1412.6596, 2014.
Mengye Ren, Wenyuan Zeng, Bin Yang, and Raquel Urtasun. Learning to reweight examples for robust deep learning. In International conference on machine learning. PMLR, 2018.
Laria Reynolds and Kyle McDonell. Prompt programming for large language models: Beyond the few-shot paradigm. In Conference Human Factors in Computing Systems, 2021.
Timo Schick and Hinrich Schüitze Generating datasets with pretrained language models. In Empirical Methods in Natural Language Processing, 2021.
Amirreza Shaban, Ching-An Cheng, Nathan Hatch, and Byron Boots. Truncated back- propagation for bilevel optimization. In The 22nd International Conference Artificial Intelligence and Statistics, 2019.
Han Shi, Renjie Pi, Hang Xu, Zhenguo Li, James Kwok, and Tong Zhang. Bridging the gap between sample-based and one-shot neural architecture search with bonas. Advances in Neural Information Processing Systems, 111188881119 2020.
Han Shi, Jiahui Gao, Xiaozhe Ren, Hang Xu, Xiaodan Liang, Zhenguo Li, and James Tin-Yau Kwok. Sparsebert: Rethinking the importance analysis in self-attention. International Conference on Machine Learning, pp. 9547-9557. PMLR, 2021.
Taylor Shin, Yasaman Razeghi, Robert I.. Logan IV, Eric Wallace, and Sameer Singh. Autoprompt: Eliciting knowledge from language models with automatically generated prompts. In Empirical Methods in Natural Language Processing, 2020.
Jun Shu, Qi Xie, Lixuan Yi, Qian Zhao, Sanping Zhou, Zongben Xu, and Deyu Meng. Meta- weight-net: Learning an explicit mapping for sample weighting. Advances in neural information processing systems, 2019.
Ankur Sinha, Pekka Malo, and Kalyanmoy Deb. A review bilevel optimization: from classical to evolutionary approaches and applications. IEEE Transactions on Evolutionary Computation, 22 (2):276-295, 2017.
12

---

Published as a conference paper at ICLR 2023
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang. Christopher D. Manning, Andrew Y. Ng, and Christopher Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In Empirical Methods in Natural Language Processing, 2013.
Swabha Swayamdipta, Roy Schwartz, Nicholas Lourie, Yizhong Wang, Hannaneh Hajishirzi, Noah A Smith, and Yejin Choi. Dataset cartography: Mapping and diagnosing datasets with training dynamics. arXiv preprint arXiv:2009. 2020.
Sunil Thulasidasan, Tanmoy Bhattacharya, Jeff Bilmes, Gopinath Chennupati, and Jamal Mohd-Yuu Yusof. Combating label noise in deep learning using abstention. 2019.
Paul Vicol, Luke Metz, and Jascha Sohl-Dickstein. Unbiased gradient estimation in unrolled computation graphs with persistent evolution strategies. In International Conference on Machine Learning, pp.
PMLR, 2021.
Martin J Wainwright. High-dimensional statisties; A non-asymptotic viewpoint. Cambridge University Press, 2019.
Shuohang Wang, Yang Liu, Yichong Xu, Chenguang Zhu, and Michael Zeng. Want to reduce labeling cost? GPT-3 can help. In Findings of the Association for Computational Linguistics: EMNLP 2021, 2021a.
Tongzhou Wang, Jun-Yan Yan Zhu Antonio Torralba, and Alexei A Efros. Dataset distillation. arXiv preprint arXiv: 10959, 2018.
Yisen Wang, Xingjun Ma, Zaiyi Chen, Yuan Luo, Jinfeng Yi, and James Bailey. Symmetric cross entropy for robust learning with noisy labels. In International Conference Computer Vision, 2019.
Zirui Wang, Adams Wei Yu, Orhan Firat, and Yuan Cao. Towards zero-label language learning. 2021b.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, and Jamie Brew. Huggingface's transformers: State-of-the-ar natural language processing. CoRR, abs/1910.03771, 2019.
Yilun Xu, Peng Cao, Yuqing Kong, and Yizhou Wang. L L_dmi: A novel ieftommation-thhoreee loss function for training deep nets robust to label noise. Neural Information Processing Systems, 2019.
Lewei Yao, Renjie Pi, Hang Xu, Wei Zhang, Zhenguo Li, and Tong Zhang. Joint-detnas: Upgrade your detector with nas, pruning and dynamic distillation. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition. pp. 10175-10184, 2021.
Jiacheng Ye, Jiahui Gao, Qintong Li, Hang Xu, Jiangtao Feng, Zhiyong Wu, Tao Yu, and Lingpeng Kong. Zerogen: Efficient zero-shot learning via dataset generation. In Empirical Methods in Natural Language Processing, 2022a.
Jiacheng Ye, Jiahui Gao, Zhiyong Wu, Jiangtao Feng, Tao Yu, and Lingpeng Kong. ProGen: Progressive zero-shot dataset generation via in-context feedback. In Findings of the Association for Computational Linguistics: EMNLP 2022, 2022b.
Kang Min Yoo, Dongju Park, Jaewook Kang, Sang- Woo Lee, and Woomyoung Park. GPT3Mix: Leveraging large-scale language models for text augmentation. In Findings of the Association for Computational Linguistics: EMNLP 2021, 2021.
Xingrui Yu, Bo Han, Jiangehao Yao, Gang Niu, Ivor Tsang, and Masashi Sugiyama. How does disagreement help generalization against label corruption? In International Conference on Machine Learning, 2019.
Xiang Zhang, Junbo Zhao, and Yann LeCun. Character-level convolutional networks for text classification. In Advances in Neural Information Processing Systems, 2015.
Zhilu Zhang and Mert Sabuncu. Generalized cross entropy loss for training deep neural networks with noisy labels. In Advances in Neural Information Processing Systems, 2018
13

---

Published as a conference paper at ICLR 2023
Bo Zhao, Konda Reddy Mopuri, and Hakan Bilen. Dataset condensation with gradient matching. arXiv preprint arXiv:2006.0543 05929, 2020.
Guoqing Zheng, Ahmed Hassan Awadallah, and Susan Dumais. Meta label correction for noisy label
learning. AAAI Conference Artificial Intelligence, 2021.
Xiao Zhou, Yong Lin, Renjie Pi, Weizhong Zhang. Renzhe Xu, Peng Cui, and Tong Zhang. Model agnostic sample reweighting for out-of-distribution learning, In Kamalika Chaudhuri, Stefanic Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.), Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 27203-2722 PMLR, 17-23 Jul 2022a. URL/https: penttps://proceedii .gs.mlr press/vil vl 62/ whw.22d.
Xiao Zhou, Renjie Pi, Weizhong Zhang, Yong Lin, Zonghao Chen, and Tong Zhang. Probabilistic bilevel coreset selection. Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.), Proceedings of the 39th International Conference Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 27287-27302. PMLR, 17-23 Jul 2022b. URL https:: nttps://proceed ngs. ml www..mm.s............ / / hww.//oo.22hhh htm
APPENDIX
A PROOFS
A.I EMPIRICAL VERIFICATION FOR ASSUMPTION AND
Summary and Discussion. In the sections below, we first show that (1) the poor performance of training the neural network with robust losses is due to the optimization difficulty, rather than the lack of good solution. We verify this by showing that the loss value of RCE is able to steadily decrease when the network trained with CE loss. However, when training the network with RCE, the loss value fails decrease. Then, show that (2) the optimal solutions of CE and RCE losses are close by plotting the loss surface for both losses around the optimal solution obtained with CE loss. We are able to observe that the RCE loss is also close to the minimum around the optimal solution of CE loss. We believe the above two experiments can verify that the Assumption and are reasonable.
(a) 3D visualization from the first angel.
(b) 3D visualization from the second angel.
(c) 3D visualization from the third angel.
(d) 2D visualization.
Figure 4: Loss surface visualization of RCE and CE losses. We parameterize the surface with parameters to vary model parameters and calculate its loss values alongside a 2D grid. In each subplot, the left visualizes the loss surface of RCE loss; and the right one visualizes the CE loss surface. (a)-(c) are the visualization from different angles; (d) is the loss contour of CE and RCE.
14

---

Published as a conference paper at ICLR 2023
A.1.1 OPTIMIZATION DIFFICULTY OF NOISE-ROBUSE LOSS.
First. we analyze the cause of difficulty in the optimization of robust loss: (1) Zhang & Sabuncu
2018) show that the gradients of the cross-entropy loss have implicit weighting term(as shown in Equation 5), which prioritizes the harder samples during training. On the other hand, this weighting does not exist in noise robust loss functions, which means these functions treat all samples equally. The lack of implicit weighting is claimed by (Zhang & Sabuncu, 2018) to be the cause of difficulty in training.
ac(f(x,; (0.e;:),.0 V)) 00
EL Tu, (2,00) Volu, (at;; 0) for CE Ei= -Vofn. (m:; O) for MAF/RCE.
(S)
(2) Our experiment shows that the surface of the robust loss has wide flat region, when the parameters are not close to the optimal solution, the gradients could vanish, which leads to difficulty in optimization. The 3D visualization of the loss surface is shown in Figure More details about drawing the figure are shown in A.1.2
Second, verify the performance degradation when training the network with robust loss is caused by optimization difficulty, rather than the lack of good solution. Without loss of generality, we compare the robust-loss lace fee. More specifically, another loss to evaluate the model trajectory. From the results in Figure ((a) we can clearly see that when optimizing the network with CE, both CE and RCE continue decline throughout the training, and reach the plateau at the same time, which means that the RCE loss does indeed have a good solution. On the other hand, from the results in Figure 5(b) when training with RCE, both CE, and RCE hardly decrease, this further verifies that RCE is difficult to optimize when training the network.
train the model with one loss, and use
iCB
Epoch
Epech
Epoch
Epoch
(a) LSTM model optimized by for
(b) LSTT model optimized by free
Figure 5: Loss curves of model training. Experiments run on the IMDb standard training set. The loss used to train the network is in red, and the other loss used evaluate the network is in grey. We have run experiments using various learning rates from 1e-2, 1e-3, le-4, 1e-5) and the curves show similar trends.
A.1.2 THE OPTIMAL SOLUTIONS OF CE AND RCE ARE CLOSE.
Despite the optimization difficulty of RCE loss, the loss surfaces of CE and RCE losses indicate that the two loss functions have close optimal solutions. More specifically, we plot the loss surfaces of CE and RCE losses centered around the solution of CE loss following (Li et al., 2018), which visualizes the loss surfaces by modifying the model weights in two random directions:
f(a L(O + ad + (n)
where 0* is the model parameters optimized by CE, 6 and 17 are the random directions in the vector space of model parameters. a and 3 are the scaling coefficients of the two directions, which control how far the parameters are perturbed. We parameterize the surface w.r.t the values of a. 3 ranging from [-1, 1]. and calculate the loss values at different positions alongside a 2D grid. We show the visualization in Figure 4
From sub-figure 5 (d), we can see that the optimal solution of CE is also close to the optimal solution of RCE (also has the lowest value near zero), which is direct experimental support for the Assumption 3 In addition, when the solution is not close to the optimal position, the ree loss surface flat and wide, which is the cause of optimization difficulty.
15

---

Published as a conference paper at ICLR 2023
A.2
PROOF FOR PROPERTY
The noise-robust property of robust loss functions have been proven in previous work (Ghos et al. 2017 Wang et al., 2019), we add them here for completeness. Specifically, we consider three cases of label noise: uniform noise, simple non-uniform noise and class-dependent noise following (Ghosh 2017).
et al.
Uniform Noise. Given a classification problem with K classes, for a loss function eobusy satisfying
Then brobust is noise-tolerant under uniform label noise if the noise rate < K Let denote the noisy label from Ssyu. the proof is as follows:
Eqn. 3)
Liobus (., Ssya) =Ese, S Coobus (f (w: 0). 5)]
-En, Eujy, Loouust (f 0), 3)]
K
Ex,y((
(11 [11
- e (f (a: 0), + K 1-1
Σ
rouust (f(x; 0),
jty
K - 11 KT
1/C
= -En.yl
Crotmss (s 0), [yy +
K-1
K 1
K 1 Кι C(0, Sclem)) -
n/C
K K-1
where C' is a constant due to the property of symmetric loss functions. Suppose 0" is the opti- mal solution for the clean dataset Selean- Therefore, have the following inequality for any O: L( Syy) L(O, S.yn) K-1-k- Kn (L(O", Selea)) L(0, Setean) < 0. Therefore, l" is also the
optimal solution for S.yn-
Class Dependent Noise. For a loss function brobust satisfying Eq 3 and 0 S ((o 0). S R- Vi E [K], and suppose ming L(O, Sclean) = 0. Then frohust noise-tolerant under classs dependent label noise if Mi < 1 ni, vjj # i, Vi,j E [K]. Thy represents the probability of class mislabelled into class j. The proof is as follows:
Lrohusst (0, Soyn) = -E..i (an.jllnoouee (Lmouua (fxx; 0), v)]
-Fur,y B. [foobuss (f (x; 0). 7)
K
-Exw((1 - Ml ) (f 0), +
Σ
Tui Protust (f(;; 0).
57y
K
K Tui Coouut (f(; O),
-Exc.u(1 - -m/)(C
robast (f(x; 0),5))
-CE,,,y1 [hy)
Emn.y
fly
Muy ) (f; 0).
Denote 0 and 0. as arg ming Lrobus (0, Setean) and arg ming Crobus Sayn), respectively. Given the above result, we have the following inequality for O* and 0*:
Lrviuaa (0",Syp)) - Lrobusa , SSyn) EERLy
(1-1gy -nys (Erodusa (s(x; 0)),)))) Cromus (J ('))j)
Since frobust is always non-negative and Crobust (0" Selean) = O, we have that Erobust (f 0*).)) 0, Vo. Thus, we have that Erobua (2; 0²), K-T
C Vi + by the symmetric property of frobust- Given the assumption on the label noise, (1 Fl nus) > 0. Therefore, for the equality to hold, we musts have Crotus (. (a;; 6- i) R Vi 4 y. According to the symmetric property of froust. know that frodust (f 0²), y) = 0, Vo. Which means that 0* achieve zero losss on all clean samples well. Therefore, 0 is also the minmmizer for L (f, Seleun)
16

---

Published as a conference paper at ICLR 2023
Non-Uniform Noise. For loss function lotust satisfying Eq and suppose ming [(0, Sciean) 0.
Then robbstt is noise-tolrant under non-uniform label noise if The K-1 KK
Va. The proof is as follows:
Crobust (0, Ssya)
=B Coobus (f ( ; 0).
-Bg. Bij robuss (f (x; 0). 0)]
-, [11 71 robust (2; v) =Em,y M7 Cobuual (5 (x; (x:0), + Em
Σ iFy
K K- (oomus (2;
The
1/o (C Lrobusa (f 0), v)
K
C + Ex,y1(1
Kπo
=Ex K
roobual (f(æ; 0),
K
Therefore, we have the following inequality for any Lrotwa Ssyn) Libbsa (0, Ssyn) (frutuss O7)) y) rabbua (f (f:; f), y))] Since lrobus is always non-negative and ), y) = O, Va. Therefore, 0* is also the optimal Crobust (0" ,S = O. we have that of 0²),
Eg.yy [11 Knz
K-1
solution for Ssyn-
A.3 PROOF FOR THEOREM 1
Proof. We first show 0* exists in the space induced by ((w). Then we show our framework uniquely return 0"
Step Existence, By Assumption 1 we know that syn (ax, y)w" " (xx, v) Petean (xx, y) in distribu-
tion. So
O(w*) = arg min Lrabusa (0, Segn(w"))
0
= arg min / wn v)(((((( x). Py) ydddd
arg min (e ( ((f x0.
arg min Be (m.)) (f((0
=0"
Step 2: Uniqueness. By Assumption 2 we have for all 0 a 0-, we have Looeus (0) Selean) <
frobusa (0, Selean). So 0" arg ming Lrobusi ( Selean). Since arg ming Lrsbus ( Sayn. arg ming Lrobus ( Sclaan). So we have
0"
arg min Lrobua (0, Ssyn).
Putting the existence and uniqueness parts together, finish the proof.
The above theorem shows that there is an explicit mapping between the optimal weighting function w" and the optimal 0" This mapping is achieved by the inner loop in formulation 2 We can then optimize Lee(0(u) ( over un to find the optimal w². Given that Assumption holdd, we can switch (0w)), Selem) to Lrobust (0(u) w) Sekean). Finally, thanks Property [ robustt we may simply optimize Crobust (cobus(((()) (0(w), Saya) over t instead. This gives rise to the outer loop in formulation
A.4 PROOF FOR THEOREM 2
Proof. Let Sv.-1 denotes the dataset that replaces any one element of Sy sym with arbitrary x. is easy to know that
Crotusst (Ō' (w)
Crohust Sy, <
M
17

---

Published as a conference paper at ICLR 2023
holds for any Then by the bounded difference inequality (Corollary 2.21 of Wainwright (2019) given w, we have with probability 6,
In(1/8)
Lodiust (w); Suyn) S Crobus (6" (w)); + ky
(6)
2M
Then we have
Crobust (0" (w)) Ssya)
2 2In([Wl/0)
Lobusa (00())))) S +
M
In((W)/8)
Crobust (0"(w); KVV
+
2M
1n(1/8)
< Crobusa (0 (w): Siyn) + NV
2M
+
2M
2 2n /6)
Crobusa ( (w)); Saya) + VVV M +c,
The first inequality because require incquality equation 6 hold uniformly for all [W] functions. The second inequality because vi is the e-approximated solution. The third inequality is applying inequality equation The forth inequality is because [W] > 1. Taking infimum over on the right hand side, we obtain the desired bound.
B RELAXING ASSUMPTION 2
Here provide additional results on relaxed assumptions. We relax Assumption as follows: Assumption 3. The optimal 6" achieves c-optimal robust loss Lrobuse θ Sclean) on the clean dataset, i.e., Croiust O Selean) Crobus (0, (0, Seleon) for all O f O with
Then we have the following results that extend Theorem
Theorem 3. If Assumption holds. there exists a w" such that ((w)) = 0" Further with Assumption 3 Band Property| Nand assume Crobmu O (w), Seteam) is ju-strongly convex and differentiable w.r.t w our method can return a w that is close to wp as follows:
w - w w"112 S /2e/p.
Further, if the minimizer of the robust loss on the clean data collides with 0", i.e., = 0. we have w w" and i =
Proof. Similar to the first step of the proof of Theorem I in Appendix A.3 we know @(w²) 0- Further by Property we know the solution of arg ming Crobust 0, Ssyn) is the same with that of arg ming Crobuu Selean). We further have
arg min Crobust (ww), Saya) = arg min Crobust ( (w). Selean).
Denote t arg ming Crotuaa (((ww))))))))))))))) Screan), it follows that
V Crobbaa ((r) Setean) = 0
By Assumption 3
we then have
Crobust (0", Selcan) Crobust(0 (0(w), Selcan)
(8)
We further have
Crobuua (0' Seleun)
= Lyinual ① (ww),, Sclean)
Crobust 1 ((w), Sctcau) + V (robuur((((())) Scken)) (w" w))
+ ||'0
-w1l
(9)
18

---

Published as a conference paper at ICLR 2023
The equality in the first line is due ((w=)) 0¹ The inequality comes from the strong convexity assumption. Combining Equ equation 7
equation [ and equation 9 we have
w S.
The result follows immediately by noting that w is the output of our method.
C ABLATION STUDY OF OUTER OBJECTIVES
In the main paper, directly use RCE an example noise-robust loss verify our framework. Here we conduct additional experiments using the other noise-robus Mean Absolute Error (MAE) loss (Ghosh et al. 2017). The results in Table show that SUNGEN framework using RCE and MAE losses both achieve promising performance, which significantly surpasses baseline methods. Besides, we consider the standard cooss-entropyCCE loss as the outer objective, the bi-level framework can only achieve marginal improvement even worse than ZEROGEN. The result is reasonable the standard CE loss in the outer loop run on synthetic data, which cannot provide accurate guidance to update the per-sample weights.
Table 7: Experimental comparison using different outer objectives. Since clean validation set exists in the zero-shot setting, all the outer objectives are calculated on synthetic data. The experiments run on 200k synthetic data using LSTM TAM.
Method
Outer Objective
IMDb
Amazon
Yelp
ZEROGEN
71.52
80.48
84.95
Bi-level
CE
74.05
79.81
83.90
RCE
84.10
84.22
89.06
SUNGEN
MAE
84.23
84.05
89.23
D SUNGEN VS. OTHER NOISE-ROBUST TRAINING STRATEGIES
We compare our method with the label smoothing and temporal ensemble woise-training strategies in Table 8 The experimental results show that method achieved significant improvement over these two counterparts. In addition, label smoothing and temporal ensemble are training strategies to alleviate the noise issue, which cannot be used to select high-qualis subset.
Table 8: Comparison between SUNGEN with other noise-robust training strategies. Experiments run on 200k synthetic data using LSTM TAM.
Method
IMDb
Amazon
Yelp
ZEROGEn
71.52
80.48
84.95
Label Smoothing
73.18
81.91
86.07
Temporal Ensemble
74.10
81.42
85,82
SUNGEN
84.10
84.22
89.06
E EXPERIMENTS ON OTHER CLASSIFICATION TASKS
As suggested, we conduct experiments on more difficult tasks GLUE in Table 9 including the NLI tasks (RTE, QNLI) and paraphrase task (MRPC). The experimental results show that SunGen can consistently surpass the baseline methods. For more challenging tasks that need expert knowledge or reasoning ability, leave them future work.
19

---

Published as a conference paper at ICLR 2023
Table 9: Comparison between SUNGEN with other noise-robust training strategies. Experiments on 20k synthetic data using DistilIBERT as TAM
Method RTE
QNLI
MRPC
PROMPTING
54.51
60.60
65.04
ZEROGEN
57.04
65.46
68.71
SUNGEN
62.82
71.82
71,25
F IMPLEMENTATION DETAILS
F.1 DATASET
We evaluate our method on eight widely-used text classification tasks, including IMDb (Maas et al. 2011), SST-2 Soocher al. (2013), Rotten Tomatoes (Pang & Lee 2005), Amazon (McAuley & Leskovec 2013). Yelp (Zhang et al. 2015), Subj (Pang & Lee 2004). AGNews (Zhang et al. 2015) and DBpedia (Zhng al. 2015). IMDB, SST-2, and Rotten Tomatoes sentiment classification benchmarks containing positive/negative movie reviews. Amazon and Yelp are comments classifica- tion tasks consisting of electronic product reviews and restaurant reviews respectively. We choose electronics and restaurant reviews as they are very different from movie reviews. Subj is subjectivity detection task justify whether the text contains factual contents or expresses opinions. AGNews (4-class classification) and DBpedia (14-class classification) are the topic and otologogy classification tasks respectively. Apart from AGNews and DBpedia, other tasks are binary classification tasks. We use full test set for evaluation except for DBpedia, for which we randomly sample 5000 test examples to reduce computational cost. Sample sizes are listed Table We report accuracy for evaluation.
F.2 FULL IMPLEMENTATION DETAILS
We compare the baselines using GPT2-X XL (Radford et al. 2019) as PLM. For text generation, we use Nucleus Sampling (Holtzman al., 2020) with p = 0.9 the decoding strategy and use GPT2-XL as the generator. For fair comparison, we use the best prompts designed by (Ye et al. (022a) for data generation. During the optimization of sample weights, we use Adam optimizer. For selecting the appropriate value of the outer learning rate, we select from (2.5e-1, le-1, 1e-2) by looking at the value of RCE loss in the outer loop. If the outer loss steadily decreases and reaches low value, then it indicates that the optimization is going well. In the inner loop, 1,000k synthetic data are used the training data; in the outer loop. 50k synthetic samples are randomly sampled as the training data for fast iteration. We use I-layer Bi-LSTM and DistilBERT base the tiny task model and run it for 1 epoch each time for fast iteration. The bilevel procedure is iterated for 50 times for each task.
For task model training, For LSTM, we use Adam BeatiiizerKKnnnnn & Ba 2015) with learning rate 1e-3. For DistiBBERT- base, we finetune each dataset using Adam optimizer with learning rate 2e-5, and other default hyper-parameters as suggested by HuggingFace Transformers library WWoff et al., 2019).. We run LSTM for epochs, and run DistilBERT-base for 3 epochs for prediction. Unless otherwise stated, we run our experiments on 200k data. We compute the average accuracy on test set over 3 runs using different random seeds.
1-layer Bi-LSTM and DistilBERT
the light-weight classifiers.
For the baseline using gold data in Table to simulate the scenario where gold data is scarce, we randomly select 1,000 samples from the standard training set as the training data in outer loop. For comparison with other denoising baselines shown in Table 3 we use the techniques the original papers. Specifically, for Confidence (Swayamdipta et al. 2020), we use the mean model probability of the true label across epochs as the confidence value and select top 200k examples; for Co-teaching (Han et al., 2018),, we use two networks, each is trained with samples selected by the other network based on the loss value; for meta- weight-net (Shu al. 2019), since do not have access to clean data, we use a part of synthetic data as validation. Co-teaching and meta- weight-ntt conducted on 200k synthetic data.
described in
20

---

Published as a conference paper at ICLR 2023
F.3 PROMPTS
For IMDb, SST-2, and Rotten Tomatoes, we use the best prompts designed by Ye et al. (2022a) in both pRompTINg and data-generationn settings. For tasks that are not investigated by ZEROGEN, following Ye et al. 2022a),, we manually design five prompts for each task and report the result of the best prompt on PROMPTING, then we use the same prompt(or minor revision version) for ZEROGEN and SUNGEN to generate data. The details of prompts are shown in Table TO
Table 10: Prompts used for PROMPTING and data generation. Note that ZEROGEN and SUNGEN use the same prompt for each task. <c> represents the input movie names/electronic categoriie/restaurant names. <MASK> position will be placed with label words. <x> represents the text that we use PLM to generate. The movie and restaurant names are generated by PLM using prompt "Movie:
"Restaurant: " The electronic categories are 41 categories collected from Amazon website.
Setting
Task
Best Prompt
Label Words
IMDb
movie review <MASK>> sentiment is: "OP>>
positive/negative
Amazon
electronics product review <MASK>
positive/negative
Yelp
restaurant review sentiment is:
positive/negative
positve/negaaive
PROMPTING
Rotten
movic review <MASK> sentiment
"<xc>
positve/negative
contaiming factual contents
Subj
The movie review <MASK> is: 'co>
(objective) / expressing opinions
(subjective)
WorlldSporss/Hasinees
AGNews
The news article is in the category of <MASK>: "ex>
Technology
CompanyiSchooll/Anin
Atblete/ Politician/
DBpedia
article classified to category of <MMAK>;; "ex
Transportation/ Building/
Nature/Village/Aene Village/Aminal/Paatt
Album/ Film/Book
IMDb
movic review <MASK>> sentiment for movie "<cs" is: "exo
positivennegative
Amazon
The product review <MASK> sontiment is: "iox>
positive/negative
Yelp
restaurant review <MASK> sontiment is: "oxp>
positive/negative
ZEROGEN
SST-2
movie review sentiment movie is:
positive/negative
& SUNGEN
Rollen
movie review
sentiment movie " iss positive/negative
comtaiming lactual contents
Subj
The movie review <MASK> is: "ex>
(objective)) / expressing opinions
(subjective)
World/Spots/Businesss
AGNews
The news article is in the category of <MASK> "CX>
Technology
Company/Schooll/Anie
Athlete/ Politician
DBpedia
The articlo classified to the category of <MASK> "KXOS
Transportaioon Building!
Valure/Village/Aninnii /aiaag///aia///aaa///
Album/ Filn/Bonk
G BI-LEVEL VS. ONE-LEVEL OPTIMIZATION
We empirically verify the effectiveness of bi-level SUNGEN than one-level optimization using Lee (One-level, (xa). From the results in Table II] observe that our framework with bi-level Lee outperforms both one-level counterparts significantly.
Table 11: Comparison between our method with one-level free on 1,000k data. LSTM is used as TAM.
Method
IMDb
Amazon
Yelp
One-level, tive SUNGEN
81.92
81.50
85.36
86.56
84.63
90.38
21

---

Published as a conference paper at ICLR 2023
H TRUNCATED BACK-PROPAGATION FOR META GRADIENT.
For solving the bilevel optimization problem, the gradient of can be calculated as follows:
Crobust
Va Receus (-- v.)
(10)
S Lo
8 Lee
Vo Lrobust B5
II
0000T
a0rwT
SST
kcj
BT
0r---1
Lee
V Cestust By
(11)
w0owT
0--1
where Eqn. (10) follows chain rule. For computational efficiency, we do not unroll the entire T steps,
but perform -tsppp truncated backpropagation as in Eqn. (11) (Shaban et al. 2019).
I COMPARISON WITH OTHER NOISE-ROBUST LEARNING METHODS
Our framework has the following advantages against other noise-robust learning methods:
Compared with heuristic methods (Han et al. 2018 Jiang et al. 2018) our framework is end-to-end and does not require excessive manual tuning and task dusk-specific knowledge.
Since both the inner and outer objectives are calculated on the same synthetic training set, we do not need any in-domain labeled data, which must the previous end-to-end reweighting methods (Ren et al., 2018 Shu et al. 2019).
Compared with methods that train the model with broustt (Zhang Sabuneu 2018 Wang et al. 2019). our approach leverages Goomsst to learn the sample weights, which enables removing the low-quality data without hurting the model's performance
J MORE RELATED WORK
Bilevel Optimization Bilevel optimization (BO)|Sinha) et al. (2017). which is what we rely upon when building our algorithm, has received much attention recently. This optimization technique has the ability to tackle problems with hierarchical structures. BO has been successfully adopted in numerous applications, such as
optimization Lorraine et al. (202)): Maclaurin et al. [2015)]Maakaa MacKay al. (2019); Franceschi et al. (2017): Vicol et al. (2021). neural architecture search Pham et al. (2018 11 Liu et al. (2018); Pham et al. (LI[(2018); Shi et al. (2020); Yao et al. (202I): Gao
al. (2022 2021a Shi et al. 2001 ) meta learning Finn al. (2017) )); Nichol Schulman (2018). dataset condensation Wang et al. (2018) 8): /hao et al. (2020): Cazenavette et al. (2022); Pi et al. (0022
and sample re-weighting Ren
20188; Shu al. (201 Zhou (202aa
K SELECTED AND REMOVED DATA
The selected and removed examples are listed in Table We take IMDb as the example task. The observation shows that most of the removed data (data with low weights) have noise label, which indicates the class of text is wrongly labeled by PLM during generation(Noisy Y). Besides, there is small part erroneous data which contains unrelated text to the task, meaning the generated text is not a movie review showing obvious emotional Cendency(Unrelltte percentage of the erroneous data is small, but it significantly degrades the model performance(e.g IMDB accuracy is decreased from 86.56 to 78.29 in Table 4). For selected data by SUNGEN, find they are actually well- wrrtteritten transitional complex sentences (bottom part of Table b), which verifies that SUNGEN tends to select correct and challenging samples.
dd) X). From Figure d), we find the
22

---

Published as a conference paper at ICLR 2023
(a) Epoch 0(54%). (b) Epoch 1 61%). (c) Epoch 2(63%). (d) Epach 3(60%). Epech 4(62%).
Figure 6: Loss histogram of SST-2 with 30% uniform label noise. Clean data and noisy data are marked by green and red respectively. The accuracy is listed in the parentheses. We observe that the loss value can not separate the clean data from the noisy ones. Therefore, the methods loss-value- based methods may not work well in our case.
L WHY LOSS- VALUE-BASE DE-NOISE METHODS DOES NOT WORK IN OUR SITUATION?
We further empirically analyze why the popular loss-value- based methods (Shu al. 2019 Han et al. 2018) fail to help sample noise-free dataset in our scenario. For this experiment, we manually construct a dataset with 30% label noise using the training set of SST-2 by randomly flipping the class label. As shown in figure 6 the loss values of the correctly labelled data and the mislabelled data are still clustered together. We conjecture that this is due to the nature of the task: not all tasks demonstrate the phenomenon that noisy data demonstrate higher loss values, which is also mentioned in works related to instance dependent noise learning (Cheng et al. 2020). Therefore, selecting subset based on the loss value is not applicable in scenario.
23


Reviews:
[{"review_id": "r47_NYMZlH", "reviewer": "Reviewer_23fs", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper studies data generation-based zero-shot learning that mainly uses a large PLM to generate training data and then trains a small tiny task model (TAM) to perform classification. The major challenge that the paper tries to address is that the synthetic data are usually noisy, and hence the paper proposes a meta-learning empowered framework to automatically learn sample weights over the synthetic dataset -- the goal is to upweight the accurate samples and downweight the noisy ones. Different from previous meta-learning approaches that assume a clean validation set for learning sample weights, this paper uses a noise-robust loss function calculated on the synthetic dataset for the outer-loop optimization, thus removing the requirement for manually annotated data. The proposed method SunGen is compared with ZeroGen on eight classification tasks and demonstrates significant improvements.", "summary_of_the_review": "I appreciate the attempt to address the major challenge with leveraging synthetic datasets generated by PLMs -- the existence of label noise. The proposed method is also novel and interesting. However, there are a few points (e.g., strong theoretical assumptions, missing discussions of loss function choices, and other empirical evaluations) that need further revisions to be made convincing.", "strength_and_weaknesses": "Pros:\n* The paper targets an important and timely topic in zero-shot learning -- generation-based approaches recently have become a promising and popular direction for zero-shot learning, but the major challenge of how to effectively leverage the potentially noisy synthetic dataset has remain unresolved. The proposed method that applies the idea of meta-learning to automatically learn sample weights without requiring manually-labeled clean dev sets is new and interesting.\n* The method is empirically effective across eight classification tasks, outperforming the ZeroGen baseline, and even achieving comparable performance to fully-supervised training that uses the clean training data (though they are mostly of smaller size than the generated dataset).\n* The paper is overall clearly written and well organized.\n\nCons:\n* The theoretical analyses are not informative. I believe that Assumption 2 is too strong (or even wrong) and makes the entire proofs trivial. Assumption 2 essentially states that the optimal solution on the clean set induced by the noise robust loss function is exactly the same with that induced by the standard cross-entropy loss. However, previous studies (e.g., Zhang and Sabuncu) have shown that noise robust losses usually worsen the converged solution compared to standard cross-entropy when used on clean datasets, so it's hardly the truth that they actually lead to similar optima. I believe Assumption 2 is actually what requires more formal proofs (Theorems 1 & 2 will trivially hold given Assumption 2). The empirical validation presented in D.1 is not convincing since similar training curve trends do not imply similar global/local optima. \n* The choice of noise robust loss function needs to be further articulated. The paper directly picks reversed cross-entropy loss without explaining the reason. There are many functions that can satisfy Eq. (3) -- even a constant loss function that does nothing for training. It is therefore necessary to justify why reversed cross-entropy loss is used (rather than other noise robust loss functions in, e.g., Ghosh et al. & Zhang and Sabuncu). This may be done empirically as ablation studies.\n* It would have made the empirical evaluation more complete if the method was compared with other noise-robust training strategies (e.g., label smoothing & temporal ensembling used in Meng et al.) and on more difficult classification tasks (e.g., the NLU tasks from the GLUE benchmark).\n* (Minor) There are a few typos and unnatural language usages that need further proofreading. For example, (1) \"despite we can generate unlimited training data in theory\" -> \"despite the unlimited training data that one can generate in theory\" (2) \"we may recover the clean distributional\" -> \"we may recover the clean distribution\" (3) \"the bi-level optimization approaches has been proven\" -> \"the bi-level optimization approaches have been proven\" (4) \"To attack the noise data\" -> \"To address the noise in data\"\n* (Minor) There seem to be too much whitespace at the bottom of each page -- the authors may need to follow the official template for formatting.\n\n---\n**Post-Rebuttal Updates**:  \nI'd like to thank the authors for providing very detailed responses and revisions to address my raised concerns above. I especially liked the additional analyses w.r.t. the noise-robust loss function, including its comparison to the standard cross entropy loss in terms of their optimization difficulty & optimal solutions, the empirical effectiveness of the chosen loss function (RCE loss) over other noise-robust training strategies and other alternative loss functions, and the relaxed theoretical assumption (Assumption 3 and Theorem 3). More detailed comments are below:\n* It's interesting to see that the noise-robust loss functions result in different optimization difficulties compared to the standard cross-entropy loss, though it's still a bit unclear to me whether their optimal solutions would be actually the same (or at least very similar). The loss surface plots and training curves are definitely good empirical evidence, but I'd be interested to know a more principled answer to this (of course, this is obviously beyond the scope of this paper, and I won't count this as a weakness of the paper at all).\n* Even if the optimal solutions induced by the noise-robust loss function concur with those of the standard cross-entropy loss, the **actual converged solutions** may differ due to optimization difficulties. Therefore, I believe that a more relaxed assumption (Assumption 3) is indeed necessary here, and I'm happy to see the new theoretical analyses built on this.\n* The new results on the three GLUE tasks and the ablations for comparing different loss functions/noise-robust training strategies look good to me.\n* The authors' responses to my further clarification questions all make sense to me. I think it'll be good to discuss the efficiency and time costs in the next paper version (I understand that the time cost is affordable due to the small classification model used, but it'll be beneficial to let the readers have a sense of how the method scales to larger models/datasets). Other than that, there is only one point that I don't quite get: \n\"Besides, our training process is “once for all”. In other words, once we derive the sample weights, we can use them to sample high-quality subsets at different scales.\"\n-> Do you intend to say that you could apply some filtering method based on the learned sample weights to select a subset of the synthetic training set in order to improve training efficiency?\n\nBased on the above considerations, I believe that the updated paper has addressed almost all of my concerns (the remaining issues are very minor and can be apparently fixed easily). Overall, this is a good paper with an important research goal, clear presentations, novel ideas, solid implementations, good theoretical insights, and promising empirical performance. I'll be happy to support the paper for acceptance.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "ePJuJsWL_MW", "reviewer": "Reviewer_mEap", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper focuses on the problem of utilizing pretrained large-scale language models for training data generation (based on careful prompts), to be used in training zero-shot classifiers. This approach stands out as an alternative to fine-tuning the language model for a new task (again for data generation) or direct prompt-based zero-shot classification.\n\nThe paper mainly aims to tackle the problem of managing sample noise in the training data generated by the language model, in an automatic way (avoiding human intervention). For this purpose a bi-level optimization approach (meta-learning-like) is proposed, where the outer loop estimates the sample training weights based on a robust loss function and the inner loop trains the classification model based on the given sample weights.", "summary_of_the_review": "Overall, the paper appears to be promising with strong positive results. However, it lacks in certain model and experimental aspects as discussed above. In particular, in my preliminary rating, I find the the lack of clarity regarding model selection and the lack of experiments evaluating major model choices (in particular on the effects of robust loss) as major experimental limitations.", "strength_and_weaknesses": "Positives:\n- The experimental results look very promising with significant improvements over sensible baselines\n- The approach overall is rational and makes sense\n- The theoretical analysis is a plus (though does not seem to give strong insights)\n\nNegatives:\n- Some claims in the paper are not fully convincing, e.g. \"In Sec. 4, we theoretically show that using Lrobust as the outer objective, our method can find a set of sample weights w∗ with just the synthetic validation set, such that w∗ maximizes the model performance on the clean data.\" This appears to be valid only for certain types of noises, one can easily come up with structured noise cases (as common in reality) where the weighting can easily attend to noisy samples. In particular, if the model generated samples are consistently problematic (rather than having uniform noise or being noisy only in a subset of samples), then the theoretical proof would not apply.\n- \"Since lrobust is now used to optimize the sample weights w, which is in a much smaller search space and has simpler structure than θ, thus is easier to optimize.\" Here, I get the intuition, however, I am not sure if it is definitely correct. In particular, one can see sample weights as a continuous relaxation to the sample selection problem. As sample selection is a combinatorial problem, it is likely to be a much harder optimization problem, compared to the naturally-continuous model parameter optimization problem (despite having a lot more optimization variables and similarly being non-convex).\n- The method is evaluated as a whole bundle with PLM. No baselines are presented as alternatives to various model components. I couldn't see a detailed model ablation study, and I wonder about alternative choices in certain parts of the model, e.g. what happens if we were to use a different robust loss or just xent loss in the outer loop?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "dsNQGV64to", "reviewer": "Reviewer_h9eZ", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The task this paper targeted is to train a model purely on a synthetic dataset without a clean oracle validation set. The paper proposed a bi-level algorithm where the coarse level uses a noise-robust loss optimizing re-weighting each data sample, and the fine level trains a model under the given weighted data sample dataset using regular cross-entropy loss. The noise-robust loss used is the reversed cross-entropy loss. The paper verified the effectiveness of this approach by both proved the convergency of this algorithm and empirically shows accuracy improvement over 8 text classification tasks.", "summary_of_the_review": "The bi-level approach proposed in this paper is intuitive and is supported by good experimental results. The lack of analysis on the choice of noise-robust loss function and the lack of baseline for training directly with reversed cross entropy loss weakens the arguments of this paper.", "strength_and_weaknesses": "Strength\n1. The bi-level optimization idea is novel and natural, and the proof is easy to follow. By using the noise-free loss over learning the data sample weighting instead of optimizing the model directly, the search space is much smaller and thus easier to optimize.\n2. The experiment results are sufficient and well-supported the effectiveness of this approach.\n3. Overall the paper is clear and well-written and easy to follow. \n\nWeaknesses:\n1. In the ablation study, it would be good to provide some analysis between the proposed bi-level framework verses a single level baseline where the model is optimized using reversed cross-entropy directly.\n2. In table 5, both ZEROGEN and SUNGEN's diversity is much smaller than Gold. It is unclear whether it is because of the model generated \"hard\" examples or noisy examples.\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "Z5avReQd1F4", "reviewer": "Reviewer_1YN5", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes SUNGEN, a framework to automatically construct high-quality data for zero-shot classification problems. The method first employs pretrained language models to generate samples for a given task and then learns the sample weights indicating data quality without requiring any human annotation. The paper provides theoretical proof that the proposed method can be noise-free. Moreover, experiments on eight text classification tasks demonstrate the strong performance of the proposed method and its ability to identify and ignore noisy samples.", "summary_of_the_review": "The paper proposes a novel method to automatically avoid noisy synthetic data in the paradigm of data-generation-based zero-shot learning. Clear analysis is provided both theoretically and empirically to show the effectiveness of the proposed method. The paper is worth acceptance.", "strength_and_weaknesses": "Strengths:\n1. The research problem of avoiding noisy samples in the new paradigm of data-generation-based zero-shot setting is well-motivated, and the underlying assumption of this work that there is no human annotation available for validation looks more realistic though challenging.\n2. The proposed method looks pretty convincing both theoretically and empirically. The experiment performance is very strong compared to several fair baselines.\n3. Further analysis especially the visualization in Figures 1 and 3 provides clear evidence for the hypothesis that the proposed method is effective for filtering out noisy labels.\n4. The paper is clear and well-written.\n\nWeaknesses:\nI didn't see significant weaknesses in this paper such that it should not be accepted. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: notable-top-25%