# Backdoor Attack Against Dataset Distillation in Natural Language Processing
## 2 Related Work
### Dataset Distillation Techniques
#### 2.4.1 DwAL  
DwAL modifies the metalearning-based dataset distillation approach put forward by [11] for application to text datasets, and incorporates attention labels that directly influence the probabilities of model self-attention. Consider the original training dataset \(D=\{(x_{i},y_{i})\}_{i=1}^{N}\), where each pair \((x_{i},y_{i})\) consists of an input and its corresponding class label; the objective of DwAL is to optimize a distilled dataset \(\tilde{D}=\{(\tilde{x}_{i},\tilde{y}_{i})\}_{i=1}^{M}\), which is initially set up randomly with \(M\ll N\). The model parameters \(\theta\) are refined through gradient descent (GD) steps using mini-batches from the distilled dataset \((\tilde{x}_{m},\tilde{y}_{m})\). Starting with initial model parameters \(\theta_{0}\), DwAL trains the model on the distilled dataset \(\tilde{D}\) for a total of \(T\) GD steps. Given that the aim of dataset distillation is for \(\theta_{T}\) to achieve good performance using the initial dataset, the optimization goal for the distilled dataset \(\tilde{D}\)is defined as follows:  
\[\mathcal{L}_{distill}(\tilde{D},\tilde{\eta};\theta_{0}):=L(x_{m},y_{m},\theta _{T}) \tag{1}\]  
\[=L(x_{m},y_{m},F(\theta_{0};\tilde{D},\tilde{\eta},T)) \tag{2}\]  
where \((x_{m},y_{m})\) represents a mini-batch from the original training dataset, \(L(\cdot)\) denotes a twice-differentiable loss function, \(\tilde{\eta}\) is the model's learning rate, which is optimized alongside the distilled dataset \(\tilde{D}\), and \(F(\cdot)\) refers to the training process over the gradient descent update steps \(T\). Consequently, the optimization challenge for dataset distillation can be expressed as follows:  
\[\tilde{D}^{*},\tilde{\eta}^{*}=\operatorname*{arg\,min}_{\tilde{D},\tilde{ \eta}}\,\,\mathbb{E}_{\theta_{0}\sim p(\theta_{0})}\big{[}\mathcal{L}_{\text{ distill}}\,\big{(}\tilde{D},\tilde{\eta};\theta_{0}\big{)}\big{]} \tag{3}\]  
where \(p(\theta_{0})\) is the distribution of \(\theta_{0}\). \begin{table}
\begin{tabular}{c c c c} \hline \hline
**Defense Methods** & **Type** & **Victim Models** & **Trigger Granularity** \\ \hline T-miner [80] & Model diagnosis & Transformer & Sentence-level \\ \hline Fine-pruning [81] & Model diagnosis & BERT & Word-level \\ \hline NAD [82] & Model diagnosis & BERT & Word-level \\ \hline BKI [83] & Dataset cleaning & Bi-LSTM & Sentence-level \\ \hline Trigger Breaker [84] & Dataset cleaning & BERT & Sentence-level \\ \hline ONION [74] & Trigger filtering & BERT & Sentence-level, \\  & & & Word-level \\ \hline DARCY [85] & \begin{tabular}{c} Poisoned text \\ detection \\ \end{tabular} & BERT & Word-level \\ \hline STRIP [77] & \begin{tabular}{c} Poisoned text \\ detection \\ \end{tabular} & Bi-LSTM & Word-level \\ \hline RAP [75] & \
\begin{tabular}{c} Poisoned text \\ detection \\ \end{tabular} & BERT &
\begin{tabular}{c} Word-level, \\ Sentence-level \\ \end{tabular} \\ \hline BFClass [86] & Dataset cleaning & BERT & Word-level \\ \hline LFR+R\&C [86] & Dataset cleaning & BERT & Word-level \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Summary of existing backdoor defenses. DwAL optimizes the distilled dataset \(\tilde{D}\) based on this objective by employing contemporary gradient-based optimization methods. To circumvent the challenges associated with directly optimizing discrete text, DwAL utilizes a series of embedding vectors for the inputs in the distilled dataset rather than using the text itself. This approach allows the loss \(\mathcal{L}_{distill}\) to be differentiable with respect to \(\tilde{D}\), enabling DwAL to optimize the distilled dataset through gradient techniques. Drawing inspiration from prior research [87; 88; 89], DwAL calculates the Kullback-Leibler (KL) divergence \(D_{KL}\) between the model's self-attention probabilities \(a(\theta)\) and the distilled attention labels \(\bar{a}\) throughout all layers and attention heads.