# Backdoor Attack Against Dataset Distillation in Natural Language Processing
## 2 Related Work
### Dataset Distillation Techniques
#### 2.4.2 DiLM  
DiLM converts datasets into textual data instead of word embeddings, thereby improving the model-independent usability and interpretability of the produced datasets. The fundamental idea behind DiLM is to optimize the continuous parameters of a language model as the target for dataset distillation, thereby avoiding challenges related to optimizing discrete text. Given a training dataset \(D_{real}=\{x_{i}\}_{i=1}^{|D_{real}|}\), DiLM seeks to create a generator model parameterized by \(\phi\) that generates a distilled synthetic dataset \(D_{syn}=\{\hat{x}_{i}\}_{i=1}^{|D_{syn}|}\), where \(|D_{syn}|\ll|D_{real}|\), with the goal of ensuring the effective performance of a learner model parameterized by \(\theta\) and trained on \(D_{syn}\). Initially, the generator model is trained solely to produce synthetic training samples. DiLM utilizes transformer language models as generators to create high-quality synthetic training samples that are suitable for subsequent modeling tasks. To assist the generator in producing class-specific outputs, DiLM includes class-specific beginning-of-sentence tokens \(<\mathit{bos}_{i}>\) added at the start of each training sample, allowing the generator to generate results aligned with their respective classes. Furthermore, an end-of-sentence token \(<\mathit{eos}>\) is appended at the end of each training sample before it is fed into the generator model, as follows:  
\[<\mathit{bos}\_{i}>\mathit{sentence}\;\;of\;\;class\;\;i<\mathit{eos}>.\]  
To facilitate text classification tasks that elucidate the connections between sentence pairs, including semantic similarity and natural language inference (NLI), DiLM employs a specific token \(<\mathit{sep}>\) to separate the two sentences, as shown below:  
\[<\mathit{bos}\_{i}>\mathit{sentence}\;\;1<\mathit{sep}>\mathit{sentence}\;\;2< \mathit{eos}>.\]  
The generator model is trained with the language modeling loss \(l_{\phi}(x_{i})\) using the previously-mentioned training samples. In this way, DiLM pretrains the parameters \(\phi\) of the generator model, producing synthetic training data that closely resemble the actual data. This synthetic dataset acts as the initial parameter for gradient-matching training. Next, the generator model is fine-tuned to produce training samples that are \"informative\" by reducing the gradient matching loss between the generated and real samples. To transfer knowledge from the initial dataset \(D_{real}\) into synthetic samples created by the generator, DiLM optimizes the gradient matching loss [13] for dataset distillation as its objective. Considering a small batch of actual samples \(\{x_{i}\}_{i=1}^{M}\) alongside a small batch of synthetic samples \(\{\hat{x}_{i}\}_{i=1}^{N}\) produced by the model, the gradient matching loss \(L_{GM}\) concerning learner model parameters \(\theta\) is calculated as follows:  
\[\mathcal{L}_{GM}=D(\
abla_{\theta}\mathcal{L}_{real},\
abla_{\theta}\mathcal{ L}_{syn}),\quad\text{where}\mathcal{L}_{real}=\frac{1}{M}\sum_{i=1}^{M}l_{ \theta}(x_{i}),\quad\mathcal{L}_{syn}=\frac{1}{N}\sum_{i=1}^{N}l_{\theta}( \hat{x}_{i}), \tag{5}\]  
where \(l_{\theta}(\cdot)\) represents the loss function and \(D(\cdot,\cdot)\) denotes the distance function based on cosine similarity, defined as  
\[D(A,B)=1-\frac{A\cdot B}{\|A\|\|B\|}. \tag{6}\]  
To tackle the challenge that the gradient matching loss \(\mathcal{L}_{GM}\) cannot be propagated back to the generator model parameters \(\phi\) directly through the generated samples \(\{\hat{x}_{i}\}_{i=1}^{N}\), DiLM implements an alternative backward propagation strategy. It enhances a tokenization model for the loss associated with downstream tasks through a nondifferentiable method. In computing the loss \(L_{syn}\) for generated samples as demonstrated in Equation (5), these losses are weighted according to their respective generation probabilities \(p_{\theta}\{\tilde{x}_{i}\}\) rather than merely averaging the losses of each sample.