# Enhancing Task-Oriented Dialogue Modeling through Coreference-Enhanced Contrastive Pre-Training
## 3 Methodology
### Dialogue Semantic Pre-Training  
Generally, a dialogue \(d\) is composed of a series of constituent utterances, denoted as \(u_{1},u_{2},\ldots u_{N}\). Each utterance typically consists of multiple tokens. We employ a multi-layer Transformer architecture and extract token representations from the hidden vectors of the final layer. Furthermore, to differentiate between the roles of the speakers in the dialogue, we prepend each utterance with two distinct tokens, namely, [_USR_] and [_SYS_]. Additionally, we incorporate a special [_CLS_] token into the beginning of the dialogue.  
As shown in Figure 2, the semantic contrastive model takes the current dialogue utterance, the previous dialogue utterances and the coreference parsed results as the input, adopts a PLM as the text encoder and trains to discriminate various coreference pairs mentioned in the dialogue. In detail, we employ a coreference discriminator as our semantic contrastive pre-training task, with the aim of fostering proximity in the representations of words or mentions that share coreference, as opposed to those with distinct coreference relationships. We leverage these coreference pairs as positive samples, training the text encoder to distinguish them from pairs exhibiting different coreference connections, serving as negative samples. Consequently, this approach enables the encoder to capture dialogue coreference semantics without requiring human annotations.  
Suppose a pair \((c,c^{\prime})\) is a positive example since they share coreference, like _Monday_ and _the same day_ in Figure 2, and a negative example otherwise. In detail, we replace one mention \(c\) in the original positive pair with another one from the negative coreference mentions labeled as \(\hat{c}\) (e.g., _a train_) by random sampling. And then, we obtain the \(m_{c}\) number of negative pairs.  
Figure 1: An overview of the proposed Coreference-Enhanced Contrastive Pre-training (CECPT).  
To distinguish the positive coreference pair from the negative ones, we establish a training objective for positive pairs (\(c\), \(c^{\prime}\)) using the contrastive loss of classifying the positive pair correctly:  
\[\mathcal{L}_{c,c^{\prime}}=-\ \mathbf{x}_{c}^{\top}W\mathbf{x}_{c^{\prime}}+ \log\left(\exp\left(\mathbf{x}_{c}^{\top}W\mathbf{x}_{c^{\prime}}\right)+ \sum_{i=1}^{m_{c}}\exp\left(\mathbf{x}_{c_{i}}^{\top}W\mathbf{x}_{c^{\prime}} \right)+\sum_{j=1}^{m_{d}}\exp\left(\mathbf{x}_{c}^{\top}W\mathbf{x}_{c^{\prime }_{j}}\right)\right), \tag{1}\]  
where \(\mathbf{x}\) is the representation of the coreference mentions (e.g., \(c\)) and \(W\) is a matrix learning the similarity of the mention pair. \(m_{c}\) and \(m^{\prime}_{c}\) are hyper-parameters for negative sampling.  
Therefore, to obtain the overall training objective for coreference semantic pre-training, we sum up the losses of all the positive pairs in the set \(P_{d}\) from all the dialogues \(d\) in the batch \(B_{d}\):  
\[\mathcal{L}(\theta)=\sum_{d\in\mathbb{B}_{d}}\sum_{(c,c^{\prime})\in\mathbb{P }_{d}}\mathcal{L}_{c,c^{\prime}}, \tag{2}\]  
where \(\theta\) denotes the trainable parameters.