Pre-training Multi-party Dialogue Models with Latent Discourse Inference
Year: 2023

Abstract:
Multi-party dialogues are more difficult for models to understand than one-to-one twoparty dialogues, since they involve multiple interlocutors, resulting in interweaving reply-to relations and information flows. To step over these obstacles, an effective way is to pre-train a model that understands the discourse structure of multi-party dialogues, namely, to whom each utterance is replying. However, due to the lack of explicitly annotated discourse labels in multi-party dialogue corpora, previous works fail to scale up the pre-training process by putting aside the unlabeled multi-party conversational data for nothing. To fully utilize the unlabeled data, we propose to treat the discourse structures as latent variables, then jointly infer them and pre-train the discourse-aware model by unsupervised latent variable inference methods. Experiments on multiple downstream tasks show that our pre-trained model outperforms strong baselines by large margins and achieves state-of-the-art (SOTA) results, justifying the effectiveness of our method. The official implementation of this paper is available at https://github.com/EricLee8/MPD_EMVI.

heading: 1 Introduction
text: Dialogue system is an important area that has been studied for a long time in natural language processing field. Different from plain texts, dialogues are harder for models to understand since they are full of informal, colloquial expressions, and many ellipses (Yang and Choi, 2019; Reddy et al., 2019; Li et al., 2022). Among them, multi-party dialogues are even more complex since they involve multiple interlocutors, resulting in interweaving reply-to relations and information flows (Gu et al., 2021; Sun et al., 2021; Gu et al., 2022b). Specifically, in multi-party dialogues, the current utterance can be
∗Work done while interning at Tencent AI Lab. † Corresponding author. This paper was partially supported by Key Projects of National Natural Science Foundation of China (U1836222 and 61733011).
a reply to any preceding utterance in the dialogue history, forming complex discourse structures.
Intuitively, it is important for models to perceive the discourse structures, or in other words, to whom each utterance is replying, when comprehending multi-party dialogues. This intuition is in line with the process we humans participate in multi-party dialogues: we first read or listen to the dialogue history, knowing who speaks what to whom, then choose an utterance as the addressee, and finally utter a response. Literature has also justified that incorporating the discourse knowledge into models is beneficial for better understanding multi-party dialogues (Li et al., 2020; Jia et al., 2020; Li and Zhao, 2021; Ma et al., 2022). Unfortunately, the process of choosing addressees is a naturally unobservable action, resulting in a large amount of multi-party conversational data without addressee labels. In this work, we focus on leveraging the unlabeled data to pre-train a model for multi-party dialogue understanding.
To utilize the discourse structure, previous works seek help from human laborers to annotate the addressee labels on small datasets, where they either explicitly model the discourse structure using Graph Neural Networks or multi-task learning (Hu et al., 2019; Sun et al., 2021; Li et al., 2021; He et al., 2021; Gu et al., 2022a), or attempt to pretrain a model using objectives that are related to addressees by supervised learning (Gu et al., 2021). These works heavily rely on annotated addressee labels, which are rare in practice since the annotation process requires large amounts of human resources. As a result, they fail to be practical in real-world applications and are hard to scale up by utilizing more unlabeled multi-party conversational data.
To make full use of the unlabeled corpora, a natural idea is to treat the unobservable discourse structure (reply-to relations) as latent variables, then adopt latent variable models to jointly infer them and optimize the discourse-aware models. How-
9584
ever, it is not that simple when it comes to practice. For the Expectation-Maximization (EM) algorithm, the posterior distribution of the reply-to relations is intractable since it requires a square-level time complexity. If we turn to Variational Inference (VI) for help, the choice of the categorical prior distribution of the reply-to relations becomes troublesome: naive assumptions such as uniform distributions are too weak to make the training process converge.
To step over the above obstacles, we subtly combine the single-turn EM algorithm and multi-turn VI into a two-stage pre-training strategy. In the first stage, we adopt the EM algorithm to jointly model the context-response matching objective and singleturn addressee inference, which requires only a linear time complexity and can preliminarily guide the model to a relatively good converging point with utterance-level knowledge. In the second stage, we extend the latent variables from single-turn addressees to multi-turn reply-to relations and optimize the model via both the EM algorithm and VI framework, where the prior distribution of the reply-to relations is no longer troublesome since it can be derived exactly from the E-steps. This stage further enhances the model with discourse-level knowledge and guides it converge to a better point.
To sum up, the contributions of this work are: • We successfully scale up the pre-training for
multi-party dialogue understanding by leveraging the huge amounts of multi-party conversational corpora without addressee labels, while previous methods fail to work on these corpora. • We subtly combine the single-turn EM algorithm and multi-turn VI framework in a two-stage pretraining process, which equips the model with knowledge of different granularities and makes it converge to an ideal point. • The pre-trained model serves as a powerful encoder for multi-party dialogues and outperforms strong baselines by large margins, achieving SOTA results on multiple downstream tasks.

heading: 2 Related Works

heading: 2.1 Multi-party Dialogue Modeling
text: Several works have studied the modeling of multiparty dialogues before. Hu et al. (2019) propose to encode the reply-to relations with Graph Structural Networks (GSN). They utilize the addressee annotations and speaker information in the dataset to construct discourse and speaker graphs, then adopt a backward-forward strategy to pass mes-
sages between utterances. Sun et al. (2021); Gu et al. (2022a) further extend the modeling from homogeneous graphs to heterogeneous graphs by utilizing the Relational Graph Convolutional Networks to encode the heterogeneous information. However, their solutions all require annotated addressee labels in the multi-party dialogue dataset, which are rare and expensive to obtain in real-world applications. On the contrary, our work requires no addressee annotations, which saves human labors and can be scaled up using large unlabeled corpora.
Most related to our work, Li and Zhao (2023) attempts to improve the response generation model for multi-party dialogues by employing the EM algorithm to infer single-turn addressees. However, their approach encounters limitations when it comes to expanding the pre-training process due to the slow generative E-steps. Additionally, their work fails to fully exploit the discourse structure of the dialogue history, as they solely focus on the single-turn addressees. In contrast, our method not only scales up the pre-training by employing faster objectives, but also extends the latent variables from single-turn addressees to multi-turn reply-to relations to enhance the model with discourse-level knowledge, which is more important in comprehending multi-party conversations.

heading: 2.2 Dialogue Pre-training
text: To bridge the gap between pre-trained language models (PLMs) on plain texts and dialogue texts, many attempts have been made to pre-train a model for dialogues. Bao et al. (2020); Chen et al. (2022b) treat the dialogue intent as discrete or continuous latent variables to pre-train a model that solves the one-to-many problem in dialogue response generation task. Mehri et al. (2019); Xu and Zhao (2021); Zhang and Zhao (2021) design different self-supervised objectives for two-party dialogue context modeling. Different from their two-party setting, our work focuses on the multi-party scenario, where the addressee information should be concerned. Gu et al. (2021) also consider pretraining a model for multi-party dialogue understanding. They pre-train their model on a small dataset with annotated addressee labels by supervised addressee-related objectives. Since annotations are required, their pre-training strategy fails to scale up by using the unlabeled data. In contrast, our method is labor-free since the addressees are inferred by unsupervised latent-variable methods.

heading: 3 Methodology
text: In general, Figure 1 illustrates the overview of the proposed two-stage pre-training strategy. The left part illustrates the single-turn ExpectationMaximization process, where we iteratively conduct E-steps to infer the latent addressee zt (leftupper part and the green arrow), and M-steps to optimize the model via addressee-aware contextresponse matching (CRM) objective (left-lower part and the orange arrow). The right part illustrates the multi-turn Variational Inference process, which is incorporated into the EM framework in the second pre-training stage. We extend the latent variables from the single-turn addressees to multiturn addressee-graphs, and jointly optimize the discourse-aware context-response matching model (the blue arrow) and the graph-prediction model qϕ by Variational Inference. In the next sections, we will introduce the two pre-training stages in detail.

heading: 3.1 Single-turn Addressee Inference
text: As mentioned in Section 1, simply applying the EM algorithm to infer all reply-to relations in the dialogue requires a square-level time complexity, which is intolerably time-consuming for the pretraining on large corpora. To solve this issue, we step back in the first pre-training stage to focus on the modeling and inference of single-turn addressees. For one thing, it requires only a linear time complexity for each training instance and hence can be optimized via the EM algorithm. For another, the addressee distributions output by the Esteps can derive the prior distribution of the reply-to
relations, which can be utilized by the Variational Inference process in the second pre-training stage.

heading: 3.1.1 Preliminaries
text: Let’s consider the process that humans participate in a multi-party dialogue in the tth turn: we first read the dialogue history Ct−1, then choose an addressee utterance zt that we want to reply, and finally utter a response sentence rt. Formally, a multi-party dialogue corpus contains dialogues with format (Ct−1, zt, rt), where the annotations of zt are lacking in most corpora. Here Ct−1 = {S1: U1[SEP]S2: U2[SEP] . . . St-1: Ut-1[SEP]St}, where Si and Ui are the speaker and utterance of the ith turn, respectively. Addressee zt ∈ [1, t− 1] is a one-hot vector that indicates to whom we reply in the current turn t. In our settings, each utterance except the first one has exactly one addressee.
The conversation process can be formulated as pθ(rt|zt, Ct−1), which models the probability of rt being the correct response given Ct−1 and zt under trainable parameters θ. In large datasets without addressee labels zt, we should infer the unobservable latent addressees. To this end, we adopt the EM algorithm to iteratively infer the addressees pθ(zt|Ct−1, rt) during the E-steps, and optimize the model pθ(rt|zt, Ct−1) using the CRM objective during the M-steps.

heading: 3.1.2 Maximization Step
text: Suppose we have already obtained the inferred addressees from the E-step, two questions should be answered in the M-step: how to design the addressee-aware model architecture, and how to
design the CRM task that enforces the model to leverage addressee information.
To answer the first question, our solution is straightforward but effective: similar to the speaker or turn embeddings in previous works (Gu et al., 2020; Zhang et al., 2021), we add an addressee embedding on top of the token and positional embeddings to indicate which utterance is the current addressee. Note that we have also tried other addressee modeling methods such as the promptbased ones, yet they are not as effective as the addressee embeddings.
To answer the second question, we first follow the common practice to formulate the CRM task as a binary classification problem (Tao et al., 2021; Su et al., 2021), where the model should distinguish positive (correct) responses r+t from the negative ones r−t in the current dialogue turn t. To make the CRM task more addressee-related, besides simple negatives that are randomly sampled from the whole training corpus, we also construct hard negatives that are sampled from the later (> t turns) utterances in the same dialogue. Liu et al. (2019) point that simple negatives are easily distinguishable from positive ones by their topic differences. In other words, they can be predicted as negatives without the specified addressee information, which can not help the addressee inference process in the E-step. In contrast, the topic of each hard negative response is coherent with the current dialogue, making them hard to be classified with only the topic or sequential features. As a result, the model is forced to seek clues from the speaker and addressee information to distinguish those hard negatives, which greatly benefits the E-step.
With the model and training data at hand, we adopt binary cross-entropy loss as the objective function for the CRM task:
LCRM = −(yt × log[ pθ(rt|zt, Ct−1) ] + (1− yt)× log[ 1− pθ(rt|zt, Ct−1) ])
(1) Here yt ∈ {0, 1} is the ground truth label that indicates whether rt is a positive response. The left lower part and the orange arrow of Figure 1 illustrate the maximization step, where we ignore Ẑdt−1 since it will be introduced in Section 3.2.

heading: 3.1.3 Expectation Step
text: The inference of latent addressees can be formulated as calculating pθ(zt|Ct−1, rt). In other words, given the dialogue history Ct−1 and current re-
sponse rt, we should infer the posterior categorical distribution of the addressee zt ∈ [1, t − 1]. Consider the factorization of this posterior distribution:
pθ(zt|Ct−1, rt) = pθ(Ct−1, zt, rt) pθ(Ct−1, rt)
= pθ(Ct−1)× pθ(zt|Ct−1)× pθ(rt|zt, Ct−1)
pθ(Ct−1)× pθ(rt|Ct−1)
= pθ(zt|Ct−1)× pθ(rt|zt, Ct−1)
pθ(rt|Ct−1) (2)
where the factorization order of the numerator follows human habits when participating in a multiparty dialogue mentioned at the beginning of Section 3.1.1. In the denominator, pθ(rt|Ct−1) is irrelevant to zt. In the numerator, we assume a uniform prior distribution pθ(zt|Ct−1), hence this term is also irrelevant to zt. Hence, we can derive that:
pθ(zt|rt, Ct−1) ∝ pθ(rt|zt, Ct−1) (3)
Adopting this equation and the trained CRM model pθ(rt|zt, Ct−1) from the M-step, we can now calculate the posterior distribution of zt by traversing all possible addressees {zit}t−1i=1:
pθ(z i t|rt, Ct−1) = pθ(rt|zit, Ct−1)∑t−1 j=1 pθ(rt|z j t , Ct−1) (4)
The left upper part and green arrow in Figure 1 shows the E-step, where we ignore Zdt−1 since it will be introduced in Section 3.2.

heading: 3.2 Multi-turn Addressee-graph Inference
text: Once the EM iterations have reached a relatively good converging point, we dive into the second stage of training by additionally integrating the multi-turn Variational Inference task into the EM framework. This stage further enhances the model with discourse-level knowledge, making it possible to converge to a better point.
The discourse-level VI extends the latent variables from single-turn addressees zt to multi-turn addressee-graphs Zdt ∈ Rt×t, which is an adjacent matrix indicating to which addressee each utterance is replying to. In other words, the model now should infer all the addressees of each utterance Ui in the dialogue context Ct. As mentioned in Section 3.1, adopting the EM algorithm to infer Zdt is intolerably time-consuming. To solve this issue, we borrow the idea of Variational Inference (Kingma and Welling, 2014) to adopt a graph-prediction model qϕ(Zdt |Ct−1, rt) with additional trainable
parameters ϕ to predict the addressee-graphs. Formally, we maximize the log-likelihood of the observed data log pθ(rt|Ct−1) (conditioned on the dialogue history Ct−1) by improving its Evidence Lower Bound (ELBO):
ELBO(θ,ϕ; rt, Ct−1) =
Eqϕ(Zdt |rt,Ct−1)[log pθ(rt|Z d t , Ct−1)] −DKL(qϕ(Zdt |rt, Ct−1)∥pθ(Zdt |Ct−1)) (5)
Three important distributions are presented in this equation. First, pθ(rt|Zdt , Ct−1) is a new formulation of the CRM task, where single-turn addressees zt now becomes multi-turn addressee-graphs Zdt . Second, pθ(Zdt |Ct−1) is the conditional prior distribution of latent variable Zdt under parameters θ. Finally, qϕ(Zdt |Ct−1, rt) is the graph-prediction model, which predicts the edges from each response to its addressee by outputting the estimated posterior distribution of Zdt . Next, we introduce the modeling of these distributions in detail.

heading: 3.2.1 Discourse-aware CRM
text: Let’s start with pθ(rt|Zdt , Ct−1). Given the dialogue history Ct−1 and the addressee-graph Zdt sampled from qϕ, we model the CRM task by imitating careful human readers: when we seriously reply to an utterance in a multi-party dialogue, instead of focusing solely on the current addressee utterance zt itself, we tend to focus more on the utterances in the reply-chain of rt, namely, the k-hop ancestors of rt in the addressee-graph Zdt . Formally, we first extract the utterance representations of the k-hop ancestors of rt to form a reply-chain information representation Hkt ∈ Rk×d, then model pθ(rt|Zdt , Ct−1) with an MLP.
To accelerate the computation of the k-hop ancestors, we construct a one-hot vector at ∈ R1×t to indicate the position of the current response rt. Right-multiplying this vector by the addresseegraph matrix Zdt for i times yields the position vector of its ith ancestor. pθ(rt|Zdt , Ct−1) can now be formulated as follows:
Hkt = concat[{at(Zdt )i}k−1i=0 ] ·Hut ∈ Rk×d pθ(rt|Zdt , Ct−1) = σ(MLPθ(flatten(Hkt ))) (6)
Here concat[·] is concatenation, flatten means squeezing the matrix into a vector, MLPθ ∈ Rkd×1 is a linear projection and σ is the Sigmoid function. In this pre-training stage, pθ(zt|rt, Ct−1) and pθ(rt|zt, Ct−1) in the equations of Section
3.1 have now become pθ(zt|rt, Zdt−1, Ct−1) and pθ(rt|Zdt , Ct−1), respectively. For more detailed proofs, please refer to Appendix A.

heading: 3.2.2 Conditional Prior Distribution
text: Then, we focus on the conditional prior distribution pθ(Z d t |Ct−1). The choice of the prior distribution is vital to the convergence of Variational Inference (Kingma and Welling, 2014; Chen et al., 2022a). Previous works either make strong assumptions over the prior distribution, like Uniform and Gaussian (Qian et al., 2022), or use additional annotation models to approximate the prior distribution (Chen et al., 2022a). However, as mentioned in Section 1, they fail to work in our scenario since naive assumptions are too weak to make the training process converge. Thanks to the EM training process, the prior distribution pθ(Zdt |Ct−1) can be derived exactly from the previous t− 1 E-steps in this dialogue. Formally, it can be calculated as:
E(i) = pθ(zi|ri, Zdi−1, Ci−1) pθ(Z d t |Ct−1) = Πt−1i=1[E(i)] · U(|zt|)
(7)
Here U(|zt|) is a uniform distribution over the length of the candidates of zt. Due to the page limit, we put the detailed derivations of this equation in Appendix B. This equation subtly combines the EM training framework and the VI process, which guides the model converge to a better point by incorporating accurate prior knowledge of the discourse-level addressee-graphs.

heading: 3.2.3 Graph-prediction Model
text: Finally, we end with the graph-prediction model qϕ(Z d t |Ct−1, rt). To compute the edges between each utterance pair, we first apply mean pooling over the corresponding token representations of each utterance to get utterance-level representations Hut ∈ Rt×d. After that, we compute the score of each utterance pair being the responseaddressee by an MLP with trainable parameters ϕ to get a scoring matrix Su ∈ Rt×t. Finally, qϕ is calculated as follows:
qϕ = Gumbel-Softmax(S u +Mu) (8)
Here Mu ∈ Rt×t is a masking matrix with −∞ values on its upper triangular part to mask invalid positions, since each utterance can only reply to its previous ones. We adopt Gumbel-Softmax relaxation to make the sampling of qϕ differentiable, following Jang et al. (2017); Maddison et al. (2017).

heading: 3.3 Pre-training Objectives
text: Besides utterance-level CRM and discourse-level graph prediction, we also design an addresseeaware masked language modeling (MLM) task to preserve the token-level knowledge, which is introduced in detail in Appendix C. To sum up, the overall training objective in the M-step is:
L = LCRM + αLKL + βLMLM (9)
Here α and β are two hyper-parameters and are set to 0 at the first pre-training stage.

heading: 4 Experiments
text: In this section, we introduce the experimental settings and present the results on downstream tasks.

heading: 4.1 Pre-training Settings
text: For the pre-training data, we use the script of (Zhang et al., 2020) to download Reddit posts from 2005 to 2020 and extract multi-party conversations to create a pre-training corpus of 17,154,613 dialogues. Since the pre-training corpus is huge, we split it into trunks of data and perform EM iterations on each of them. For backbone models, we choose BERTbase (Devlin et al., 2019) and ELECTRAlarge (Clark et al., 2020). The former takes 4 days to converge in 8 NVIDIA A100 GPUs and the latter takes 12 days. For more details about the pre-training, please see Appendix D.

heading: 4.2 Downstream Settings
text: To test the capability of our pre-trained model, we conduct experiments on four downstream tasks based on multi-party dialogues.
Discourse Parsing requires the model to parse the reply-to links (addressee-graphs) in a multiparty dialogue and classify their relation types at the same time. For this task, we adopt Molweni (Li et al., 2020) as the benchmark dataset and use the F1 score of graph-prediction (F1G) and relation classification (F1RL) as the evaluation metrics.
Successful New Entry Prediction is to predict whether a newcomer’s message will be responded to by other participants in a multi-party dialogue, which is formulated as a binary classification task. For this task, we adopt SNEP (Wang et al., 2022) as the benchmark dataset and use Area Under Curve (AUC) and F1 score as the evaluation metrics.
Extractive Question Answering requires the model to extract an answer span from the dialogue context given a question. For this task, we also
adopt Molweni as the benchmark and use ExactMatch (EM) and F1 score as the evaluation metrics.
Response Generation aims at generating an appropriate response given the speaker and a specified addressee in a multi-party dialogue. For this task, we adopt Ubuntu IRC dataset (Hu et al., 2019) as the benchmark dataset and use BLEU, METEOR, and ROUGE-L as the evaluation metrics.
For more details about the datasets (statistics, data sources, etc.), please refer to Appendix E.
During the fine-tuning process, we discard the graph-prediction model qϕ since our model no longer requires explicit discourse modeling thanks to the implicit discourse knowledge learn from the pre-training. In our experiments, we make taskspecific designs for each downstream task to fully utilize the addressee embedding to lay emphasis on important utterances that are not necessarily addressees, hence we call it Adaptation Model. For more details about the task-specific designs, please refer to Appendix F. To test the universality and simplify the usage of our pre-trained model, experiments are also conducted where we discard the addressee embedding and use only the parameters that are exactly the same as BERT, hence we call it Vanilla Model. Following previous works (Li et al., 2020; Gu et al., 2021; Wang et al., 2022), we mainly conduct our experiments based on BERTbase.
In Table 1, MPC-BERT (Gu et al., 2021) is introduced in Section 2.2, which is pre-trained on a small dataset with annotated addressee labels using supervised learning. BERT+CRM is an ablation model that is pre-trained using only the first stage (but with full data), which means only the CRM loss and EM training are adopted. +MLM means addressee-aware MLM objective is further added in the pre-training process and +VI represents our full model with two-stage pre-training. To study whether two-party dialogue models can still work in the multi-party scenario, we also conduct experiments on SPIDER-BERT (Zhang and Zhao, 2021), which is a model pre-trained on two-party dialogues using self-supervised objectives.

heading: 4.3 Experimental Results
text: We can see from Table 1 that our full model (+VI) significantly outperforms BERTbase and MPCBERT on all tasks, justifying the effectiveness of discourse knowledge modeling by incorporating VI into the EM training framework with two-stage pretraining. Besides, BERT+CRM is already strong
enough to outperform MPC-BERT or to achieve comparable results, demonstrating the importance of scaling up the pre-training by EM algorithm and incorporating turn-level addressee knowledge. Also, adding addressee-aware MLM adds to performance gains, yet relatively slight. Finally, SPIDERBERT performs relatively worse than multi-party models, which indicates the significance of designing models and objectives that are specific for multiparty dialogues. For more analyses about why the two-party objectives fail to work on the multi-party scenario, please refer to Appendix G.
Another observation is that the performance drops of the Vanilla Model compared with Adaptation Model is relatively minor on all dataset, which means it remains powerful even without the taskspecific designs. This observation demonstrates that the discourse knowledge is indeed learned and stored in our pre-trained model.
Besides BERTbase, we also experiment with ELECTRAlarge to investigate whether our method can still enhance stronger PLMs. In this experiment, we compare the original ELECTRAlarge and our full model under the setting of Adaptation Model. As shown in the lower part of Table 1, our model outperforms ELECTRAlarge by large margins. This observation reveals that even strong PLMs, such as ELECTRAlarge, still lack the knowledge to well understand multi-party dia-
logues, while our method can effectively enhance them by leveraging the discourse information inferred from the unlabeled datasets.
Our model can also improve the performance of response generation by enhancing the encoder side. Table 2 presents the results on the Ubuntu IRC dataset, where GSN (Hu et al., 2019) and HeterMPC (Gu et al., 2022a) utilize the discourse annotations in this dataset to explicitly model the reply-to relations by constructing homogeneous or heterogeneous graph neural networks. In contrast, the annotations are not used by our model since it is able to implicitly capture the reply-to information by the discourse knowledge learned during pre-training. As shown in Table 2, our model outperforms previous models even under the condition that we do not use additional annotations, demonstrating the strong capability of our model to understand the discourse structures.

heading: 5 Analyses
text: In this section, we make in-depth analyses to investigate more insights from our method.

heading: 5.1 Ablation Study
text: Since our model is trained on massive amounts of data, a natural question is whether the performance gains are from just seeing more conversations. To investigate this, we conduct experiments by remov-
ing the addressee-aware EM training process and only performing normal CRM and MLM on the full data. Also to test the out-of-domain generalization ability of our model, for this ablation experiment, we choose SNEP-Twitter and Discourse Parsing tasks since their data sources (Twitter and Ubuntu) are different from our pre-training source (Reddit).
Table 3 shows the ablation results, where we observe sharp performance drops when removing the EM training. This observation demonstrates the strong robustness and transferability of our model in out-of-domain data, thanks to the addressee knowledge learned from the EM process.

heading: 5.2 Zero-shot Graph-Prediction
text: To investigate to what extent the discourse knowledge is learned by our model, we test the zeroshot graph-prediction task on both Reddit and Molweni datasets. Note that during the pre-training stage, our model is trained on the pseudo-addresseegraphs that are inferred from the unlabeled dataset, hence we call this experiment zero-shot. Table 4 shows the F1G scores of both datasets, where we observe good in-domain performance in Reddit and out-of-domain generalizability in Ubuntu (the Molweni dataset).

heading: 5.3 Addressee Distribution Shifts
text: At the beginning of our pre-training process, there are no annotated addressee labels in the training corpus, and the initial model is too weak to infer reasonable addressees using Eq. (4). To cold-start the EM bootstrapping process, we simply set the addressee of every response to be the last utterance
in the dialogue history (i.e., Ut−1), then perform the first round of M-step. This cold-start approach is different from, and much simpler than Li and Zhao (2023), where they utilize a trained discourse parser to label the addressees for the first M-step.
This strategy is simple but exhibits surprisingly good convergence: the distribution of the inferred addressees shifts from one-hot (the initial distribution) to a distribution that is close to the real addressee distribution in an annotated validation set, just after a few trunks. Figure 2 illustrates the distribution shift, where we draw the validation addressee distance distribution of the last E-step on each trunk. At the initial point, the addressees are all set to the last utterance, hence the percentage of addressees with distance 1 is 100%. With the increase of truck numbers, the addressee distance distribution gradually shifts and becomes closer and closer to the real distribution.

heading: 5.4 Pre-training Trending
text: Figure 3 illustrates the trending of both CRM scores (MRR and Recall@1) and addressee pre-
diction accuracy of ELECTRAlarge during the pretraining process. After the 10th trunk (the second pre-training stage), we compute the average and standard deviation over the ±10 trunks of the index and show them in the figure as lines and shades.
First, we can see that both metrics grow together and mutually, which indicates with a stronger CRM model comes better addressee prediction accuracy, demonstrating the correctness of Eq. (3). Besides, the first stage of training reaches its convergence at around the 10th trunk, by further incorporating VI at this point, both metrics keep growing and reach their top at around the 120th trunk. Finally, the standard deviation is large at the beginning of the second stage of pre-training but gradually decreases with the convergence of the model.

heading: 6 Conclusion
text: In this paper, we point out that the lack of annotated addressee labels hinders the scaling-up of multi-party dialogue pre-training. To overcome this obstacle, we propose to utilize the unlabeled datasets by combining the EM algorithm and Variational Inference to jointly infer the discourse labels and pre-train the model with discourse-aware objectives on different granularities. Experimental results and extensive analyses have justified the effectiveness and transferability of our model on multiple downstream tasks.
Limitations
Despite the contributions of our work, there are also unavoidable limitations of it.
First, our method is based on the setting that each utterance in the dialogue except the first one has exactly one addressee. This setting holds tightly in online forums such as Twitter or Reddit, yet has its limit in group chats or meetings, where an utterance can reply to multiple or no addressees. However, this scenario is relatively rare in multiparty conversations. Considering this scenario is challenging and complicated since the one-to-many reply-to relations can cause the single-turn EM algorithm intractable. For this part, we leave it to future works.
Second, the Ubuntu IRC benchmark of response generation task is extracted from the Ubuntu Chat Corpus (Lowe et al., 2015), where people discuss the technical issues on the Ubuntu operating system. Due to the lack of human annotators with knowledge of Linux and Ubuntu, we do not con-
duct human evaluations on this dataset. However, we do provide the generated responses in our supplementary materials for those who are interested in the human evaluations.

heading: A Derivation of E-step in Stage-2
text: In the second stage, the maximization step becomes the modeling of pθ(rt|Zdt , Ct−1), and the expectation step becomes computing the posterior distribution of pθ(zt|rt, Zdt−1, Ct−1), accordingly. We also factorize this posterior distribution and omit θ for simplicity:
p(zt|rt, Zdt−1, Ct−1) = p(zt, Ct−1, Zdt−1, rt)
p(Ct−1, rt, Zdt−1)
= p(Ct−1) p(Zdt−1|Ct−1) p(rt, zt|Ct−1, Zdt−1) p(Ct−1)p(Zdt−1|Ct−1) p(rt|Ct−1, Zdt−1) = p(zt|Ct−1, Zdt−1) p(rt|Ct−1, Zdt−1, zt)
p(rt|Ct−1, Zdt−1) (10)
In this equation, the factorization also follows human habit when we seriously participate in a multiparty dialogue: we first read the dialogue history (Ct−1), then analyze the discourse structure (replychains) of it (Zdt−1|Ct−1), then choose an addressee utterance we want to reply (zt|Zdt−1, Ct−1), and finally utter a response to it (rt|zt, Zdt−1, Ct−1). In the last row of this equation, the denominator is irrelevant to zt, and we also assume uniform distribution of p(zt|Ct−1, Zdt−1) in the numerator, which is also irrelevant to zt. At this point, we can derive that:
p(zt|rt, Zdt−1, Ct−1) ∝ p(rt|zt, Zdt−1, Ct−1) (11)
and calculate the posterior distribution of zt by traversing all possible addressees {zit}t−1i=1:
p(zit|rt, Zdt−1, Ct−1) = p(rt|zit, Zdt−1, Ct−1)
t−1∑ j=1
p(rt|zjt , Zdt−1, Ct−1) (12)

heading: B Derivation of Prior Distribution
text: We now derive how to compute the conditional prior distribution pθ(Zdt |Ct−1), where we also omit θ for simplicity. Firstly, we have
p(Zdt |Ct−1) = p(zt, Zdt−1|Ct−1) = p(Zdt−1|Ct−1) p(zt|Ct−1, Zdt−1)
(13)
Here p(zt|Ct−1, Zdt−1) is assumed to be a uniform distribution in Appendix A, so we have:
p(zt|Ct−1, Zdt−1) ∼ U(|zt|) (14)
where |zt| is the length of the candidates of zt. We now focus only on p(Zdt−1|Ct−1). Let’s note E(t) = p(zt|rt, Zdt−1, Ct−1), we have:
p(Zdt−1|Ct−1) = p(z1, z2, . . . , zt−1|Ct−1) = p(z1|Ct−1) . . . p(zt−1|z1, . . . zt−2, Ct−1) = Πt−1i=1 p(zi|Zdi−1, Ct−1) = Πt−1i=1 p(zi|Zdi−1, Ci) = Πt−1i=1 p(zi|ri, Zdi−1, Ci−1) = Πt−1i=1 [E(i)]
(15) In this equation, we use an intuitive constrain that p(zi|Zdi−1, C≥i) = p(zi|Zdi−1, Ci) and t − 1 ≥ i, since in real-world scenario, we can not see the future dialogue contexts. Combining Eq. (14) and (15), we get:
pθ(Z d t |Ct−1) = Πt−1i=1[E(i)] · U(|zt|) (16)
which is exactly the same as Eq. (7).

heading: C Masked Language Modeling Details
text: For addressee-aware masked language modeling (MLM) object described in Section 3.3, the three kinds of special words are masked with a higher probability. Specifically, for normal words, we mask them with a probability of 15%, for special words, the probability is 60%. The special words are randomly masked first. If the total masking ratio is over 30%, we randomly cancel some masks to reduce it below 30%. If the total masking ratio is below 15%, we repeat the masking process on those normal words to make the final masking ratio from 15% to 30%.

heading: D Pre-training Details
text: As mentioned in Section 4.1, we split the pretraining data into several trunks and perform EM iterations on each of them. In our experiment, each trunk contains 600,000 (Ct−1, r +/− t ) pairs and the total number of trunks is 158. We perform 3 EM iterations for each trunk. At the end of each trunk, we will load data from the next trunk and perform E-step to infer the initial addressees for the first M-step of the next trunk. Note that the addressee initialization of the first trunk is a heuristic that sets the addressees of all response to the last utterance in the dialogue history, which is mentioned in Section 5.3.
After each E-step, we do not use all the training samples for the next M-step. Instead, we pick the samples with top 50% addressee prediction confidence scores for the next round of M-step. The confidence score is hard to design since simply adopting the highest probability calculated by Eq. (4) will cause length bias: dialogues with shorter context length will have larger highest probability. To solve this issue, we adopt two normalizing methods to normalize the logits output by the model to the same scale, and use the difference between the largest logits and the second largest logits max− second_max to indicate the confidence level. Specifically, the two normalizing methods are min-max normalizing and average normalizing, respectively:
smin-maxi = si −min(S)
max(S)−min(S)
saveragei = si −min(S)
avg(S)
(17)
Here S = {si}t−1i=1 is the logits scores output by the model. For each E-step, we compare the addressee prediction accuracy of the top 50% samples of both normalizing methods in the validation set, then choose the higher one as the normalizing method to select samples for the next round of M-step in the training set.
To preserve the knowledge learned from the previous trunks and meanwhile fully utilize the newly inferred addressees in each E-step, we remain the parameters of the PLM unchanged and re-initialize the parameters of the addressee embeddings and CRM classifier after each E-step. For the second pre-training stage, we also keep the parameters of the graph-prediction model unchanged.
We start the second stage of pre-training when the vanilla EM algorithm comes to its convergence. Specifically, when the addressee prediction accuracy stops to increase for continuous three trunks, we consider the EM iterations have converged and start the second stage of training by enabling the KL loss and switch the CRM model to the discourseaware version. In our experiment, the EM algorithm converges at around the 10th trunk. In the second stage of pre-training, the hyper-parameters in Eq. (9) are set to α = 1.0 and β = 0.5, respectively.
We adopt Simulated Annealing during the Variation Inference to make the pre-training process stable and converge better. Specifically, the temper-
ature coefficient τ of Eq. (8) is set to a high value (10.0) at the beginning of the second pre-training stage, then gradually decreases 0.1 with the graphprediction model getting stronger and stronger. Formally, in the ith trunk of the second pre-training stage, τ is calculated as τ = max(0.1, 1n−0.9).

heading: E Dataset Details
text: Molweni is a multi-party dataset for both discourse parsing and question answering tasks. It is sampled from the Ubuntu Chat Corpus (Lowe et al., 2015) and is annotated with question-answer pairs and discourse relations (reply-to links and edge types). This dataset contains multi-party dialogues discussing technical issues on the Ubuntu System, hence its topic and domain are very different from our pre-training corpus Reddit. Despite this, our model still generalizes well on this dataset by outperforming the baseline models by large margins. Table 5 shows the statistics of the Molweni dataset, where each utterance is annotated with its addressee and the relation type, each dialogue is annotated with several questions.
Successful New Entry Prediction (SNEP) is a multi-party dialogue dataset taken from Reddit and Twitter posts. This task is to predict whether a newcomer’s message will be replied to by other users in a multi-party dialogue. This task would be an important part of the research in online assistants and social media. Table 6 shows the statistics of the SNEP dataset, where Reddit and Titter are two subsets.
Ubuntu IRC Benchmark is a dataset for multiparty dialogue response generation task. This dataset is also from the Ubuntu Chat Corpus (Lowe et al., 2015) and contains annotated addressee labels for each utterance. The generation task is formulated as follows: given the dialogue history and
a specified addressee, the model should generate an appropriate response that is well related to the addressee. This dataset contains around 380,000 dialogues in total. For developing and testing set, there are 5,000 dialogues, respectively. For the evaluation scripts to compute ROUGE, METEOR, and BLEU, we use the same script as (Gu et al., 2022a).

heading: F Adaptation Model Details
text: To make full use of the pre-trained addressee embedding, we design task-specific adaptation method for each downstream task.
For discourse parsing, the use of addressee embedding happens after the reply-to links are predicted. For each reply-to link, we model the addressee (the utterance that is pointed by another) with the addressee embedding and perform the relation classification.
For successful new entry prediction, we infer the addressee of the response to be studied (to predict whether it is a successful new entry) and adopt the addressee embedding to encode the dialogue. We perform mean pooling over the tokens of the response to get a vector, then adopt a binary classifier to make the final prediction.
For extractive question answering, we treat the question ans "response" and the utterance that contains the final answer (key-utterance) span as "addressee". Specifically, during training, we construct key-utterance labels with the annotated answer span and add an auxiliary key-utterance prediction module to predict the key-utterances. We adopt teacher forcing to model the answer span prediction task with the guidance of ground-truth key-utterance information by indicating the keyutterance with the addressee embedding. During inference, we first infer the key-utterance by the key-utterance prediction module, then use the predicted ones to model the answer span prediction task.

heading: G Failure of Two-party Objectives
text: Let’s take some common objectives of two-party dialogue pre-training for example.
First, consider the Utterance Order Restoration (UOS) objective that aims to restore the order of permutated utterances in two-party dialogues, or similarly the Utterance Swap Detection (USD) objective that determines whether there exists swapped utterances in the context. In multi-
party dialogues, the order of two utterances that reply to the same root-utterance can be swapped, making these two objective inapplicable.
Second, consider the Utterance Restoration and Response Generation/Selection objectives, where the former restores masked utterance tokens using MLM and the latter generates or selects the ground truth response. These objectives can be too difficult for the model to learn without addressee information, due to the one-to-many problem of responseto-context when given different addressees.
The key motivation of this paper and the most difficult part of adopting self-supervised learning on multi-party dialogue is the lack of addressee information, which is subtly addressed by our EM+VI pre-training approach.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:

heading: 3 A1. Did you describe the limitations of your work?
text: The last section.
A2. Did you discuss any potential risks of your work? Not applicable. Left blank.

heading: 3 A3. Do the abstract and introduction summarize the paper’s main claims?
text: Section 1.
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? Section 3 and 4.

heading: 3 B1. Did you cite the creators of artifacts you used?
text: Section 4.
7 B2. Did you discuss the license or terms for use and / or distribution of any artifacts? They are all publicly available and the license are available on GitHub.

heading: 3 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided
text: that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Section 3 and 4.
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Not applicable. Left blank.

heading: 3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and
text: linguistic phenomena, demographic groups represented, etc.? Appendix E.
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Appendix E.
C 3 Did you run computational experiments? Section 4 and 5.

heading: 3 C1. Did you report the number of parameters in the models used, the total computational budget
text: (e.g., GPU hours), and computing infrastructure used? Section 4.
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.

heading: 3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found
text: hyperparameter values? Section 4 and Appendix D.

heading: 3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary
text: statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Section 4.

heading: 3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did
text: you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Appendix E.
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? Not applicable. Left blank.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? Not applicable. Left blank.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? Not applicable. Left blank.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? Not applicable. Left blank.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? Not applicable. Left blank.

heading: Future Work
text: zt−2, Ct−1) = Πt−1i=1 p(zi|Zdi−1, Ct−1) = Πt−1i=1 p(zi|Zdi−1, Ci) = Πt−1i=1 p(zi|ri, Zdi−1, Ci−1) = Πt−1i=1 [E(i)]
(15) In this equation, we use an intuitive constrain that p(zi|Zdi−1, C≥i) = p(zi|Zdi−1, Ci) and t − 1 ≥ i, since in real-world scenario, we can not see the future dialogue contexts. Combining Eq. (14) and (15), we get:
pθ(Z d t |Ct−1) = Πt−1i=1[E(i)] · U(|zt|) (16)
which is exactly the same as Eq. (7)