DIALOG-POST: Multi-Level Self-Supervised Objectives and Hierarchical Model for Dialogue Post-Training
Year: 2023

Abstract:
Dialogue representation and understanding aim to convert conversational inputs into embeddings and fulfill discriminative tasks. Compared with free-form text, dialogue has two important characteristics, hierarchical semantic structure and multi-facet attributes. Therefore, directly applying the pretrained language models (PLMs) might result in unsatisfactory performance. Recently, several work focused on the dialogue-adaptive post-training (DialPost) that further trains PLMs to fit dialogues. To model dialogues more comprehensively, we propose a DialPost method, DIALOG-POST, with multi-level self-supervised objectives and a hierarchical model. These objectives leverage dialogue-specific attributes and use selfsupervised signals to fully facilitate the representation and understanding of dialogues. The novel model is a hierarchical segment-wise self-attention network, which contains innersegment and inter-segment self-attention sublayers followed by an aggregation and updating module. To evaluate the effectiveness of our methods, we first apply two public datasets for the verification of representation ability. Then we conduct experiments on a newly-labelled dataset that is annotated with 4 dialogue understanding tasks. Experimental results show that our method outperforms existing SOTA models and achieves a 3.3% improvement on average.

heading: 1 Introduction
text: As an indispensable way of communication, dialogue is related to many research and application scenarios in academia and industry. Better dialogue representation and understanding serve for several tasks, including intent classification, emotion recognition, and response selection, thus how to represent and model dialogues is an essential topic. Compared with free-form text, dialogue modeling has to pay more attention to the following
∗Corresponding author.
characteristics: (1) hierarchical semantic structure (Serban et al., 2016; Xing et al., 2018; Zhang et al., 2019), i.e., dialogue → utterance → token, and (2) multi-facet attributes (See et al., 2019; Shen et al., 2021a), such as speaker-shift, content-relatedness, fact-awareness, and coherence. Therefore, directly applying pre-trained language models (PLMs) to the dialogue understanding tasks is inappropriate.
To better utilize PLMs for dialogue representation and understanding, researchers use data samples from dialogue corpora to conduct a second phase pre-training of PLMs, i.e, dialogue-adaptive post-training (DialPost). At first, the training objectives were just those for general language modeling (Masked Language Modeling and Next Sentence Prediction) (Whang et al., 2020, 2021; Xu et al., 2021). After that, researchers tried to design some novel objectives that fit dialogue characteristics more. For example, Wu et al. (2021) utilized Span Boundary Objective and Perturbation Masking Objective in post-training to capture the dialogue semantics in span and token levels. Liu et al. (2021) and Wu et al. (2020) constructed positive and negative samples for context-response pairs, and continued training PLMs with contrastive learning to better maintain the dialogue coherence.
Existing DialPost methods either focus on tokenlevel or utterance-level semantics, which only consider a limited subset of dialogue attributes, e.g., speaker-shift (Xu and Zhao, 2021), coherence (Li et al., 2020a), and response-similarity (Wu et al., 2020). However, the comprehensive modeling of multi-facet attributes with multi-level training objectives is not well explored. Moreover, previous DialPost methods handle the whole dialogue as a linear sequence of successive tokens and feed it to PLMs that obtain the token representations indiscriminately with flat self-attention mechanisms. Such a way of modeling is sub-optimal to capture the hierarchical semantic relations of dialogues (Zhang and Zhao, 2021).
10134
To tackle the above issues, we propose a posttraining method for dialogues, namely DIALOGPOST, which consists of five Self-Supervised Objectives (SSOs) and a hierarchical model. The former is designed to capture the multi-facet attributes of dialogues, while the latter is used to model the hierarchical relations in dialogues. Specifically, SSOs correspond to two token-level, one utterance-level, and two dialogue-level selfsupervised learning tasks. For the token-level objectives, we use different sampling approaches to mask spans and roles, which capture factawareness and speaker-shift, respectively. For the utterance-level objective, we corrupt a dialogue via two operations on utterances, and then train the model to maintain coherence by either detecting the corrupted utterances or recovering the utterance order. For the dialogue-level objectives, we model the content-relatedness of both utterance-context pairs and context-context pairs by utterance position prediction and dialogue-based contrastive learning. The model is a Hierarchical Segmentwise Self-Attention network (HSSA) that contains inner-segment and inter-segment self-attention layers along with an aggregation and updating module.
To evaluate the proposed method, we conduct experiments in two aspects, i.e., dialogue representation and understanding. We first verify the representation ability of DIALOG-POST with dialoguebased semantic textual similarity (D-STS) and semantic retrieval (SR) tasks on two public datasets, JDDC and ECD. DIALOG-POST outperforms baselines by 1.7% in D-STS task of JDDC. Then, we
annotate a dataset with four dialogue understanding tasks and conduct experiments on them. Experimental results show that our method consistently outperforms baselines and achieves a 87.5% average score (+3.3%) for dialogue understanding.
Our contribution can be summarized as follows: (1) We propose a post-training method (DIALOGPOST) for dialogue representation and understanding, which consists of five multi-level SSOs and a hierarchical model. (2) We conduct extensive experiments to evaluate DIALOG-POST with two public and one newly-labelled dataset. (3) We analyse the effectiveness of each component of DIALOGPOST, and conduct ablation study to demonstrate the necessity of objectives in different levels.

heading: 2 Approach
text: In this section, we introduce the multi-level selfsupervised objectives (SSOs) and HSSA model.

heading: 2.1 Multi-Level SSOs
text: As illustrated in Figure 1, we design five multilevel SSOs to post-train the dialogue encoder, which consist of two token-level SSOs (LDSM and LDRM ), one utterance-level SSO (LDUC), and two dialogue-level SSOs (LDUP and LDCL).
Token-Level SSOs. A good conversation should avoid presenting contradictory contents about facts (Zhang and Zhao, 2021). Therefore, the ability of realizing important words and phrases, denoted as fact-awareness, is a fundamental attribute and helps keep the factual consistency. Here, we design a Dialogue Span Masking (DSM) objective,
LDSM , to capture the fact-awareness. First, we sample 50% utterances from a dialogue. Then, we perform the span masking (Joshi et al., 2020) for each selected utterance, and the model needs to recover those masked spans. By this means, the facts in each utterance and their dependency within or cross utterances can be learned.
Speaker-shift is a distinctive attribute of dialogues (Gu et al., 2020). In a real scenario, two speakers carry out a conversation in an interactive way, and one speaker may continuously shoot multiple utterances (Xu and Zhao, 2021). We propose the Dialogue Role Masking (DRM) objective , LDRM , which aims to predict the masked role tokens. Before that, 80% of role tokens are randomly masked in a dialogue. In Figure 1, the masked tokens and speaker roles are marked with “_”.
Utterance-Level SSO. An utterance is the most basic semantic unit in dialogues (Jiao et al., 2019; Zhu et al., 2020; Li et al., 2020b; Henderson et al., 2020), and utterance corruptions could break the entire coherence. To better maintain the coherence by mimicking possible corruptions, we propose a Dialogue Utterance Corruption (DUC) objective, LDUC . Given a dialogue D containing m utterances, i.e., D = {u1, u2, ..., um}, nc = ⌈0.3 ∗m⌉, we could corrupt a dialogue via 2 operations:
• Replace: We sample nc utterances from other dialogues D′ with each utterance u′j ∈ D ′ ,
j ∈ [1, nc]. Then, we replace nc randomly selected utterances in D with the sampled ones, and assign each utterance a label Y = {y1, y2, ..., ym}, where yt is 0 for the replaced utterance; otherwise 1 for the original utterance. The goal is to predict 0 or 1 for yt.
• Shuffle: We sample nc utterances from D and then shuffle them to change their order. The goal is to predict orders of the nc shuffled utterance, and the size of label set Y equals to nc with each yt ∈ [1, nc].
In practice, we randomly apply one operation, and use different classification heads to predict Y for either “Replace” or “Shuffle”. Two examples are given in Figure 1 for better understanding.
Dialogue-Level SSOs. A conversation usually contains topic changes and redundant messages regarding a utterance or partial context. Therefore, we need to detect relevant information via exploring the relationship of utterances and contexts. Previous works mainly focus on the response
selection task (Liu et al., 2021; Wu et al., 2020) that measures the similarity of each context-response pair. To consider utterances in different positions, not only the last one (i.e., response), we model the content-relatedness of utterance-context pairs, and propose a Dialogue Utterance Position (DUP) objective, LDUP . We first regard an utterance as query, and a list of consecutive utterances as context, then their relationship can be defined as follows: (1) Before: query ub is before the context {uk, uk+1, ..., um}, i.e., 1≤b<k; (2) After: query ua is after the context {u1, u2, ..., uj}, i.e., j<a≤m; (3) Inside: query ui is inside the context {u1, ..., ui−1, ui+1, ..., um}, i.e., 1<i<m; (4) Unrelated: the context is {u1, u2, ..., um}, while query u ′ is sampled from another dialogue. Finally, we feed the context and query into a dialogue encoder under the sequence-pair classification setting (Devlin et al., 2019).
In addition, we extend an utterance to consecutive utterances, and capture the content-relatedness of context-context pairs with a Dialogue Contrastive Learning (DCL) objective, LDCL. Specifically, we randomly sample nc = ⌈0.3 ∗m⌉ consecutive utterances Dp = {u1, u2, ..., unc} from a dialogue D. Then we replace each utterance in Dp with a special token “[UMASK]”, and construct an incomplete dialogue Dr=D/Dp. Given a batch of Dr-Dp pairs, we apply the in-batch contrastive learning loss (Wang and Isola, 2020; Gao et al., 2021) to compute LDCL:
LDCL = − 1
N
N∑
i
log esim(f(Dri ),f(Dpi ))/τ
∑ j ̸=i e sim(f(Dri ),f(Dpj ))/τ .
For a given Dri , we calculate the cosine similarity with the corresponding Dpi against the other partial context Dpj . We use the average output of the encoder f(·) and set temperature τ to 0.1.
Continuous Multi-Task Learning. Inspired by Sun et al. (2020), we apply the popular continuous multi-task learning (CMTL) framework for model training. CMTL can pre-train models with multitask objectives efficiently and prevent knowledge forgetting of previous tasks when training with the current task objective(s). Since our method consists of several tasks, CMTL is extremely proper for our experiments. The final objective is calculated as:
L = LDSM +LDRM +LDUC +LDUP +LDCL.
Table 1 illustrates the details of training process. For each stage (denoted as Si), we train the model
with multiple tasks and each task with used for given steps, i.e., for S2, we train the model using DRM for 5K steps and DSM for 30K steps.
2.2 HSSA Model
As shown in Figure 2, the proposed hierarchical segment-wise self-attention (HSSA) model contains several layers, and each layer is a block consisting of inner-segment self-attention, intersegment self-attention, segment updater, and feedforward sub-layers.
For the l-th layer, we split the dialogue hidden states Dl−1 ∈ Rn×d from the previous layer into nB segments, where n is length of input sequence, and each segment Dsegi contains B hidden states. For each Dsegi , we first apply the self-attention mechanism SA(·) (Vaswani et al., 2017) to obtain an inner-segment representation Hinni = SA(Dsegi) ∈ RB×d. Then, we aggregate Hinni , and compute the attention scores between the aggregated state and each segment state:
Agg(Hinni) = 1∑ eMj
B∑
j=1
Hinni,j ∗ eMj ,
αij = softmax( Agg(Hinni)H T inni,j√
d ), j ∈ [1, B],
where Mj is the attention mask that is −inf for non-attended tokens and 0 for the rest.
To obtain the sub-layer output H̃inn, we use an attention-based pooling method:
H̃inni = Wp( B∑
j=1
Hinni,j ∗ αij)T + bp,
H̃inn = [H̃inn1 , H̃inn2 , ..., H̃innn/B ],
where Wp ∈ Rd×d and bp ∈ Rd are parameters of linear transformation.
We then apply the self-attention mechanism to get the inter-segment hidden states Hint = SA(H̃inn). The inner-segment and inter-segment self-attention share the same set of parameters. Next, we use an updater to update B hidden states in each segment with the corresponding inter-segment representation Hinti ∈ R1×d:
H̃segi,j = βi,j ∗ Hinti + Hinni,j ,
βi,j = softmax( Hinni,jH T inti√
d ), j ∈ [1, B].
The segment representations are concatenated and fed to the feed-forward layer to get the output {Dl1,Dl2, ...,Dln}. We then apply a residual connection to the output and Dl−1 with layer normalization. Note that HSSA model does not include extra parameters, thus we can fully initialize the model with pretrained language models, such as BERT. Moreover, the segment-based attention reduces the computational burden. HSSA can reduce the memory cost from O(n2) to O(nB+( nB )
2+n), which also improves the training and inference efficiency.

heading: 3 Experiments
text: To verify the effectiveness of DIALOG-POST, we conduct extensive experiments on both dialogue representation and understanding tasks. We first introduce the experimental setup, then elaborate implementation details, and finally illustrate the main experimental results.

heading: 3.1 Experimental Setup
text: Post-Training Data. For fair comparison (Xu and Zhao, 2021; Zhang and Zhao, 2021; Xu et al., 2021), we utilize two public dialogue datasets, JDDC (Chen et al., 2020) and ECD (Zhang et al., 2018), to conduct all experiments. JDDC1 is a large-scale multi-turn dialogue corpus released by
1Dataset is available at http://jddc.jd.com.
JD2, which contains more than 1 million real conversations between users and customer service staff in E-commerce scenario. ECD is a large-scale dialogue corpus collected from Taobao3. Finally, 2,044,196 dialogues with 27,951,337 utterances in total are used for post-training. Evaluation Tasks. Two typical groups of evaluation are considered to verify the effectiveness of DIALOG-POST. The first group is evaluation on dialogue representation, which uses utterance embeddings obtained by the dialogue encoder to fulfill two tasks, the semantic retrieval (SR) and the dialogue-based semantic textual similarity (DSTS) (Liu et al., 2021). The SR task is a retrieval task that ranks utterance candidates by calculating the semantic similarity between embeddings of a query utterance and those candidates. The D-STS task aims to classify each utterance pair into five degrees ranging from 1 to 5 according to their semantic relevance. We utilize the public evaluation sets of JDDC and ECD release by Liu et al. (2021).
The second group of evaluation consists of four popular downstream tasks of dialogue understanding, which are Intent Classification (IC), Sentiment Recognition (Senti), Context-Question Matching (CtxQ), and Context-Response Matching (CtxR). CtxQ and CtxR are two critical tasks for retrievalbased dialogue systems, and we formulate them as binary classification problem here. The downstream understanding tasks usually rely on the domain of dialogue corpus. To avoid domain inconsistency, we construct four datasets for the above tasks by re-annotating the data sampled from JDDC. Please refer to Appendix A for more details of
2http://www.jd.com. 3http://www.taobao.com.
the annotation.
Evaluation Metrics. Following Liu et al. (2021), we report the mean average precision (MAP) and mean reciprocal rank (MRR) scores for SR, and the Spearman’s Correlation (denoted as Corr.) score for D-STS. For four understanding tasks, we calculate Macro-F1 (denoted as F1) for IC, Accuracy (denoted as ACC) for Senti, and AUC (Area under the ROC plot) for CtxQ and CtxR. To avoid the impact of randomness in neural networks, we report the evaluation results of 5 runs in the format “avg±std.dev”. Details of each evaluation task are illustrated in Table 3. Baselines. We choose two branches of models as our baselines. The first branch is PLMs posttrained with dialogue data via original objectives, including: (1) BERT (Devlin et al., 2019), which utilizes Masked Language Modeling and Next Sentence Prediction (NSP) objectives for pre-training. (2) ERNIE (Sun et al., 2019, 2020), which leverages external knowledge base to mask entities and phrases. (3) ELECTRA (Clark et al., 2020), which devises the replaced token detection task to pre-
train the language model as a discriminator. The second branch is the dialogue-adaptive posttraining models, including: (4) UMS (Whang et al., 2021), which proposes three utterance manipulation strategies for dialogues to promote response selection and context understanding. (5) PLATO (Bao et al., 2020b, 2021), which utilizes UniLM (Dong et al., 2019) to pre-train dialogue encoder with a discrete latent variable via act recognition and response generation tasks. (6) TOD-BERT (Wu et al., 2020), which combines the contrastive learning loss and MLM to train the dialogue encoder. (7) DialCSE (Liu et al., 2021), which designs the matching-guided embedding and turn aggregation with contrastive learning to obtain the context-aware utterance representation. (8) DialBERT (Zhang et al., 2021), which proposes several dialogue-specific self-supervised tasks to train a dialogue encoder. (9) DomainAP (Wu et al., 2021), which combines the pre-training objectives of SpanBERT (Joshi et al., 2020) and perturbation masking objective to enhance the model performance in downstream dialogue tasks.
All above baselines are back-boned with BERT4
(Devlin et al., 2019). For fair comparison, we posttrain all models with the same training data as mentioned before. For SR and D-STS tasks, we infer the utterance embeddings by feeding utterances into the model without fine-tuning. For IC, Senti, CtxQ and CtxR tasks, we fine-tune all models5 with the corresponding datasets, then conduct the performance evaluation.
4We choose the whole word masking Chinese BERT (Cui et al., 2020) as the base model.
5Our code and dataset can be found from: https:// github.com/zhangzhenyu13/dialogue-post

heading: 3.2 Implementation Details
text: Hyper-parameters of HSSA. Previous research (Zhong et al., 2021) indicates that segment-based attention and full self-attention are complementary on catching the local and global dialogue semantics. Inspired by this, we take the hybrid manner for HSSA implementation, i.e., the first 10 layers are the HSSA blocks with segment size of 8, 16, 32, 32, 64, 64, 64, 128, 128, 128, while the last 2 layers are the original Transformer blocks. We use the self-attention layer weights from a Chinese BERT with whole word masking (Cui et al., 2020) to initialize both the inner-segment and inter-segment self-attention sub-layers in HSSA. The input embedding layer is the same as that of BERT. Therefore, HSSA has no extra parameters compared to BERT. Moreover, we also post-train a BERT model (denoted as DIALOG-POST-BERT) with the multilevel SSOs. Unless otherwise specified, the model base of DIALOG-POST in our work is HSSA.

heading: 3.3 Experimental Results
text: Evaluation on Dialogue Representation. As a novel method for dialogue representation, we first verify the performance of our model on SR and D-STS tasks. Previous research (Liu et al., 2021) shows that using the average of all token embeddings is better than using the “[CLS]” token embedding for utterance representation, thus we utilize the average token embedding in our experiments. The results in Table 2 shows that: (1) All models post-trained with dialogue-adaptive methods surpass the general-purpose PLMs by a large margin, which indicates the advantages of various self-supervised training objectives to catch the dialogue characteristics during representation learn-
ing. (2) Among the baselines, DialCSE (Liu et al., 2021) has the best performance, we argue that the advantage mainly comes from the context-aware response-based contrastive learning, which benefits the semantic matching tasks naturally by eliminating the gap between training and evaluation. (3) Our proposed method DIALOG-POST beats all baselines on both datasets, demonstrating the superiority of multi-level SSOs during post-training, which can generate better representations for dialogue utterances by catching the multi-facet attributes of dialogues. For SR on ECD, the performance of DIALOG-POST-BERT is slightly better than DIALOG-POST. We conjecture that it is because the ECD corpus has shorter dialogue contexts, which may limit the ability of HSSA. Evaluation on Dialogue Understanding. We evaluate our method on four popular downstream tasks for dialogue understanding, including IC, Senti, CtxQ, and CtxR. Unlike the evaluation on dialogue representation, we fine-tune the models with task-specific datasets. The results in Table 4 show that: (1) Compared to all baselines, DIALOG-POST yields substantial improvements across four understanding tasks, achieving 3.9%, 3.5%, 1.7%, and 2.3% absolute improvements against previous SOTA approaches on IC, Senti, CtxQ, and CtxR tasks, respectively. (2) DIALOG-POST also leads to further improvement (+0.3% average) compared with DIALOG-POST-BERT, revealing the capacity of HSSA on grasping the structure of dialogues. We argue that, understanding dialogues (e.g., the intents and emotions) relies on deep semantic meanings from the hierarchical dialogue structure, which requires the model to catch the multi-granularity semantic relations from the tokens, utterances, and the whole dialogue. By harnessing the multi-level SSOs and HSSA model, our method can better understand the intrinsic dialogue structure, and finally boost performance on downstream tasks.

heading: 4 Discussion
text: In this section, we conduct further in-depth discussions to analyse the HSSA model, the contribution of each SSO, and visualize the training process of CMTL. Due to space limitation, we report the results of JDDC/SR, JDDC/D-STS, IC and Senti.

heading: 4.1 Ablation Study of HSSA
text: As mentioned in Section 2.2, we stack 10 layers of HSSA blocks and 2 layers of Transformer blocks
in our model. The last 2 Transformer layers are devised to capture the full dialogue semantics based on the global self-attention (SA) mechanism. Here, we first replace the last 2 Transformer layers with 2 HSSA layers (denoted as “w/o trs”). Table 5 shows that the performance degrades significantly on D-STS, SR, and IC, indicating the necessity of global self-attention. It is notable that the performance of Senti becomes slightly better with all HSSA blocks. Since the input of Senti task is an utterance without context, it is possible that the 12-layer HSSA focusing on the local attention has some advantages. Moreover, we also try to remove the updater (“w/o updater”), the inter-segment self attention (“w/o Hint”), or the inner-segment self attention (“w/o Hinn”) sub-layer from HSSA. The results in Table 5 demonstrate that all variants lead to a pronounced performance degradation, which proves the rationality of each sub-layer.

heading: 4.2 Ablation Study of SSOs
text: Here, we conduct the ablation study for five SSOs. We follow the same training order (DRM → DSM → DUC → DUP → DCL) as CMTL mentioned in Table 1, but remove one training objective each time while keeping the remaining four. Table 6 shows that each training objective contributes to the overall performance to some extent, indicating the multi-level SSOs are complementary. Besides, DCL brings the most benefits, which implies the effectiveness of DCL on capturing the contentrelatedness of context-context pairs.

heading: 4.3 Visualization of CMTL Training
text: Figure 3 illustrates the curves of training loss for each task across different training steps. The lines
with the same color represent the training loss of with the same objective. Here, we compare the training loss of CMTL and single task training. For example, for the red lines, the solid one is much lower than that of the dashed one, which indicates that the training process converges much faster by applying CMTL. It also shows the former training tasks may facilitate the latter, and finally promote the stability of the whole model training.

heading: 5 Related Work
text: Dialogue Encoding Networks. To handle the particularity of dialogue structure, previous works have proposed several typical networks for dialogue encoding, including hierarchical attentionbased models (Jiao et al., 2019; Zhu et al., 2020; Li et al., 2020b), recurrence-based models (Shen et al., 2021b; Yang et al., 2019), and long conversation oriented models (Zhong et al., 2021). For example, Jiao et al. (2019) and Zhu et al. (2020) encode each utterance at first, and then leverage LSTMs and Transformers to aggregate the utterances, while Shen et al. (2021b) use memory caches to encode utterances sequentially. Huang et al. (2021) integrate sparse attention to encode long dialogue sequences (e.g., with 5000 words).
In this work, we propose a novel dialogue encoding network HSSA to capture the semantic structure of dialogues. It takes the dialogue as input and leverages inner-segment and inter-segment selfattention to capture the hierarchical dependencies. Finally, we devise an updater to obtain the contextual encoding of dialogues by aggregating the inner- and inter-segment representations.
Dialogue Post-training. With the booming of PLMs (Devlin et al., 2019; Radford et al., 2019; Bao et al., 2020a), researchers try to apply PLMs in the field of dialogues. An intuitive idea is to conduct a second-stage pre-training with massive dialogue corpora, but without changing the training objectives (Zhang et al., 2020; Xu et al., 2021). Recently, some works (Jiao et al., 2019; Feng et al., 2020; Zhang et al., 2021; Xu and Zhao, 2021) are proposed to design several new objectives for dialogue-adaptive post-training and achieve astonishing performance on downstream tasks of dialogue understanding. PLATOs (Bao et al., 2020b, 2021) leverage the large-size unified language model (Dong et al., 2019) to fulfill context encoding and response generation tasks with curriculum learning. Response selection is widely used as a self-supervised post-training task due to the convenience of constructing training data (Mehri et al., 2019; Su et al., 2021; Liu et al., 2021; Whang et al., 2021). Wu et al. (2021) propose Span Boundary Objective and Perturbation Masking Objective to capture the dialogue semantics in span and token levels. Above works either focus on token-level or utterance-level semantics, and only consider a small set of dialogue attributes.
Differently, we propose five self-supervised objectives in token, utterance and dialogue levels, aiming to modeling multi-facet dialogue attributes, including fact-awareness, speaker-shift, coherence, and content-relatedness of both utterance-response pairs and context-context pairs.

heading: 6 Conclusion and Future Work
text: In this paper, we propose a novel dialogue-adaptive post-training method, DIALOG-POST, by devising five multi-level training objectives and a hierarchical dialogue encoder network. These training objectives capture the multi-facet attributes of dialogues by leveraging token-level, utterance-level, and dialogue-level self-supervised signals. The dialogue encoder learns the hierarchical semantic structure of dialogues. To validate the effectiveness of our method, extensive experiments on dialogue representation and understanding tasks are conducted. Experimental results demonstrate the competitiveness of our method against strong baselines of both tasks. In the future, we will explore more efficient model architectures and try to pretrain the dialogue-oriented PLMs from scratch.
Limitations
Although the proposed method achieves exciting results, there are still some issues that need to be addressed in the future: (1) When designing the structure of HSSA layers, we assume that humans tend to understand a dialogue from the local to global perspective, which supports the existence of inner- and inter-segment self-attention layers. (2) We use 2 public Chinese corpora, JDDC and EDC, for post-training. Though there are diverse topics in them, it is desired to introduce other corpora from different domains and languages. (3) SSL tasks are arranged in post-training via CMTL (Sun et al., 2020) based on the intuitive understanding of their semantic levels and difficulties. Therefore, to combine the power of each SSL task more effectively, new training strategies need to be explored.

heading: A Appendix
text: A.1 Details of Dialogue Understanding Tasks In this section, we introduce the details of dataset annotation and show some examples from the dialogue understanding task. The original JDDC (Chen et al., 2020) corpus provides intent labels for each utterance, and three challenging sets of response generation. Considering intent classification, sentiment recognition, context-query matching, and context-response matching are very common tasks of dialogue applications in industry, we construct an evaluation dataset for dialogue understanding, which consists of 4 downstream tasks.
We sample 5,000 dialogues from JDDC and invite 4 graduate students to finish the annotation. For each data sample, at least three people finish the annotation and the majority voting is applied to decide the final label. The annotation agreement (Fleiss’ Kappa (Fleiss and Cohen, 1973) score) is 0.83, showing the good quality of the annotation. The evaluation sets are derived from JDDC corpus, and we hope they can facilitate the dialogue understanding for future research.
We list the description of each task below, and show some examples in Table 8. Note that for a dialogue D = {u1, s1, u2, s2, ..., un−1, sn−1, un, sn}, un and sn are the current user query and staff response, and the previous utterances are denoted as context. u and s represent the user and service staff.
• Intent Classification (IC) aims to predict the intent of user query based on the dialogue context. Since the JDDC corpus is in Ecommerce scenario, the intents are related to E-commerce activities and actions, such as “Warranty and return policy”, “Delivery duration”, “Change order information”, and “Check order status”. Understanding user intents is the foundation of industrial dialogue systems. Since context plays a critical role in intent classification, we combine the current user query and last two user utterances before it as an unit, and annotate the intent label for the current user query.
• Sentiment Recognition (Senti) aims to detect the emotions from user utterances. The categories include “happy”, “sad”, “angry”, “feared”, “disappointed”, “anxious”, and “other”. For this task, each user utterance is considered individually for annotation.
• Context-Question Matching (CtxQ) aims to determine whether the semantic meanings are similar given a context-question pair. CtxQ is widely used to find a question in “frequent asked question (FAQ)”, which is highlyrelevant to a context, and return its answer as the response to context. Before that, the standard question-answer (QA) pairs are stored in the database.
• Context-Response Matching (CtxR) aims to determine whether an utterance can be the appropriate response to a given context. The task is also denoted as response selection if multiple response candidates were given.
The classes in training set are uniformly distributed with each class holds nearly same amount of examples. For test sets, the largest class holds 90 examples and the rest classes each hold about 30 examples. For test set of Senti, each class holds roughly 40 to 50 examples. The |positive| : |negative| (examples) are 317:276 and 301:319 for CtxR and CtxQ respectively.
A.2 Training Efficiency and Memory Cost Table 7 illustrates the comparison of BERT and HSSA on memory cost and training speed, and HSSA is more computationally efficient, especially for long dialogues. We post-train the models on Tesla P40 GPUs with batch size of 16.
A.3 Complete Ablation Study of HSSA In Section 4.1, we only show the experimental results on 4 tasks due to space limitation. Here, we supplement the complete experimental results on all test sets in Table 9 and 10 to demonstrate the contribution of each module in HSSA.
A.4 Complete Ablation Study of SSOs We illustrate the complete experimental results for the ablation study of SSOs mentioned in Section 4.2. Table 11 and 12 show the results on dialogue representation and understanding tasks.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:

heading: 3 A1. Did you describe the limitations of your work?
text: Section 7 Limitations.
A2. Did you discuss any potential risks of your work? Not applicable. Left blank.

heading: 3 A3. Do the abstract and introduction summarize the paper’s main claims?
text: Section Abstract and Introduction.
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? Section 3.1

heading: 3 B1. Did you cite the creators of artifacts you used?
text: Section 3.1

heading: 3 B2. Did you discuss the license or terms for use and / or distribution of any artifacts?
text: Section 6, those are open and free artifacts for non-commercial use.

heading: 3 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided
text: that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Section 6, artifacts are open and free to use.
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Not applicable. Left blank.

heading: 3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and
text: linguistic phenomena, demographic groups represented, etc.? We attach the artifacts in supplementary materials where there is a README file for description and usages. Besides, the appendix also illustrates some examples.
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Appendix A.1 and Section 3.1
C 3 Did you run computational experiments?

heading: Appendix A.2

heading: 3 C1. Did you report the number of parameters in the models used, the total computational budget
text: (e.g., GPU hours), and computing infrastructure used? Appendix A.2
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.

heading: 3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found
text: hyperparameter values? Section 4.1

heading: 3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary
text: statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Table 4.
C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Not applicable. Left blank.
D 3 Did you use human annotators (e.g., crowdworkers) or research with human participants?

heading: Appendix A.1

heading: 3 D1. Did you report the full text of instructions given to participants, including e.g., screenshots,
text: disclaimers of any risks to participants or annotators, etc.? Appendix A.1

heading: 3 D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students)
text: and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? Appendix A.1

heading: 3 D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to
text: crowdworkers explain how the data would be used? Appendix A.1

heading: 3 D4. Was the data collection protocol approved (or determined exempt) by an ethics review board?
text: It’s collected from the public dataset.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? Not applicable. Left blank.

heading: Future Work
text: In the future, we will explore more efficient model architectures and try to pretrain the dialogue-oriented PLMs from scratch. Limitations
Although the proposed method achieves exciting results, there are still some issues that need to be addressed in the future: (1) When designing the structure of HSSA layers, we assume that humans tend to understand a dialogue from the local to global perspective, which supports the existence of inner- and inter-segment self-attention layers. (2) We use 2 public Chinese corpora, JDDC and EDC, for post-training. Though there are diverse topics in them, it is desired to introduce other corpora from different domains and languages. (3) SSL tasks are arranged in post-training via CMTL (Sun et al. , 2020) based on the intuitive understanding of their semantic levels and difficulties. Therefore, to combine the power of each SSL task more effectively, new training strategies need to be explored