SlowGuess/NAACL_2022_OCR huggingface dataset:
- MTRANS. Replacing the [Add] operation in the first residual connection layer with [Concat].
- LATER-MTRANS. Replacing the [Add] operation in the second residual connection layer with [Concat].
- BOTH-MTRANS. Replacing the [Add] operations in both the first and second residual connection layers with [Concat].
Our intuition of substituting the summation with concatenation is that the residual layer with concatenation would introduce additional parameters, and we expect these additional parameters to retain more history information. As shown in Table 2, we obtain some gains while using MTRANS. Additionally, we also report the  $\mathrm{F1}_{all}$  scores on DuConv/Persona-Chat/CMU-DoG datasets while using LATER-MTRANS and BOTH-MTRANS here. LATER-MTRANS achieves 88.18/65.32/58.44 points, and BOTH-MTRANS achieves 88.40/66.12/59.72 points against the standard Transformer achieving 88.25/65.27/58.58 points. Although BOTH-MTRANS achieves the
best performance, we finally choose MTRANS since BOTH-MTRANS brings a large volume of additional parameters which leads to a huge model size while the increasing of model parameters caused by MTRANS is acceptable.
We implement the model in PyTorch(Paszke et al., 2019), and use the pre-trained language model of multilingual BERT (mBERT) or XLM-RoBERTa (XLM-R) made available by the Transformer library (Wolf et al., 2020) as the backbone. We train the model using AdamW(Loshchilov and Hutter, 2018) with a linear learning rate schedule. For each model, we run five different random seeds and report the average score. More details and hyper-parameters are listed in Table 7.
We compare to following baseline models,
1. SimpleBERT/SimpleXLMR (Shi and Lin, 2019). It uses the Chinese BERT or XLM-R as the backbone and simply concatenates the entire dialogue context with the predicate.
2. CSRL-BERT/XLMR (Xu et al., 2021). It uses the Chinese BERT or XLM-R as the backbone but attempts to encode the conversation structural information by integrating the dialogue turn and speaker embeddings in the input embedding layer.
3. CSAGN/CSAGN-XLMR (Wu et al., 2021b). It uses the Chinese BERT or XLM-R as the backbone and employ the relational graph neural network to model predicate- and speaker-aware dependencies. We implement this baseline based on the code https://github.com/hahahawu/CSAGN.
E Application Models
Rewriting Model. We adopt the model proposed in (Xu et al., 2020) which directly concatenates the predicate-argument structures, the conversation context and the question as a sequence, and then feeds them into the model with special attention masks. During decoding, the model takes CSRL pairs and the context to generate the rewritten question word by word. The input representation, attention strategies and loss function of our model are same as (Xu et al., 2020)'s. We initialize the model
using the base BERT model and use AdamW with a linear learning rate schedule to update parameters.
Note that we only attempt to introduce the CSRL information as a condition into our generation-based model. We did not include the CSRL information into the state-of-the-art rewriting models, i.e., SARG and RUN because these models rewrite the sentence by learning a text editing matrix instead of directly learning the distributions of the target words. Unfortunately, there are no straightforward ways to include our CSRL information into these models to help the matrix learning.
Response Generation Model. Our model for response generation is directly borrowed from UniLM (Dong et al., 2019) or mBART (Liu et al., 2020b). For UniLM, the generation process is same with the rewriting task, wherein the extracted semantic pairs, the context and the response are concatenated into a sequence and encoded with the special mask. For mBART, we just concatenate the extracted predicate-argument pairs with the context into a sequence, and then feed the sequence into the encoder for training; during decoding, our model takes semantic information and the context as input to generate the response word by word. The input representation, attention strategies for CSRL structures and loss function are same as the rewritten model's. We initialize the model using the base multilingual BERT or mBART and use AdamW with a linear learning rate schedule to update parameters.
We list the hyper-parameters of CSRL experiments (Table 7), rewriting experiments (Table 8) and response experiments (Table 9) below.
Name	Value
Language model	xlm-roberta-base
Hidden state size	512
Word-level encoder layers	2
Pred.-arg encoder layers	1
Batch size per GPU	24
Max learning rate	5e-5
Min learning rate	1e-5
Max lr for LM fine-tuning	1e-5
Min lr for Lm fine-tuning	1e-6
Max sequence length	512
Max training epochs	50
Max training steps	15000
Early-stop patience	10
Table 7: Hyper-parameters in CSRL experiments.
Name	Value
Language model	bert-base-cased
Hidden state size	768
Batch size per GPU	16
Max learning rate	3e-5
Min learning rate	1e-5
Max sequence length	512
Max decode length	32
Max training epochs	20
Early-stop patience	5
Table 8: Hyper-parameters in rewriting experiments.
Name	Value
Language model	mBERT and mBART
Hidden state size	768
Batch size per GPU	16
Max learning rate	5e-5
Min learning rate	3e-5
Max sequence length	512
Max decode length	64
Max training epochs	20
Early-stop patience	5
Table 9: Hyper-parameters in response generation experiments.