GuoFeng: A Discourse-aware Evaluation Benchmark for Language Understanding, Translation and Generation
Authors: Longyue Wang
Zefeng Du
DongHuai Liu
Deng Cai
Dian Yu
Haiyun Jiang
Yan Wang
Shuming Shi
Zhaopeng Tu
Venue: Submitted to ICLR 2023 (2023)
Keywords: Discourse
Evaluation Benchmark
Pre-trained Models
Natural Language Processing

Abstract: Modeling discourse -- the linguistic phenomena that go beyond individual sentences, is a fundamental and challenging problem in natural language processing (NLP). However, existing evaluation benchmarks mainly focus on the evaluation of inter-sentence properties and overlook important discourse phenomena that cross sentences. To bridge the gap, we propose a GuoFeng benchmark that can evaluate intra-sentence discourse properties across a diverse set of NLP tasks, covering understanding, translation, and generation. GuoFeng consists of 9 document-level testsets in the literature domain, which contain rich discourse phenomena (e.g. cohesion and coherence) in Chinese and/or English. For linguistic analysis, we also propose a diagnostic test suite that can examine whether the target models learn discourse knowledge. We evaluate 17 general- and in-domain models based on Transformer and advanced pre-training architectures, showing that fine-grained pretraining based on document-level training data consistently improves the modeling of discourse information. We will release the datasets, pretrained models, and leaderboard, which we hope can significantly facilitate research in this field.

Under review as a conference paper at ICLR 2023
GuoFeng: A DISCOURSE-AWARE EVALUATION BENCH- MARK FOR LANGUAGE UNDERSTANDING, TRANSLA- TION AND GENERATION
Anonymous authors Paper under double-bline review
ABSTRACT
Modeling discourse the linguistic phenomena that go beyond individual sen- tences, is fundamental and challenging problem natural language processing (NLP). However, existing evaluation benchmarks mainly focus on the evaluation of inter-sentence properties and overlook important discourse phenomena that cross sentences. To bridge the gap, we propose GuoFeng benchmark that can evaluate intra-sentence discourse properties across a diverse set NLP tasks, covering understanding, translation, and generation. GuoFeng consists of document-level testsets in the literature domain, which contain rich discourse phenomena (c.g. cohesion and coherence) in Chinese and/or English. For linguistic analysis, we also propose a diagnostic test suite that can examine whether the target models learn discourse knowledge. We evaluate 17 general- and in-domain models based on Transformer and advanced pretraining architectures, showing that fnn-ggained pretraining based on document-level training data consistently improves the model- ing of discourse information. We will release the datasets, pretrained models, and leaderboard, which hope can significantly facilitate research in this field.
INTRODUCTION
To evaluate the general performance of models, previous work proposed a variety of benchmarks, covering different tasks and languages such GLUE (Wang et al.) 2018), CLUE (Xu et al. 2020) and XGLUE (Liang et al.) 2020). How- ever, existing benchmarks pay little attention to discourse phenomena, which and challenging problem in natural language pro- cessing (NLP) (evitt et al.) 1992). The natural language generally consists of meaningful, uni- fied, and purposive groups of sentences, which organized as whole according discourse properties (Cook) 1989). As shown in Figure the discourse property manifests two ways: (1)
Audi an automaker that makes luxury cars.
was established by August Horch 1810.
produces autstanding quality.
fundamental
Conssion
Colerence
Legend
ophorr
Coneeennce
Smuchir
Rearion
Repertion
Figure Discourse definition and example. The concept of discourse is detailed in Appendix A
cohesion, where the dependency between words or phrases makes them logically and consistently connected; (2) coherence, where the structural relation between segments sentences enables them semantically and meaningfully composed.
To bridge the gap, we introduce a GuoFeng benchmark for the target evaluation on the discourse modeling. GuoFeng comprises three parts:
GuoFeng Benchmark: consists of nine Chinese/English discourse-awee aware tasks covering broad range of NLP tasks (understanding, translation, and generation), data quantities (from 26.4K to 2.4M), and difficulties. Besides, most benchmarking datasets are newly created in this work.
GuoFeng Diagnostic Dataset:
To understand the discourse information learned by models, GuoFeng also includes a dataset hand-crafted 600 examples for probing trained models. Each instance in the dataset is contrastive pair, where the correct candidate is the original instance in

---

Under review as a conference paper at ICLR 2023
Dataset
Task
Description
Metric
Language
Train
Test
Domain
Understanding
Task
SI
Speaker Identification
F1
48.0K
17.5K
novel mixed composition
zh
ZPR
Zero Pronoun Recovery
F1
2.2M
8.1K
zh
MRC
Machine Reading Comprehension
Accuracy
26.4K
6,5K
mzh, czh
Translation Task
NT
Novel Translation
BLEU
1.9M
1.3K
novel
zh-ten
CCT
Classical Chinese Translation
BLEU
778.1K
5.3K
dianji poetry
czh >mzh
PT
Poetry Translation
BLBU
47.1K
2.7K
zh zh-ten
Generation Task
TE
Text Expansion
BLEU
2.4M
10K
book
en
TI
Text Infilling
PPL
233K
10K
book
zh
Text Completion
PPI.
233K
10K
book
Table I: An overview of our discourse- aware evaluation benchmark, covering language understanding, translation and generation. All datasets consist of document-level texts in the literature domain, which are rich in discourse phenomena. Eight of them are newly created by us and one is expanded based on existing corpus (i.e. MRC). covers three languages: English (en), Modern Chinese (mzh/zh) and Classical Chinese (czh). We only report commonly-usee evaluation metrics here while more metrics are discussed in Appendix |B "#" means the number of instances (e.g. sentences, pairs or documents). "Test" represents both validation and testing sets.
the benchmark and the incorrect one is perturbation by modifying discourse devises or structures in the correct candidates.
GuoFeng Training Data: We release large-scale, document-level data (400G) in Chinese and English, which the same literature domain with the benchmark. The training data enables fine-grained pretraining to better model discourse information required by the benchmark.
To better understand challenges posed by GuoFeng, we conduct experiments variety of state- of-the-art models, including Transformer and pretrained models. We found that these tasks display different levels of difficulty, resulting in different behaviors and performances across models. Fur- thermore, the fine- fine-grati gree-graded pretraining based on the document-level and discourse-ric GuoFeng data improves performances particularly cohesive translation and coherent generation. However, the best models still achieve a fairly low absolute score, highlighting the difficulty modeling discourse.
There three main contributions in this work:
We propose diverse set of sescourse--wart tasks to evaluate monolingual and cross-lingual models' ability to understand and generate nature language.
Challenging Tasks:
Considerable Resources: We build and release variety of discourse-and aware resources, including benchmarking datasets, diagnostic test suite, large-seale pretraining corpus and discourse-aware pretrained models.
Comprehensive Comparisons: We systematically compare many advanced pretraining methods on the benchmark, and identify current challenges in discourse modelling for future exploration.
DISCOURSE-AWARE TASKS
To comprehensively evaluate the target models, GuoFeng covers three types of NLP tasks, including language understanding, translation and generation. We design the benchmarking tasks using the following criteria: (1) our tasks should measure the ability of models handle discourse phenomena, thus we define discourse-relan related tasks at different levels of difficulty; (2) our datasets should contain rich discourse phenomena, thus build document-level datasets with whole contexts extracted from literary texts. To this end, we introduce nine discourse-awaad tasks, which are representative of challenging NL.P tasks, and easily applicable real-world situations.
2

---

Under review as a conference paper at ICLR 2023
Inp:
$88
Out:
Speaker= B1/
40 20
Logend
Inp:
B: #
ZPR
B: 7T
Comeacie West
Out:
B: T
Zera ProeLt
A: #880
7
Inp:
MRC Aakoo
1 7 ti
M. 78 PA
Out:
Answer=(B) 2
Discourse Context and Features
Task Description
Figure 2: Illustration of the proposed understating tasks in terms discourse properties and task definition. As seen, SI needs recognize named entity and resolve coreference. While ZPR demands the further ability to tackle zero anaphora and gender identification. MRC is the hardest because it should fully understand coherence (e-g. discourse structure based on temporal relation) apart from cohesion in previous tasks. English translations of example sentences are listed in Table 6
Accordingly, the benchmark contains collection of nine datasets in Chinese and/or English: eight of which are newly created, and one is expanded based on existing data. Table lists the details of the benchmark, where each task contains training, validation, and testing sets. the following subsections, we mainly introduce task definition, discourse awareness, and data construction. More details such as additional evaluation results are described in Appendix BB]
2.1 LANGUAGE UNDERSTANDING TASKS
NLU aims to analyze what human language means, containing various tasks such natural language inference and story comprehension. Discourse one of the fundamental problems for NLU models. difficult determine the referents of pronouns and definite noun phrases, and understand elliptical sentence fragments, as well as a host of other long-range language phenomena that have not even been adequately characterized much less conquered (Bates) 1995). As shown in Figure 2 we classify tasks into three difficulty levels according to the length of contexts and the amount of knowledge, required for discourse modeling.
SI (Speaker Identification) Given paragraph that may contain an utterance and the surrounding context, SI aims to identify the corresponding speaker(s) for the utterance or the content within quotation marks if speaker exists. To archive this goal, models need to examine the existence of quotes, recognize named entities or phrases that can serve as speakers, and resolve coreference. We construct the dataset that contains 66K instances based eighteen Chinese novels. Unlike previous SI datasets such as P&P (He et al.) 2013) in which all speakers are entities, speakers in our dataset also be phrases, pronouns, multi-entities.
ZPR (Zero Pronoun Recovery) ZPR aims to recover omitted pronouns in terms of position and form, according to its anaphora information the given sentence (Yang Xue 2010 Zhang et al. 2019b] [Song et al.) 2020). The BaiduKnows is widely-used Chinese ZPR corpus, which contains only 5K human-annotate sentences extracted from a Q&A forum (Zhang et al. 2019b). The insufficient data limits the investigation of model performance on ZPR. Inspired by Wang et al. (2016). we automatically built a large-scale training set from Chinese- Chinese-English movie subtitles using word alignments. For clean test set, hire experts to manually annotate 8K sentences covering five domains (i.e. 1.7K novel, 2.2K movie subtitle, 1.2K Q&A forum, 1.6K news, and 1.5K resume). The label set contains 30 Chinese pronouns according to person, number, and gender.
3

---

Under review as a conference paper at ICLR 2023
Inp:
O T 71
His mind had already flown to faraway
Out:
EARK - A
Inp:
60 OREE
fitt?
CCT
18 M002E1, 18:0
CRRIAA 03
Out:
60 O
cote Dobasioe
6% FO
Inp:
6 F
0
PT Verkrs
OR
your
Out:
T
an apprentice pine
Discourse Context and Features
Task Description
Figure 3: The illustration of the proposed translation tasks in terms of discourse properties and task definition. As seen, a variety of elements may be omitted in the Chinese input but should be recalled in English translation. NT mainly deals with zero pronouns while CCT needs further tackle omitted connective words that are the marker of discourse structure. PT is the most difficult task because even prepositions could further omitted. English translations of example sentences are listed in Table 6
MRC (Machine Reading Comprehension) The goal of MRC is to answer questions based on the understanding of its meaning given an unstructured text (Liu et al. 2019a Zeng al., 2020). We collected the Haihua2021 corpus, which contains 8K articles extracted from reading comprehension tests in primary/high school examinations/ Each article followed by at least one question with 2~5 choices and one correct answer. We manually create 2K articles as additional supplement. Different from previous benchmarks based on Wikipedia texts (Cui et al. 2019) or Chinese idioms (Zheng et al., 2019), the Haihua2021 corpus is in the literary domain (L. modern/ancient composition and poetry) that contains rich discourse phenomena. Different from the C benchmark (Sun et al. 2020) where problems collected from Chinese-as-a-second.
examinations, this dataset is extracted from more challenging examinations designed for native speakers.
2.2 LANGUAGE TRANSLATION TASKS
Language translation is sequencc-to--equeene generation task to translate text from one language another. Discourse information important for document-level translation produce cohesive and coherent translations (Wang et al., 2017 Bawden et al. 2018). As shown in Figure 3 we design three translation tasks of increasing hardness, which differ the conciseness source sentences in Chinese. The more concise the Chinese text, the more discourse information is needed for translation.
NT (Novel Translation) The significant challenges for translating novels are entity consistency, anaphora resolution, and lexical choice (Matusov) 2019). We build a document-level Chinese-English corpus, which is extracted from web fictions (150 books in 14 genres). We manually align them at both document and sentence levels. Different from previous document-level MT datasets such as LDC3 and OpenSubtitlef from the news and movie subtitle domains, ours is the first fiterature-domalt MT corpus containing richer linguistic phenomena especially discourse.
htttp::/www.bieett http://wwww..aaaaa.../////etttt/
https://www. de. wwwp::/w.....ue.eeecm edu
httpss://oqu..nll..e//
4

---

Under review as a conference paper at ICLR 2023
To est healthilly I avoid eating food.
Inp:
To eat healthily, avoid eating food. I also vegetables fresh fruit.
est
In order to healthily usually avoid axting food high In fat, like French fries. also eat litll meat. cat a lot of vegetables fresh fruit which full vitamins.
blok
Out:
est vegetables and fresh fruit.
Legand
75
ddV EE 18 70:2 1E
[4 Dd
530. 12
Comectee Word
77
38113029
@i Zera Pronoct
On Centace
T #
Out:
7.
G Onetted
ABnE cal Cobaric
K- Pacchaddr
5 (. W7
12/0////1 7 stor
Inp:
bd
16 00 WEE
16
0 7
BO8 BRG 7527 FRR.
Out:
1B/7 ME,
27 1 T.R.
Discourse Context and Features
Task Description
Figure 4: The illustration of the proposed generation tasks in terms of discourse properties and task definition. As seen, discourse structure and main contents have been specified in TE, thus the task needs to generate cohesive words. While TI should further consider cohesion relations when generating a whole sentence based on the previous and following ones. TC the most difficult because it needs to generate more sentences with unified structure. English translations of example sentences are listed in Table 6
CCT (Classical Chinese Translation) Compared with modern Chinese as in novel translation, classical Chinese texts contain more implicit connectives and pronouns, which require discourse information for information recovery. We construct document-level Mlassicllmmoden Chinese dataset, extracted from Chinese classics across history branch Different from the Niu NiuTrans Classical- Modern corpus that has no discourse context, ours maintain the original context.
PT (Poetry Translation) Poetry translation is regarded as one the hardest tasks in computational linguistics, even artificial intelligence in general (Genzel al. 2010 Ghazvininejad et al. 2018). Chinese poetry is more concise with implicit coherence, which is generally reflected through situa- tional context and contextual context. For example, Chinese poetry does not any cohesive means, but the semantic is still clear. We build document-level Chinese Poetry-English corpus, covering different types of Chinese poetry (e.g. Shi, Ci, Qu, and Fu) translated by famous translators.
2.3 LANGUAGE GENERATION TASKS
Language generation a sequence generation task to produce text based on a given context (Reiter & Dale 1997). Generating long and coherent text is an important but challenging task, particularly on lexical cohesion (Wanner, 1996. Guan al., 202T). As shown in Figure 4 we design three representative generation tasks that differ in degrees of freedom. The more open-ended the generation task, the more difficult to generate accurate cohesive devices and discourse structure.
TE (Text Expansion) We also define a new task, which has been seldom studied previously: given a predefined text, the goal of TE is to insert appropriate words, phrases, or clauses for adding more details and deepening the meaning, while retaining coherence and cohesiveness. We use semi-automatic generation method to obtain large-scale training data. The raw data extracted from English books detailed in Table 14 Specifically, we use the Stanford Parse to produce the syntactic tree of a text, and then manually design some rules to delete the modifier words and phrases in the text. We use the remaining words as the input and predict the dropped modifier. Since some
https///en.wiiieedia..oooo
nese classics
http:///tthhb.oo///uuraaaaCCCC wal.caassscco cal -Moero
hutpaa//ggithbb.cmm//tttttttt/
5

---

Under review as a conference paper at ICLR 2023
Type
Input/Souree/Previous
Hypothesis/Targget/eea
Understanding Task
Correct: AR
Cohesion
(Phoebe really wontr to buy 7V.
(But Joey won't 't her buy it.)
Incorrect: Li 2:00 3 fl
(Jim really wants to buy 7V.)
Correct: 2.
Coherence
(The little princess escaped from the eastle. In the
(Where did the little princess go after she excaped?)
the forest.)
Incorrect: 1/ M MA W
(B) #6
(The little princess escaped from the castle. However, she did not hide the forest.)
(Forest)
Translation Task
Context: RE
Correct:
He thinks Qingshuang funny.
Cohesion
(King Ding looked Qingshuang with smile.)
Current: 14
Incorrect: think the clear cream funny.
Context: 1 1f
Correct: Thus, she can only write to this level.
Coherence
(Lisa did go wchool)
Current: F T9 M
Incorrect: She can only write to level.
Generution Task
Correct: d (1
Cohesion
(Mu Xiaaxiao was reprimanded for no reason.)
(She was sullen on the table.)
Incorrect: (B5 #: :.---
(She was overjoyed lying on tuble.)
TEXM X
Correct: T2
Coherence
(It rained heavily, and Chen Xu was hungry and cold.)
(So he quickly cooked a pot of noodles.)
Incorrect: Mil HiEE 3. ---00 (However, he quickly cooked pot noodles.)
Table The illustration the proposed test suite. We design each contrastive instance with correct and incorrect discourse markers in terms of cohesion and coherence. Tested systems are asked to rank candidates according to their model scores.
delete operations may produce ill-formed text, we filter out the training instances the remaining text has large perplexity measured by language model.
TI (Text Infilling) The TI task aims to predict a text snippet given its surrounding context (Zhu al. 2019). To evaluate the discourse-level model capability,
focus the sentence infilling task that predicts a missing bridge sentence 4o given two preceding sentences (x-2 and x- and two subsequent sentences (r) and v2) (Huang al. 2020 Cai et al. 2020). We build new TI dataset by extracting consecutive 5-sentence paragraphs from Chinese web fictions used in the NT task.
TC (Text Completion) The TC task to predict a writing continuation given a preceding prompt. We focus on multi- sentence paragraph completion for target evaluation of discourse modeling, which completes a multi-sentence paragraph Esee given its leading sentence V.. use the same data collected for the TI task to construct the TC dataset. Specifically, given a sentence x we aim to predict the concatenation of 11, Top 11, and T2.
DISCOURSE-AWaRd TEST SUITE
The general-purposs automatic metrics (e.g. BLEU and PPL) may be not sufficient to distinguish
model performance in terms of discourse (Wong & Kit 2012 Müller et al.) 2018; Voita et al. 2018
2019 [in et al. 2011). To better measure the ability of models on discourse modeling, we handcraft discourse-aware test suite that is complementary to general evaluation. More specifically, each instance a contrastive pair with correct and incorrect propoteties terms of cohesion or coherence. We directly employ the original texts in the benchmark testset correct candidates. For incorrect candidates, we add noises to the correct one by modifying its discourse devices (cohesion) and structures (coherence). Given the input and output, each instance. Accordingly, we assess models on their ability to rank the correct candidate higher than the incorrect To end, totally construct 600 instances, and each task/type contains 100.
calculate the model score candidates in

---

zh Under review as a conference paper at ICLR 2023
Corpus
Model
Language
Size
Task
Size
Sources
BERT (base)
zh
110M
U,
L.5GB
Wiki
RoBERTa (base)
zh
110M
U.T
15GB
Wiki, EXT Corpus
3
RoBERTa (large)
zh
340M
U.
15GB
Wiki, EXT Corpus
AnchiBERT (base)
zh
102M
U.T
1.5GB
Classical Chinese
MengziBERT (base)
zh
103M
U.T
300GB
Wiki, Common Crawl
BART (large)
zh,
406M
U. G
200GB
Wiki, WuDao Corpus
mBART (CC25)
zh, en, etc.
610M
T
1.4TB
Common Crawl
GPT2 (base)
102M
G
14GB
CLEU Corpus
GPT2 (large)
762M
G
40GB
Web Text
T5 (buse)
zh
231M
14GB
CLEU Corpus
11
T5 (large)
en
770M
G
745GB
C4
12 GuoFeng (family)
zh, en
U. G
400GB
Literature
Table Summary of pretrained models used as baseline, varying model architecture, parameter scale, training data, and targeted task (i.e. understanding, translation, and generation). #1~11 are publicly available. #12 denote series of pretrained models that are continuously trained on our literature-domain data initialized by corresponding parameters in #1~11.
Considering three types of tasks, we employed different strategies, varying in the place that is modified. Table iillustrates how we design contrastive pairs:
Understanding Tasks: for incorrect candidate, add noises the input æ (i.e. - x') while keeping hypothesis y unchanged. Thus, each instance contains correct (r, y) and incorrect (a', y) candidates. We calculate model scores by feeding them into corresponding models.
Translation Tasks: inspired by Bawden et al. (2018) and Voita al. (2019), add noises the target translation to generate an incorrect candidate (i.c. - y while keeping the source input x unchanged. We ask model to score two candidates (x. y) and (z, y') via force-decoding method.
Generation Tasks: for TE task, we add noises to expanded sentences (i.e. y') in text while keep the predefined one unchanged. Given (x. y) or (ar, y'), models can calculate their models scores. In TI and TC, we regard generation models as the language model, which can directly calculate perplexity of a sequence sentences. Thus, we add noises to any sentences text.
BENCHMARK RESULTS
4.1 BASELINES
Plain Models We use the Transformer (Vaswani et al. 2017) with base and big configurations as our plain models. use the Adam optimizer with 31 0.9 and 32 0.98, and employed large batching Ott et al. (2018) for model training. We set the max learning rate to 0.0007 and warmup-steps to 16000. All the dropout probabilities are set to 0.3.
Existing Pretrained Models We systematically compare SOTA pretraining models
structed discourse-aware benchmark, including BERT (Devlin et al. 2019), RoBERTa (Cui et al. 2020), AnchiBERT (Tian al. (02T). MengziBERT (Zhang al. 202 BART (Lewis al. 2020 Shao et al. 2021), mBART (Liu et al. 2020). GPT2 (Radford et al. 2019 Zhhoo et al. 2019) and T5 (Rafell al. 2020 Zhao et al.) 2019). Table 3 shows the summary information of the pretrained models detailed in Appendix CC We fine-tuned these public models on the corresponding datasets for downstream tasks For translation tasks, we use BERT-based pretrained models (e.g. BERT, RoBERTa) to initialize the encoder of NMT' models. We choose the hyper- srper-pprameters based on the performance on the validation set for each model. We fine-tune each model twice and report the averaged test results. The fine-tuning param-eassmeters are detailed in Table 13
Discourse-Aware GuoFeng Pretrained Models vary in different domains (Yang et al. 2015), leading to differences in model behavior and quality
The frequencies and types of discourse phenomena
7

---

Under review as a conference paper at ICLR 2023
Understanding
Translation
Generation
Model
SI
ZPR
MRC
NT
CCT
PT
TE
TF
TC
Plain Models
Transformer (base)
9.1
10.8
39.9
22.0
43,9
6.6
20.2
15.5
20.0
Transformer (big)
4.4
11.1
38.4
22.5
44.8
6.6
22.4
14.7
18.5
Existing
Pretrained Models
BERT (base)
85.1
17.4
49.2
21.6
57.0
6.7
AnchiBERT (base)
81.3
23.2
47.2
22.2
58.2
6.7
MengziBERT (base)
86.9
31.5
47.3
21.8
57.2
6.1
RoBERTa (base)
86.3
28.5
48.6
22.2
57.7
6.4
RoBERTa (large)
88.7
33.0
50.8
22.8
59.5
6.0
BART (large)
86.5
32.8
50.2
20.6
56.7
7.0
41.6
10.2
12.2
mBART (CC25)
23.5
23.1
15.0
GPT2
37.3
13.5
13.5
T5
35.0
12.2
16.1
GuoFeng Pretrained Models
RoBERTa (base)
87.7
31.2
46.6
22.5
58.7
6.6
RoBERTa (large)
89.6
34.3
52.2
21.0
60.1
BART (large)
87.6
33.5
50.3
21.0
62.7
7.1
45.2
8.5
8.0
mBART (CC25)
24.0
25.1
15.7
GPT2
40.4
9.6
9.1
Table 4: Performance of baseline models GuoFeng benchmark. A similar table is presented on the online platform. Bold denotes the best result each column. SI and ZPR are measured by F1 while MRC by accuracy. We report BLEU for NT, CCT, PT and TE, and PPL for others. Results using additional evaluation metrics are reported in Appendix
across domains. However, most existing pretrained models are trained on datasets without discourse information (e.g. sentence level) in mixed domains (e.g. Wikipedia and news). Considering that texts in literature domain contains rich discourse phenomena (De Beaugrande & Dressler 1981). we construct a targe-scale, in in-domain and document-level datasets Chinese and English. The data statistics are detailed in Table [1] To fill the gap, we follow Wang et al. (2022) to train the existing pretraining models (course-ge grained pretraining) on the document-level GuoFeng data in the literal domain tfineggraine pretraining) model discourse phenomena. For each GuoFeng model, use the existing pretrained models for weight initialization, and we further train the models on the GuoFeng data with the same loss. We limit the input length to 512 tokens for RoBERTa models and 1024 tokens for BART, mBART, and GPT models. The pretraining peper-paraeeters details of the release Guofeng models can be found in Table 15
4.2
MAIN RESULTS
Table lists the results on the proposed benchmarks, where several observations can be made. Concerning the existing pretrained models, pretraining improves performance over plain models in all tasks, which is consistent with previous studies. These results validate that the proposed benchmarks are reasonable. Among the BERT variants with the base setting, AncientBERT trained on small-scale classical Chinese data outperforms other models on CCT and PT, demonstrating the necessity of bridging the domain gap. Enlarging the model capacity (e.g. from base to large setting) consistently improves performance.
Clearly, fine-gras grained pretraining on GuoFeng data outperforms their coarse- coarse-grained counterparts, demonstrating the effectiveness and necessity of modeling discourse information. The RoBERTa models work better on language understanding tasks, and the BART variants produce superior performances the language translation and generation tasks.
8

---

Under review as a conference paper at ICLR 2023
Understanding
Translation
Generation
Model
S1
ZPR
MRC
NT
ACT
PT
TE
TI
TC
Existing Pretrained Models
RoBERTa (large)
0.1/0.5
0.4/0.4
0.3/0.2
0.7/0.7
0.8/0.8
0.5/0.6
BART (large)
0.2/0.4
0.3/0.4
0.3/0.2
0.7/0.7
0.7/0.8
0.5/0.6
0.8/0.7
0.6/0.2 0.5/0.6
mBART (CC25)
0.8/0.7
0.2/0.4
0.6/0.6
GuoFeng Pretrained Models
RoBERTa (large)
0.1/0.5
0.4/0.4
0.4/0.2
0.7/0.7
0.8/0.8
0.6/0.6
BART (large)
0.2/0.5
0.4/0.4
0.4/0.2
0.8/0.7
0.8/0.8
0.6/0.6
0.8/0.7
0.6/0.4 0.5/0.7
mBART (CC25)
0.8/0.8
0.4/0.4
0.8/0.7
Table 5: Results of selected baseline models on GuoFeng diagnostic dataset. We assess models on their ability to rank the correct candidate higher than the incorrect one according to model score. We report accuracy (%) in terms of cohesion/coherence.
4.3 ANALYSIS
We evaluate three representative models the diagnostic dataset: RoBERTa (large), BART (large), and mBART (CC25). Each model is fine-tuned on the training data the corresponding downstream task (e.g. SI) and then tested on diagnostic dataset. As illustrated in Table GuoFeng pretrained models generally improve the cohesion and coherence accuracies over their searse-grained coun- terparts, which reconfirms discourse information. Although the numbers are not comparable across tasks, we find that pre- training models on the understanding tasks generally perform worse on discourse modeling. One possible reason is that the understanding tasks are mostly classification tasks, whose signals may not be sufficient guide models learn discourse information.
claim that fine-grained pretraining on GuoFeng data helps model
RELATED WORK
Evaluation benchmarks are important for developing deep learning models, which enable comparison between different models and probe models for understanding of specific linguistic phenomena.
Conneau & Kicla (2018) collected SentEval containing several sentence-level classification tasks to test the representational power of models. Closely related to this work, DiscoEval (Chen al. 2019) extended these tasks to evaluate discourse-related knowledge in pretrained models. DiscoHval only evaluates sentence encoder with language understanding tasks in English. In contrast, we extend the tasks to a boarder range of NLP tasks, which can evaluate different types of models (e.g. encoder-based BERT, decoder- based GPT, and encoder-decodes based mBART). In addition, our benchmarks cover both Chinese and English.
GLUE (Wang al., 2018) and SuperGLUE (Wang al. 2019) included wider variety of natural language understanding tasks, further examining the capabilities of the models and making the results comparable for multi-task learning. Followed researchers extend the benchmarks to other languages, such as CLUE (Xu et al., 2020) and LOT (Guan et al.) (022) in Chinese, and XGLUE (Liang et al.) 2020) in multiple languages. While these works focus on evaluating inter-sentence information our benchmark evaluates intra-sentence discourse phenomena that cross sentences.
CONCLUSION
This paper introduces a GuoFeng benchmark for Chinese and/or English that can evaluate intra- sentence discourse properties across diverse set of NLP tasks, covering understanding, translation, and generation. We also propose diagnostic test suite that can examine whether the target models learn discourse knowledge for in-depth linguistic analysis. Extensive experiments demonstrate that fine-grained pretraining based on document-level training data consistently improves the modeling of discourse information. We offer the datasets, pretrained models, and leaderboards to facilitate research in this field.
"LOT Guan
2022 evaluates models' abilitics model long but ignores discourse information.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Satanjeev Banerjee and Alon Lavie. Meteor: An automatic metric for mt evaluation with improved correlation with human judgments. In ACL, 2005.
Madeleine Bates, Models natural language understanding. Proceedings the National Academy of Sciences, 92(22):9777-99882 1995.
Rachel Bawden, Rico Sennrich, Alexandra Birch, and Barry Haddow. Evaluating discourse phenom- ena in neural machine translation. In NAACL, 2018.
Deng Cai, Yizhe Zhang, Yichen Huang, Wai Lam, and Bill Dolan. Narrative incoherence detection. arXiv preprint arXiv:2012. 2020.
Mingda Chen, Zewei Chu, and Kevin Gimpel. Evaluation benchmarks and learning criteria for discourse-aware sentence representations. In EMNLP-IJCNLP, 2019.
Alexis Conneau and Douwe Kicla. Senteval: An evaluation toolkit for universal sentence representa- tions. In LREC, 2018.
Guy Cook. Discourse. Oxford University Press, 1989.
David Crystal. Dictionary of Linguistics and Phoneties. Oxford: Blackwell Publishers, 1985.
Yiming Cui, Ting Liu, Wanxiang Che, Li Xiao, Zhipeng Chen, Wentao Ma, Shijin Wang, and Guoping Hu. A span-extraction dataset for Chinese machine reading comprehension. In EMNLP. 2019.
Yiming Cui, Wanxiang Che, Ting Liu, Bing Qin, Shijin Wang, and Guoping Hu. Revisiting pre-trained models for Chinese natural language processing. In EMNLP: Findings, 2020.
Robert De Beaugrande and Wolfgang Ulrich Dressler. Einführung in die Textlinguisik, volume 28. Tübingen: Niemeyer, 1981.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep
bidirectional transformers for language understanding. In NAACL, 2019.
Benjamin Franklin Elson and Velma Pickett. Beginning morphology and syntax. Summer Inst of Linguisties, 1983.
Dmitriy Genzel, Jakob Uszkoreit, and Franz Josef Och. Rhyme and meter. In EMNLP, 2010.
"poetic" statistical machine translation:
Marjan Ghazvininejad, Yejin Choi, and Kevin Knight. Neural poetry translation. In NAACL, 2018.
Jian Guan, Xiaoxi Mao, Changjie Fan, Zitao Liu, Wenbiao Ding, and Minlie Huang. Long text generation by modeling sentence-level and discourse-level coherence. In ACL, 2021.
Jian Guan, Zhuoer Feng, Yamei Chen, Ruilin He, Xiaoxi Mao, Changjie Fan, and Minlie Huang. LOT: A senry-centric benchmark for evaluating chinese long text understanding and generation. TACL, 2022.
Michael Alexander Kirkwood Halliday and Ruqaiya Hasan. Cohesion in english. Longman, 1976.
William Hanks. Discourse genres in a theory of practice. American Ethnologist, 14(4):668-992, 1987.
Hua He, Denilson Barbosa, and Grzegorz Kondrak. Identification of speakers in novels. In ACL, 2013.
Yichen Huang, Yizhe Zhang, Oussama Elachqar, and Yu Cheng. INSET: Sentence infilling with INter-SEntential transformer. In ACL, 2020.
Paul Me Kevitt, Derek Partridge, and Yorick Wilks. Approaches to natural language discourse processing. Artificial Intelligence Review, 6(4):333-364, 1992.
10

---

Under review as a conference paper at ICLR 2023
Fang Kong and Guodong Zhou. A tree kernel-based unified framework for chinese anaphora resolution. In EMNLP, 2010.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. Bart: Denoising sequence-to-seequeec pre-training for natural language generation, translation, and comprehension. In ACL, 2020.
Charles Li and Sandra A Thompson. Mandarin Chinese: functional reference grammar. Uni-
versity of California Press, Oakland, California, USA, 1989.
Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao, and Bill Dolan, A diversity pversity-promoting objective function for neural conversation models. In NAACL, 2016.
Yaobo Liang, Nan Duan, Yeyun Gong, Ning Wu, Fenfei Guo, Weizhen Qi, Ming Gong, Linjun Shou, Daxin Jiang, Guihong Cao, Xiaodong Fan, Bruce Zhang, Rahul Agrawal, Edward Cui, Sining Wei, Taroon Bharti, Ying Qiao. Jiun Hun-Hung Chen, Winnie Wu, Shuguang Liu, Fan Yang, Rangan Majumder, and Ming Zhou. XGLUE: A new benchmark dataset for cross-lingual pre-training, understanding and generation. CoRR, 2020.
Ziheng Lin, Hwee Tou Ng, and Min- Yen Kan. Automatically evaluating text coherence using discourse relations. In ACL. 2011.
Shanshan Liu, Xin Zhang, Sheng Zhang, Hui Wang, and Weiming Zhang. Neural machine reading comprehension: Methods and trends. Applied Sciences, 918):3669,, 2019a.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv: 1907. 1692, 2019b.
Yinhan Liu, Jiatao Gu, Naman Goyal, Xian Li, Sergey Edunov, Marjan Ghazvininejad, Mike Lewis, and Luke Zettlemoyer. Multilingual denoising pre- pre-training for neural machine translation. 2020.
Robert E Longacre. Storyline concerns and word order typology in East and West Africa, volume 10. Los Angeles: African Studies Center, UCLA, 1990.
William C Mann and Sandra A Thompson. Rhetorical structure theory: Toward a functional theory of text organization. Text-Interdisiillnary Journal for the Study of Discourse, 83)):233-211 1988.
Evgeny Matusov. The challenges of using neural machine translation for literature. In Proceedings of the qualities of literary machine translation, 2019.
Ruslan Mitkov. Anaphora resolution. Routledge, 2014.
Mathias Müller, Annette Rios, Elena Voita, and Rico Sennrich. A large-scale test set for the evaluation of context-aware pronoun translation in neural machine translation. In WMT, 2018.
Myle Ott, Sergey Edunov, David Grangier, and Michael Auli. Scaling neural machine translation. In WMT, 2018.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. BLEU: A Method for Automatic Evaluation of Machine Translation. ACL, 2002.
Telmo Pires, Eva Schlinger, and Dan Garrette. How multilingual is multilingual bert? In ACL, 2019.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, Peter J Liu, et al. Exploring the limits of transfer learning with a unified text-to-text transformer. Mach. Learn. Res.. 21(140):1-67, 2020.
Pranav Rajpurkar, Jian Zhang. Konstantin Lopyrev, and Percy Liang. SQuAD: 100,000+ questions for machine comprehension of text. In EMNLP, 2016.
11

---

Under review as a conference paper at ICLR 2023
Sudha Rao, Allyson Ettinger, Hal Daumé III, and Philip Resnik. Dialogue focus tracking for zero pronoun resolution. In NAACL, 2015.
Ricardo Rei, Craig Stewart, Ana C Farinha, and Alon Lavie. COMET: A neural framework for MT evaluation. In EMNLP, 2020.
Ehud Reiter and Robert Dale. Building applied natural language generation systems. Natural Language Engineering, 3(1):57-87, 1997.
Sanders and H Pander Maat. Cohesion and coherence: Linguistic approaches. Reading, 99: 440-466, 2006.
Yunfan Shao, Zhichao Geng, Yitao Liu, Junqi Dai, Fei Yang, Li Zhe, Hujun Bao, and Xipeng Qiu. Cpt: A pre-trained unbalanced transformer for both chinese language understanding and generation. arXiv preprint urXiv:2109.05729 2021.
Matthew Snover, Bonnie Dorr, Richard Schwartz, Linnea Micciulla, and John Makhoul. A study of translation edit rate with targeted human annotation. In AMTA, 2006.
Linfeng Song, Kun Xu, Yue Zhang, Jianshu Chen, and Dong Yu. ZPR2: Joint zero pronoun recovery and resolution using multi-task learning and BERT. ACL, 2020.
Kai Sun, Dian Yu, Dong Yu, and Claire Cardie. Investigating prior knowledge for challenging chinese machine reading comprehension. TACL, 2020.
Huishuang Tian, Kexin Yang. Dayiheng Liu, and Jiancheng Lv. Anchibert: pre-trained model for ancient chinese language understanding and generation. In 2021 IEEE International Joint Conference on Neural Networks, 2021.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. In NeurlPS, 2017.
Elena Voita, Pavel Serdyukov, Rico Sennrich, and Ivan Titov. Context-aware neural machine translation learns anaphora resolution. In ACL, 2018.
Elena Voita, Rico Sennrich, and Ivan Titov. When good translation is wrong in context: Context- aware machine translation improves on deixis, ellipsis, and lexical cohesion. ACI, 2019.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. GLUE: multi-task benchmark and analysis platform for natural language understanding. In EMNLP, 2018.
Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. Superglue: A stickier benchmark for general-purpose language understanding systems. In NeurIPS, 2019.
Longyue Wang, Zhaopeng Tu, Xiaojun Zhang, Hang Li, Andy Way, and Qun Liu. A novel approach for dropped pronoun translation. In NAACL, 2016.
Longyue Wang, Zhaopeng Tu, Andy Way, and Qun Liu. Exploiting cross-sentence context for neural machine translation. In EMNLP. 2017.
Wenxuan Wang, Wenxiang Jiao, Yongchang Hao, Xing Wang, Shuming Shi, Zhaopeng Tu, and Michael Lyu. Understanding and improving sequence-ter sequence-to-sequees to-seunence pretraining for neural machine translation. In ACL, 2022.
Leo Wanner. Lexical choice in text generation and machine translation. Machine Translation, 11(1): 3-35, 1996.
Billy TM Wong and Chunyu Kit. Extending machine translation evaluation metrics with lexical cohesion to document level. In EMNLP, 2012.
Deyi Xiong, Guosheng Ben, Min Zhang, Yajuan Lv, and Qun Liu. Modeling lexical cohesion for document-level machine translation. In IJCAI, Beijing, China, 2013.
12

---

Under review as a conference paper at ICLR 2023
Liang Xu, Xuanwei Zhang, and Qianqian Dong. Cluecorpus2020: A large-scale chinese corpus for pre-training language model. ArXiv, abs/2003.01355, 2020.
Yaqin Yang and Nianwen Xue. Chasing the ghost: recovering empty categories in the chinese treebank. In COLING, 2010.
Yaqin Yang, Yalin Liu, and Nianwen Xue. Recovering dropped pronouns from chinese text messages. In ACL-IICNLP, 2015.
Changchang Zeng, Shaobo Li, Qin Li, Jie Hu, and Jianjun Hu. A survey machine reading comprehension-taaee evaluation metrics and benchmark datasets. Applied Sciences, 10(21): 7640, 2020.
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Weinberger, and Yoav Artzi Bertscore: Evaluating text generation with bert. In International Conference on Learning Representations, 2019a.
Weinan Zhang. Ting Liu, Qingyu Yin, and Yu Zhang. Neural recovery machine for Chinese dropped pronoun. Frontiers of Computer Science, 2019b.
Zhuosheng Zhang, Hanqing Zhang, Keming Chen, Yuhang Guo, Jingyun Hua, Yulong Wang, and Ming Zhou. Mengzi: Towards lightweight yet ingenious pre-trained models for chinese. arXiv preprint arXiv:21 10..2110..6696, 2021.
Zhe Zhao, Hui Chen, Jinbin Zhang, Wayne Xin Zhao, Tao Liu, Wei Lu, Xi Chen, Haotang Deng, Qi Ju, and Xiaoyong Du. Uer: open-source toolkit for pre-training models. In EMNLP, 2019.
Chujie Zheng, Minlie Huang, and Aixin Sun. ChiD: A large-scale Chinese Diom dataset for cloze test. In ACI, 2019.
Wanrong Zhu, Zhiting Hu, and Eric Xing. Text infilling. arXiv preprint arXiv: 1901 00158, 2019.
13

---

Under review as a conference paper at ICLR 2023
A CONCEPT OF DISCOURSE
discourse is an instance of language use whose type can be classified on the basis of such factors as grammatical and lexical choices and their distribution in main versus supportive materials, theme, style, and the framework of knowledge and expectations within which the addressee interprets the discourse (Elson & Pickett 1983 Crystal 1985 Hanks 1987, Longacre 1990). A discourse contains seven fundamental properties including cohesion, coherence, intentionality, acceptability, informatively, situationality and intertextuality (De Beaugrande & Dressler] 1981). Among them, cohesion and coherence have often been studied in discourse analysis (Sanders & Maat 2006 Xiong
al. 2013
A.1 COHESION
Cohesion occurs whenever "the interpretation of some element in the discourse is dependent on that of another" (Halliday & Hasan 1976). The referential cohesion (i.e. anaphora and coreference) and lexical cohesion (i.e. reperition and collocation) are commonly-use cohesive devices. The examples are shown Figure
Anaphora
(a)
Audi
automaker that luxury cars.
It
was established by August Horch.
Corvferince
(b) invited the 1st
Chief Executive
to our school. Mr. Tung Chee-hwa
story.
Repatition (same wort) synonyms)
(c) Which
dress/frock are you going wear? wear my green frock/dress
Collocation
(d) Once time
there was ugly duckling.
Figure 5: Examples of different cohesion devices.
Anaphora is the use of an expression whose interpretation depends specifically upon antecedent expression. The anaphoric (referring) term is called an anaphor. Sometimes anaphor may rely on the postcedent expression, and this phenomenon is called cataphora. As shown in Figure 5]a). the pronoun "It" an anaphor, which points to the left toward its antecedent "Audi". Zero anaphora is a more complex case of anaphora. In pro-drop languages such as Chinese and Japanese, pronouns can be omitted to make the sentence compact yet comprehensible when the identity of the pronouns can be inferred from the context. These omissions may not be problems for our humans since we can easily recall the missing pronouns from the context.
Coreference Two or more expressions (e.g. nouns) in text refer to the same referent. As the referents point to persons or things in the real world, the coreference relation can exist independently of the context. As shown in Figure 5]b), the noun phrases "HK Chief Executive" and "Mr. Tung Chee-hwa" point to the same person, although their surfaces are totally different.
Lexical Cohesion Lexical cohesion refers to the way related words are chosen to link elements of text. The "repetition" indicates the linking between the same word, or synonyms, antonyms, etc. As shown in Figure(S(c). the synonyms "dress" and "frock" across two sentences are the repetition case. In the "collocation" form, related words are typically put together or tend to repeat the same meaning. For example, the phrase "once upon time" in Figure 5)d) is collocation case.
A.2 COHERENCE
Coherence is created referentially, when different parts of a text refer to the same entities, and relationally, by means of coherence relations such as "CauseeConsseuuecce"" between different
14

---

Under review as a conference paper at ICLR 2023
discourse segments. The discourse structure such as RST (Rhetorical Structure Theory, (Mann & Thompson 1988) is usually used to analyze the coherence of text.
RST relations applied recursively in text until all units in that text constituents in predefined relation. As shown in Figureló) the result of such analysis is that RST structure is typically represented as tree, with one top-level relation that encompasses other relations lower levels. There are
number of predefined relations such as "Attribution" " (causality) and "Contrast" (adversative relation), and the leaves are presented segments/parts of the text
Telxco Corp. said its president resigned and its Houston work force has been trimmed by 15 %. The marker of computer
systems said the personmel changes were needed to improve the efficiency of its manufacturing operation. The company said it hasn't 't named a successoe to Ronald Button, the president who resigned. Its Houston work force now totals 230.
Elaboration
Faboration
Attribution
Actribution
Attributionn
compony SPd
Elaboration
loint
Themakker of commppier
Enabioment
Attribution
Burald preeddmt
Horaton work finne
president
pensrrd
Improve Be
chonges nosded
dfficency or las
tooll 230
manufeturng operaioo
Figure 6: An example of coherence represented by RST tree.
B ADDITIONAL TASK AND RESULT DETAILS
The English translation of examples in Figure 3 and 4
are listed in Table 6
B. LANGUAGE UNDERSTANDING TASKS
SI The macro-averaged F1 and exact match (EM) can be used as the evaluation metrics following standard extractive machine reading comprehension tasks (e.g. (Rajpurkar et al. 2016)). The EM scores are reported in Table
ZPR
Table 7 shows all Chinese pronouns with English translation. Chinese pronouns correspond the personal pronouns in English, and the Chinese pronominal system relatively simple as there is no inflection, conjugation, or case makers (Li & Thompson 1989). Thus, there is no difference between subjective and objective pronouns (we call them "basic pronouns"). Besides, possessive and reflexive pronouns can be generated by adding some particle or modifier based on the basic pronouns.
The (zero) anaphora resolution is an alternative task on discourse understanding, which aims to identify the antecedent of a referential (zero) pronoun (Kong & Zhou 2010) Mitkov However, we did not consider this task for two reasons: (1) more than 50% zero pronouns are non-anaphoric which can not be modelled in the resolution task (Rao et al. 2015); (2) different from previous benchmarks such as OntoNotes and CLUEWSC2020 which mainly focus on explicit pronouns, while ZPR considers implicit pronouns which are complementary each other. We follow common practice to use micro Fl score as the standard evaluation metric. And we also report precision and recall in Table g
MRC Considering the average length of texts, the Haihua2021 corpus is also more challenging than 3 (i.e. the length ratio is 177:
httpp://www.ff..cca/aac/
15

---

Under review as a conference paper at ICLR 2023
Task
Discourse Context
Task Description
Figure Xing Jiu followed Mu Qing into the and the co-pilot position.
ST
Inp: "Um, yes."
Out: Speaker=Xing Jiu"an
"Are you in bad mood?" Mu Qing asked.
"Um, yes."
ZPR
A: Phoebe would love to buy a TV.
Inp: B: Joey won't 't let & buy Ø??
B: Joey won't let 21 buy 2??
Out: B: Joey won't let her buy
A: Yes.
MRC
The little princess climbed out of the castle window while her mother was sleeping.
Inp: Where did the little princess go after she escaped?
She elimbed down the south wall and slipped out.
(A) South Wall; (B) Forest; (C) Castle; (D) Mountain.
Finally Ø walked into the forest without telegraph poles.
Out: Answer=(B) Forest
Figure 3
NT
King Ding sat on the side, smiling as he looked at Qing. Shuang's 's astounded thoughts.
Inp: 2 mind had already flown to faraway place.
Out:
Ø mind had already flown to a faraway place_
CCT
C when she playing Xiao, only her beautiful face remain usual, but also her charm increases. Why?
Inp: @. when shrinking @ month blow, mouth appears be smaller. Out: Besides, when shrinking her month blow, her mouth appears to be smaller.
@ Ø is playing, @ fingers press the holes on the flute. and in this way, Ø tender and slim fingers will seem be slimmer and fairer.
0 when shrinking 2 month to blow, Ø mouth appears to smaller.
PT
I ask your lad beneath
Inp: ask your lad beneath tree. Out:
"My master's 's gone for herbs, says he,
"Amid the hills know not where. For clouds have veiled them here and there."
Figure
TE
TT
Mu Xiaoxiao looked his back aggrieved, why did suddenly change like this?
Inp: Mu Xiaoxiao looked at his back aggrieved,
why did suddenly change like this? [x] [x]
She inexplicably trained for while, which made her feel bad.
When she got class S, she lying on the table and was sullen.
When she got to class S, she was lying on the table and was sullen.
Out: She was inexplicably trained for while, which made her feel bad.
TC
Chen Xu hungry and cold. He used small gas stove to cook pot of noodles. The two gathered around the pot and devoured every- thing.
Inp: Chen Xu was hungry and cold. [x] [x] [x]
Out: The two gathered around the pot and de- voured everything. After they ate the noodles, they felt alive.
After they ate the noodles, they felt alive.
Table 6: English translations of examples in Figure 2 and to map discourse phenomena into English language.
Some are literal translations in order
B.2 LANGUAGE TRANSLATION TASKS
There are a number of evaluation metrics for measuring general performance of MT systems. BLEU the most widely-used one, which measures the precision of n-grams of the MT output compared to the reference, weighted by brevity penalty to punish overly short translations (Papineni et al. 2002). TER is an error metric for machine translation that messures the number of edits required to change system output into one of the references (Snover et al. (006). METEOR incorporates semantic information by calculating either exact match, stem match, or synonymy match (Banerjee Lavie 2005). Furthermore, COMET is a neural framework for training multilingual MT evaluation models which obtains new SOTA levels correlation with human judgements (Rei et al. 2020). We report TER, METEOR and COMET in Table
NT (Novel Translation) We crawl 45,134 chapters in 152 books from web fiction websites, covering 14 genres such as fantasy science and romance. We manually align them into parallel corpus with document boundary. The copyright owner has agreed open the data to research community
16

---

Under review as a conference paper at ICLR 2023
Form
Subject
Object
Possessive adjective
Possessive
Reflexive
1st SG
क8 (I)
2 (me)
# (my)
#2.00 (mine)
E1
A9
(myself)
2nd SG
fR (you)
8: (you)
14:69 (your)
A1 (yours)
19: H
R9
(yourself)
SGM
fiz (he)
fi (him)
fibA (his)
Aib (his)
11E H
(himself)
3rd SGF
(she)
iib (her)
At (f (her)
AM (hers)
1
A9
(herself)
3rd SGN
E (it)
" (me)
' 69) (its)
'2 ff) (its)
1 ff (itself)
-
1st PL
2i (we)
48 (us)
Hs (your)
Med [8] (yours)
14 E
A9
(yourselves)
2nd PL
fi (you)
M:ci (you)
ft (our)
fB (ours)
(ourselves)
3rd PLM
f (they)
te:fil (them)
f (their)
Ht (theirs)
Ht E
(themselves)
3rd PLF
S (they)
dbb.fl (them)
kB (their)
MB ] (theirs)
RET
E
f9
(themselves)
3rd PLN
Efi (they)
'Eiil (them)
(69) (their)
E (89 (theirs)
E
2
ft (themselves)
Table 7: Ehinese-English pronouns with corresponding forms. The pronoun types are short for:
person 1st, 2nd, 3rd, singular = SG, plural = PL. male = M, female = F and neutral = N.
ST Exact Match
ZPR
Model
Precision
Recall
Plain Models
Transformer (base)
0.3
10.2 10.5
11.5
Transformer (big)
0.1
11.9
Existing Pretrained Models
BERT (base)
81.9
26.1
31.0
AnchiBERT MengziBERT
76.9
24.6
84.0
36.6
29.6
RoBERTa (base)
83.4
29.0
29.9
RoBERTa (large)
85.9
39.3
28.7
BART (large)
83.7
38.3
30.2
Guoleng Pretrained Models
RoBERTa (base)
85.2
32.0
30.6
RoBERTa (large)
87.2
38.7
30,8
BART (large)
84.6
39.0
30.5
Table More results on understanding tasks using additional evaluation metrics, including Exact Match, Precision, and Recall. This is complementary to Table
under the Open Data Commons Open Database License (ODbL). Thus, researchers can use the dataset for non-commercial research purposes and follow the principle of fair use (e.g. CC BY).
B.3 LANGUAGE GENERATION TASKS
There are number of automatic evaluation metrics for measuring the quality of generated texts. We use two groups of metrics: (1) siserence-hased scores BLEUU (Papineni et al. 2002) and BERTScore (Zhang et al. 2019a), which measure the lexical and semantic similarities between the generated texts and the ground-truth references respectively. Note that, for open-ended text generation tasks such as TI and TC, reference-basee metrics are less reliable because the generated text could be of high quality but different from the ground-truth reference. How to accurately measure the performance of open-ended text generation is still an open question and is beyond the scope of this paper. (2) Dist-n scores (Li al. 2016) calculate the ratio of distinct n-grams in generated text to evaluate lexical diversity. We report Dist-2 and Dist-4 as well. Additional results are shown in Table I0
TE To better understand the task of TE, we present some examples Table In order to retain the coherence and meaning of the souree document, the expanded parts in the target text tends to be modifier phrases or clauses. In general, the expanded contents in our task are mainly summarized into 5 categories, shown in Table 12 To evaluate the TE models,
two metrics: BLEU and PPI..
17

---

Under review as a conference paper at ICLR 2023
NT
CCT
PT
Model
TER*
MET.
COM.
TER
MET.
COM.1
TER+
MET.
COM.1 t
Plain Models
Transformer (base)
70.0
24.0
0.10
70.4
0.65
104.5
10.3
-0.82
Transformer (big)
69.2
24.5
0.14
77.5
0.65
103.6
11.1
-0.78
Existing Pretrained Models
BERT (base)
69.6
24.0
0.10
27.6
0.65
97.0
13.3
-0.68
AnchiBERT
69.5
24.3
0.11
27.1
0.67
91.4
13.4
-0.66
MengziBERT
69.8
23.9
0.10
0.67
11.8
RoBERTa (basc)
69.7
24.0
0.13
27.2
0.65
96.0
12.2
-0.71
RoBERTa (large)
68.9
25.0
0.15
27.0
0.68
97.5
11.9
-0.75
BART (large)
70.9
24.6
0.12
28.2
0.65
90.5
13.5
-0.74
mBART(CC225)
64.7
26.9
0.28
0.23
72.4
21.7
-0.14
GuoFeng Pretrained Models
RoBERTa (hase)
69.7
24.2
0.14
0.67
96.0
12.3
RoBERTa (large)
70.0
24.9
0.15
28.3
0.67
95.9
13.0
-0.74
BART (large)
70.0
24.8
0.12
25.6
0.70
90.4
13.3
-0.74
mBART (large)
64.8
27.0
0.30
0.23
22.0
Table 9: More results on translation tasks using additional evaluation metries, including TER, METEOR and COMET. This is complementary to Table
TE
TT
TCT
Model
PPL
BLEU
BERTscore
Dist-2
Dist-4
BLEU
BERTscore
Dist-2
Dist-4
Existing Pretrained Models
BART (large)
48.0
3.7
62.2
0.2
0,6
2.7
60.3
0.1
0.4
GPT2
50.7
1.6
59.4
0.2
0.5
2.1
57.6
0.0
0.2
54.8
3.3
61.1
0.1
0.5
2.2
57.3
0.1
GuoFeng Pretrained Models
BART (large)
34.0
3.9
62.5
0.2
0.7
4.2
61.0
0.0
0.6
GPT2
45.0
2.2
60.2
0.7
4.7
60.2
Table 10: More results generation tasks using additional evaluation metrics, including BLEU, BERTscore, Dist-2 and Dist-4. This complementary Table 4
TI To evaluate different models, take the following automatic metries: Perplexity (PPL), BLEU (Papineni et al. 2002). BertScore (Zhang et al.) (019a) and diversity scores (Dist-2/4) (Li
al. 2016). We report degree of diversity by calculating the ratio of distinct 2-grams/4-grams in generated text.
TC To evaluate different models, we take the following automatic metrics: Perplexity (PPL), BLEU, BertScore and diversity scores.
C ADDITIONAL EXPERIMENT DETAILS
evaluate the following public pretrained models on GuoFeng Benchmark and Test Suite:
BERT (base): we use the base model (12 layer encoder, hidden size 768, vocabulary size 21 128) published by Devlin et al. (2019), which was pretrained on Chinese Wikipedia dump of about 0.4 billion tokens using the losses of mask language model (MLM) and next sentence prediction. [
RoBERTa (base): Cui et al. (220)) model with the same architecture of BERT (base) except it uses whole word masking and is trained on additional 5 billion tokens with only MLLM pretrained
task. This model uses BERT (base) as the initial weight. 10
httpp::/hhugiingfacccco//// hesee
nese- roberterrbaaaaaaa www.w/wtt//eee//maiiii
18

---

Under review as a conference paper at ICLR 2023
In 1823 James Monroe proclaimed the doctrine. The United States was infant, threatened by European actions.
Source
To this effect, in 1823 President James Monroe proclaimed the doctrine that still bears his name. The United States at this time actions.
Target
infant, weak country, threatened by European
First the rule. Political democracies have not been institutionalized in parts of Latin America. No democratic regime had lasted half century.
Source
First was the weakness of democratic rule. Political democracies even now still have not been firmly institutionalized in parts of Latin America. In the past no democratic regime had lasted half century.
Target
The peasant was sentenced to death, and was to be rolled into the water. He was led forth, and priest was brought.
Source
The innocent little peasant was unanimously sentenced to death, and was to he rolled into the water, in a barrel pierced full of holes. He was led forth, and priest was brought who was to say mass for his soul.
Target
Table Three examples to illustrate the task of TE, where the blue span in Target are expanded content generated based on the source input as context.
Expanded Content Type
Exemplar Spans
(1) adjective (phrase)
innocent little
(2) adverb (phrase)
firmly, even now still, unanimously President, weak country. weakness of democratic To this effect, In the past, at this time, in harrel pierced full of holes that still bears his name, who was to say for his sou
(3) noun (phrase)
(4) prepositional phrase
(5) attributive clause
Table 12: The expansion types in TE task are summarized. All the exemplar spans are highlighted in
texts in Table 11
RoBERTa (large): Cui et al. (2020) the large model size of RoBERTa model (24 layer encoder, hidden size 1024, vocabulary size 21128) This model has the same training procedure of RoBERTa- wwm-ext (base). This model is trained from scratch. [1
AnchiBERT: Tian al. [202] model continues pretraining based on the BERT (base) model with the 39.5M anchient Chinese tokens. It uses the same tokenizer and other techniques as BERT- base.
MengziBERT; Zhang et al. [001 model initial on the RoBERTa (base) (Liu et al. 2019b with special-designed objectives.
BART (large): Shao et al. (2021 train large model (12 layer encoder and 12 layer decoder, hidden size 1024, vocabulary size 21 128) with denoising auto-encoding (DAE) objective. This model is
trained on the open source large-scale raw text, Chinese Wikipedia, and a part of WuDaoCorpus. The training data contains 200GB cleaned text ranging from different domains.
(2019) use large model (12 layer encoder and 12 layer decoder, hidden size 1024, vocabulary size 250,000), trained with 25 language web corpus. This model is trained from scratch.
mBART (CC25): Pires et al.
GPT22: Zhao al. 2019 train 12-layer decoder-only Transformers and its vocabulary is size 21,128. This model is trained with the CLUUECorpusSmall corpus. 16
92
19

---

Under review as a conference paper at ICLR 2023
T5: For Chinese langauge, Zhao al. 2019) train 12-layer encoder-decoder Transformers and its vocabulary is size 21.128. This model is trained with the CLUECCrpusSmal corpus. For English langauge, Raffel et al. (2020) provides transformer model (24 layer encoder and 24 layer decoder, hidden size 1024, vocabulary size 32,000), trained with unified generation task on C4 dataset.
Task
Batch Size
Max Length
Epoch
Learning Rate
512
5
ZPR
512
40
5e-6
MRC
512
10
2e-5
NT
3K token
1024
30K step
lc-4
ACT
3K token
1024
30K step
le-4
PT
3K token
1024
30K step
le-5
TE
32
512
3
2e-4
TI
24
64
3
2
TC
24
512
8
2e-5
Table 13: A summary hyper- parrmmer for fine-tuning downstream tasks.
Size
Category
Genre
Description
# Document
# Sentence
# Character/Word
Chinese Language 1,1,9,1771191
Electronic
Novel
91,620,211
58,639,,54,317
Web Fiction
Classical
38,495,887
490,733,235
24,613,514,541
Masterpiece Publication
Modernist
Book
324,912
4,141,874
155,189,807
Poetry
378,323
1,495,466
31,746,541
Shi, Ci, Qu, Fu
Ancient
Couplet
8,979,186
8,979,186
192,214,600
Antithetical Couplet
Classical
1,011
1,947,136
53,721,504
Ancient Text
Lyrics
452.715
4,952,039
165.338,679
World's Songs
Screenplay
5,213
10,426,213
156,390,000
Movic Script
Others
Movic
66,050
24,108,241
642,392,397
Movic Subtitle
Dialogue
3,642
1,653,469
49,406,618
Talk, Message
Total
140,327,150
1,717,564,050
84,699,369,004
English Language 422,757,234
Electronic
Novel
33,156,134
26,777,401,794
Web Fiction Masterpiece Publication
Classical
3,104,507
39,593,119
2.507,247,359
Modernist
Book
324,912
4,162,821
78,695,499
Ancient
Poetry
2,269
21,456
148,222
World's Poetry
Lyries
3,088,688
110,268,328
632,820,393
World's Songs
Movie Script
2,826
12,534.815
67,433,609
Movie Script
Others
Movie
155,670
56,819,567
315,189,001
Movie Subtitle
Dialogue
9,191
4,172,736
27,208,957
Talk, Message
Total
39,844,197
650,330,076
30,406,144,88 144,834
Table 14: Statistics of data for GuoFeng pretraining. All data are extracted from literature texts with discourse context. We count number of characters Chinese and number of words English.
20

---

Under review as a conference paper at ICLR 2023
Model
RoBERTa
GPT2
BART
mBART
Tokenization
BERTtok. Adam
BERTrok.
BERTtok.
SentPicce
Optimizer
Adam
Adam
Adam
Masking
word
word
word
Vocabulary Size
21128
21131
21128
250000
Learning Rate
3e-4
3e-4
3e-4
3c-4
Batch Size
4K
4K
Training Step
1M
1M
1M
1M
Max Length
512
512
512
1024
Layer
12/24
36
24
12/24
Head
12/16
36
12/16
Total Param.
110m/340m
737M
477M
669M
Table 15: The summary of hyper-parameters used for Guofeng pretrained models.
21


Reviews:
[{"review_id": "kzkLSa4Mqv", "reviewer": "Reviewer_CWMG", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper describes a new collection of datasets, the goal being to analyze discourse phenomena.", "summary_of_the_review": "Overall, this is worthwhile work, but there are a few issues:\n\n* the lack of any measures of quality of these datasets is problematic. I would be hesitant to use this data without such measures\n\n* Clarity is an issue in several places. what does \"aims to recover omitted pronouns in terms of position and form, according to its anaphora information in the given sentence\" mean?, for example\n\n* I think the argument that these are \"discourse level\" tasks is a stretch in several cases. For example, several of the datasets are conventional translation datasets.\n\nOverall I think the goal to release data is valuable, but the paper needs to be improved.", "strength_and_weaknesses": "Strengths: data set construction is valuable, and the goals of the work seem worthwhile.\n\nWeaknesses: first, there is no attempt to evaluate the quality of the resulting data, either through expert evaluations or inter-annotator consistency (I think the former is the only way to really check for this). Second, many of the details of the paper are unclear. Third, many claims are made without sufficent support. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "aeDDttvQZd", "reviewer": "Reviewer_eMwY", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents the GuoFeng benchmark consisting of i) 9 NLP tasks, ii) a contrastive testing set, iii) a large scale training data. The aim is to evaluate the abilities of models for cohesion and coherence across clauses and sentences. The authors also conduct experiments with transformer models using various settings, including randomly initialized ones, pre-trained models, and pre-trained models on the GuoFeng training data. ", "summary_of_the_review": "Overall, this paper can benefit from a major revision, although the resources are interesting for the community. More insights from the experiments are expected, not just model A is better than model B, expecially about what are the new challenges that are not covered by the prior works, evidenced by the experimental results.", "strength_and_weaknesses": "Strengths:\n\n* The datasets are valuable for the document-level NLP research, which need to address the coherence and cohesion across clauses and sentences.\n* The experiments provide baseline results for future research.\n\nWeaknesses:\n* I have difficulties to fully understand the settings of the 9 tasks. How to evaluate each task?\n     * SI: are the speakers always mentioned before the utterance of interest? What are the expected outputs if there are multiple utterances with quotation marks?\n     * ZPR: What are Chinese-English movie subtitles? Are they Chinese movies with English subtitles, English movies with Chinese subtitles, or both?\n     * NT: Is eclipsis not important for novel translation? How about concept consistency in scientice fictions?\n     * CCT: do you mean traditional Chinese? Novels written in traditional Chinese?\n     * PT: what's the target language? English?\n     * How to evaluate the generation tasks if there are multiple plausible answers?\n* What is the inter-annotator agreements for the tasks requiring human annotations?\n* What are the inter-annotator agreements for the discourse-aware test suite? How are the annotators instructed to construct adversarial examples or the noises are added by algorithms? What kind of noises?\n* Why coreference resolution models are not considered as part of or as baselines of applicable tasks? \n* It is not clear to me what are the new challenges arising from the benchmark? ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "X2qpGvWL1D", "reviewer": "Reviewer_zDLD", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a novel set of benchmark resources for discourse-aware evaluation of language undersatnding, translation, and generation, focusing on Chinese and English languages. The release includes benchmarking data, diagnostic test suite, pretraining corpora and pretrained models. The benchmark is applied on relevant models across a wide variety of tasks, showing benefits of document-level pretraining in Transformer architectures when it comes to discourse.", "summary_of_the_review": "Contained above.", "strength_and_weaknesses": "Strengths:\n- Potentially a very useful resource for discourse-aware evaluation across tasks.\n- Very thorough description of the resources, including in-depth background in the appendix.\n- Considerable experiment breadth.\n- High-quality writeup.\n\nWeaknesses:\n- The paper does not reflect thoroughly on the linguistic bias introduced by choosing Chinese and English as the focus languages. There should be a Limitations section that addresses how the proposed approach would scale in low-resource scenarios or if extending the breadth of linguistic coverage in the datasets. What would it take to introduce another language? What if that language is low-resource? How would that support the findings? Can we simulate that with the existing data to an extent?\n- Experiment breadth is not backed with experiment depth, at least in the main part of the paper, as the Main Results and Analysis are rather superficial. This is a minor concern since the main focus of the paper is to provide the resource.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject