Enhancing Ancient Chinese Understanding with Derived Noisy Syntax Trees
Year: 2023

Abstract:
Despite the rapid development of neural-based models, syntax still plays a crucial role in modern natural language processing. However, few studies have incorporated syntactic information into ancient Chinese understanding tasks due to the lack of syntactic annotation. This paper explores the role of syntax in ancient Chinese understanding based on the noisy syntax trees from unsupervised derivation and modern Chinese syntax parsers. On top of that, we propose a novel syntax encoding component – confidence-based syntax encoding network (cSEN) to alleviate the side effects from the existing noise caused by unsupervised syntax derivation and the incompatibility between ancient and modern Chinese. Experiments on two typical ancient Chinese understanding tasks, ancient poetry theme classification and ancient-modern Chinese translation, demonstrate that syntactic information can effectively enhance the understanding of ancient Chinese over strong baselines, and that the proposed cSEN plays an important role in noisy scenarios.

heading: 1 Introduction
text: Ancient Chinese literature, such as classical poetry, books, and records, is a highly representative and distinctive cultural heritage that is receiving increasing attention from the NLP academia. However, directly applying modern Chinese processing methods to ancient texts is not appropriate due to the differences in syntax and semantics between ancient and modern Chinese. Chinese is one of the oldest written languages in the world, with a history of at least 6,000 years (Norman, 1988). Over time, the language has undergone many changes, such as the transition from literary to vernacular Chinese in the early 20th century (Weiping, 2017), resulting in a significant gap between ancient and modern Chinese.
∗ Corresponding authors.
Syntactic features has been utilized in a wide range of NLP tasks, including coreference resolution (Fang and Fu, 2019; Trieu et al., 2019; Jiang and Cohn, 2022), machine reading comprehension (Zhang et al., 2020; Guo et al., 2020), and machine translation (Currey and Heafield, 2019; Zhang et al., 2019a; Bugliarello and Okazaki, 2020). Despite the effectiveness of syntax in modern Chinese understanding (Li et al., 2018; Xia et al., 2019; Zhang et al., 2020), few studies have incorporated syntactic information into ancient Chinese processing. Most works only take into account explicit features, such as era (Chang et al., 2021) and imagery (Shen et al., 2019), ignoring implicit syntactic features. The main reason for this lies in two aspects: (1) the linguistic gap between ancient and modern Chinese makes it difficult for supervised modern Chinese syntax parsers to correctly parse ancient Chinese expressions; (2) training a supervised ancient Chinese syntax parser from scratch can be highly costly due to the lack of annotated data.
Unsupervised syntax parsing or directly employing modern Chinese parsers will inevitably cause noise and performance degradation. A unlabeled example and corresponding human annotation on ancient Chinese sentence "可怜人似月中孀(It is
83
pitiful like Chang’e in the moon)" are shown in Figure 1. To address this challenge, we propose a novel syntax encoding structure – confidencebased syntax encoding network (cSEN), which alleviates the negative effect of noise by measuring confidence of arcs in syntax graphs. Specifically, confidence is calculated by performing Biaffine transformation over the sequence representation and the derived syntactic graph adjacency matrix. With this obtained confidence, our model is capable of distinguishing useful syntactic features from noise.
Moreover, compared with modern Chinese, ancient Chinese has more concise expressions and thus more compact structures, each token is highly relative to the preceding and following one. Considering such linguistic characteristic, we incorporate another graph feature – left-right branch (LRB), which captures local features to further improve ancient Chinese understanding. Experiments are conducted on two typical ancient Chinese understanding tasks, thematic classification of ancient poetry and ancient-modern Chinese translation. Results show that our model achieves significant improvements over powerful baselines, and our proposed cSEN can effectively handle the noise in the derived syntax trees. To our best knowledge, our proposed cSEN is the first solution that makes the syntax practical in ancient Chinese processing. The proposed cSEB can serve as a backbone for enriching our understanding of ancient texts, offering a scalable and consistent solution for education, research, and broadening the public’s access to these significant cultural treasures.
Overall, the contributions of this paper can be concluded in four folds:
• This study fills the research gap of exploring the role of syntax in ancient Chinese understanding. Our work demonstrates that syntactic information, even noisy parses from unsupervised derivation, can benefit ancient Chinese understanding substantially.
• We propose a novel architecture – confidencebased syntax encoding network (cSEN), which alleviates the negative effect of noise in syntax parses, thus making it practical to utilize derived syntactic information to enhance ancient Chinese understanding.
• The effectiveness of cSEN is evaluated on two typical ancient Chinese understanding
tasks, ancient poetry thematic classification and ancient-modern Chinese translation. Results show that our model yields significantly better performance in noisy scenarios over powerful baselines.
• We create a new dataset for the thematic classification of ancient Chinese poetry, with 22,360 poems divided into 10 theme categories. This dataset offers a data foundation for related research and helps to eliminate the lack of available ancient Chinese annotated corpora.

heading: 2 Related Work

heading: 2.1 Syntax Role in Modern Chinese Understanding
text: As syntax is highly correlated with semantics, syntactic features, including constituent and dependency structures, have been utilized in many modern Chinese understanding tasks and have been shown to be helpful clues. Li et al. (2018) explored the effect of syntax on semantic role labeling (SRL) and confirmed that high-quality syntactic parsing can effectively enhance syntactically-driven SRL. Xia et al. (2019) designed a syntax-aware multi-task learning framework for Chinese SRL by extracting implicit syntactic representations as external inputs for the SRL model. Jiang et al. (2018) incorporated syntactic features to expand identified triplets for improving Chinese entity relation extraction. Zhang et al. (2020) proposed a syntax-aware approach for solving machine reading comprehension, which incorporates explicit syntactic constraints into the attention mechanism for better linguistically motivated word representations. Sun et al. (2022) utilized syntactic features, which capture depth-level structure information, including non-consecutive words and their relations, to enhance recognition of Chinese implicit intersentence relations. Zhu et al. (2022) incorporated syntactic dependency information to determine entity boundaries for improving Chinese named entity recognition. Despite the increasing attention that syntax is receiving in modern Chinese understanding, few studies have attempted to utilize syntactic features for ancient Chinese understanding.

heading: 2.2 Ancient-Modern Chinese Translation
text: Unlike bilingual translation tasks, such as ChineseEnglish, ancient and modern Chinese are written
using the same characters. Despite that, translating between ancient and modern Chinese can still be challenging for native speakers. This is due to two factors: (1) the syntactical structure and grammatical order of ancient Chinese are different from those of modern Chinese, making ancient Chinese expressions more concise yet also more confusing; (2) ancient Chinese frequently employs allusion, metaphor, and symbolic imagery to implicitly evoke sensory and emotional experiences, which increases the complexity of disambiguating the intended message.
In recent years, advancements in deep learning have led to significant progress in neural machine translation. For example, Zhang et al. (2019b) proposed an unsupervised algorithm that constructs sentence-aligned ancient-modern pairs, and an endto-end neural model with copying mechanism and local attention to translate between ancient and modern Chinese. Liu et al. (2019) applied RNNbased (Bahdanau et al., 2014) and Transformerbased (Vaswani et al., 2017) machine translation models to this task. Considering the monolingual nature of this task, Yang et al. (2021) utilized pretrained model UNILM (Dong et al., 2019) and an ancient Chinese pre-trained model Guwen-BERT to enhance performance. Over time, the Chinese language has evolved a lot, resulting in different characteristics of ancient Chinese in different eras. To address this, Chang et al. (2021) proposed a time-aware translation method, where the model predicts both the translation results and its particular era, and uses the predicted chronological feature as auxiliary information to bridge the linguistic gap between Chinese language in different eras.

heading: 2.3 Classification of Ancient Chinese Poetry
text: Classification of ancient Chinese poetry provides a basis for higher-level tasks, such as sentiment or style controllable poetry generation (Yang et al., 2018; Chen et al., 2019; Shao et al., 2021). In the past, statistical features and machine learning algorithms were commonly used. For example, Hou and Frank (2015) proposed a weakly supervised sentiment classification approach, which created a sentiment lexicon based on Weighted Personalized PageRank (WPPR). Shen et al. (2019) incorporated imagery features for analyzing the sentiment of Tang Poetry. In recent years, neural classifiers have been introduced to the task and made remarkable progress in performance. For instance, Xuan
et al. (2018) designed a poetry style recognition model by stacking a genetic algorithm with CNN, and Tang et al. (2020) combined CNN with a gated GRU for solving poetry sentiment classification.

heading: 3 Model
text: In this section, we describe architecture of the proposed cSEN. We first present a basic GAT encoder, then introduce our cSEN. The overview of cSEN is shown in Figure 2.

heading: 3.1 Vanilla GAT
text: GAT is often applied over a sentence encoder to extract graph-based representations of the input text. Given input token sequence T = {t1, t2, . . . , tl}, l denotes the sequence length. The output of the sentence encoder is denoted as matrix H ∈ Rl×n, where each row hi ∈ Rn is the representation of token ti.
With dependency structure of the input sequence from a syntax parser, we construct a dependency graph G = (V, E), where V is the set of tokens and E is the set of arcs. In the graph encoding, we employ the form of adjacency matrix to describe the graph, in which the positions with arcs and diagonal are assigned to ones, denoted as M(dep). Linear transformation is performed by multiplying the sentence representation H with a matrix W ∈ Rn×n′ for feature extraction, where n′ denotes the transformed feature dimension: Z = HW. Then, a pair-wise attention operation is performed. For every pair ti, tj ∈ V , it concatenates corresponding representations zi and zj , then takes the dot product with vector a ∈ R2n′ and applies a LeakyReLU activation function:
S(raw)[i, j] = LeakyReLU([zi ⊕ zj ]T a), where ⊕ represents the concatenation operation, and S(raw) is a score matrix with the size of (l × l) that captures inter-node relations. To integrate the graph structure, the adjacency matrix M(dep) is used to constrain the function scope before a regular Softmax operation is performed. By doing this, each token can only attend to its head tokens and itself. The obtained attention weights matrix then is used for scaling the transformed sentence representation Z and calculating the final attentional output:
W(attn) = Softmax(S(raw) ×M(dep)).
H(attn) = W(attn)Z.

heading: 3.2 Confidence-based GAT
text: As discussed above, GAT guides the encoding process by constraining the scope of the attention computation. Therefore, the presence of noise in the graph will inevitably impact the encoding output. To alleviate the negative effects of noise on the model’s performance, we propose a confidencebased GAT, which measures the confidence of the graph adjacency matrix, helping the model distinguish reliable syntactic information from noise.
Similar to vanilla GAT, we first model the pairwise relationships. Two separate linear transformations are performed over the sentence representation H to obtain the role-aware representations. The outputs are denoted as H(d) and H(h) respectively, both of which have the size of (l × n′): H(d) = HW(d);H(h) = HW(h). Then, Biaffine attention (Dozat and Manning, 2016) are calculated on the role-aware representations for pair-wise relationship scoring: S(bi) = H(d)UH(h)T , where U is an intermediate matrix with the size of (n′ × n′). Confidence scores are calculated by concatenating the pair-wise relationship scores and the adjacency matrix and passing them through processing as follows, S(fuse) = ReLU(FFNN(fuse)( [ S(bi) ⊕M(dep) ] )),
S(conf) = Sigmoid(FFNN(proj)(S(fuse))). where FFNN(fuse) performs a linear transformation
to fuse the two feature spaces along with an ReLU activation, and FFNN(proj) is used to reduce the dimension from 2l to l, so that Sigmoid can be applied to project the confidence features to the same magnitude as the attention scores. With this obtained confidence scores S(conf), we can remedy the original attention restrain process:
W(conf) = Softmax(W(attn) + S(conf)), H(conf) = W(conf)Z.
In summary, cSEN alleviates the negative effect of noise in graphs through a two-fold process. First, cSEN measures the confidence of the derived syntax parses. This confidence score is then used to soft-mask noisy arcs and highlight previously undetected ones. Second, considering the linguistic characteristics of ancient Chinese, the Left-Right Branch feature is incorporated to broaden the scope of syntax graph encoding and smooth out noise and incompatibility. The combined effect of these aspects helps alleviate performance degradation caused by noise.

heading: 3.3 Left-Right Branch Feature
text: Inspired by the ubiquity of local dependencies in ancient Chinese, we introduce a novel straightforward and effective feature, left-right branch, to further improve the GAT encoding. To model local inter-token relations, we populate a matrix M(lrb) of the same size as M(dep) following
M(lrb)[i, j] = {
1, if j ∈ {i− 1, i+ 1} 0, otherwise.
This indicates that there exist arcs in the graph connecting the node and its close left and right neighbors. The left-right branch features are encoded using another GAT component, yielding a sequence representation Z(lrb) and a positional-informationintroduced attention weight matrix W(lrb). The outputs from M(dep) and M(lrb) are combined with a gated mechanism to produce the final output:
H(lrb) = W(lrb)Z(lrb). g = Sigmoid(FFNN(gate)( [ H(conf) ⊕H(lrb) ] )),
H(output) = g ×H(conf) + (1− g)×H(lrb).

heading: 4 Experiments
text: We evaluate the effectiveness of cSEN module using two typical ancient Chinese understanding tasks: Thematic classification of ancient poetry and ancient-modern Chinese translation. We build our model by incorporating the cSEN module into existing solid baselines. For the classification task, we follow the work of (Vaibhav et al., 2019) which has a BERT-GAT-BiLSTM backbone architecture. And for the translation task, our model is based on (Jin et al., 2020) where dependency graphs are incorporated into neural sequence-to-sequence models with a pointer network.

heading: 4.1 Data
text: To address the scarcity of annotated data for thematic classification, we constructed a novel dataset1. Two graduate students specializing in Chinese literature study annotated 22,360 poems, categorizing them into one of ten distinct themes under the guidance of an experienced ancient Chinese linguist. This meticulous process ensured high-quality, reliable annotations. Any conflicted labelling between the two annotators was resolved through consultation with the supervisor, guaranteeing a consistent annotation standard. The dataset is then randomly divided into a training set (20,360), a development set (800), and a test set (1,200). The distribution of themes in the dataset is detailed in Table 1.
For the ancient-modern Chinese translation, we adopt the ancient-modern Chinese parallel corpus contributed by the open source NiuTrans project2. The corpus contains 967,255 sentence pairs extracted from ancient Chinese books. We divided
1Upon publication of this paper, this dataset will be made available for research purposes.
2https://github.com/NiuTrans/Classical-Modern
the corpus into training, validation, and test sets with corresponding sizes of 900,000, 60,000, and 7,255.

heading: 4.2 Syntax Parsing
text: We experiment with two settings – modern supervised parsers and ancient unsupervised syntax derivation. For modern supervised parsing, we adopt the Biaffine dependency parse (Dozat and Manning, 2016) and train it on CTB7 (Xue et al., 2010). For unsupervised syntax derivation, we follow the work of Wu et al. (2020), which utilizes linguistic knowledge gained from pre-trained language model BERT to infer syntactic dependency structure without direct supervison. We attempt two variants of BERT for syntax derivation and backbone sentence encoder, BERT-wwm-ext (Cui et al., 2021) and Anchi-BERT (Tian et al., 2021). BERT-wwm-ext is trained on the modern Chinese corpus containing 5.4B words, while Anchi-BERT is trained upon a ancient Chinese corpus with the size of 39.5M tokens. In addition, we treat the left-right branch as a special kind of syntax parses. Anchi-BERT is trained on a smaller ancient Chinese corpus (39.5M tokens), while BERT-wwm-ext is trained on a larger modern Chinese corpus (5.4B tokens). We also treat left-right branch features as a distinct class of syntax parses.
For clarity, the syntactic parses from the Biaffine parser, BERT-wwm derivation, and Anchi-BERT derivation are denoted as BiAF, WWMD, ANCD respectively, in the following part.

heading: 4.3 Implementation and Hyper-parameters
text: For the thematic classification, our model is built by stacking BERT, a graph encoder, and a single-layer LSTM. For the baseline, we do not incorporate syn-
tax parses, rendering the graph encoder ineffective in shaping the attention scope. The graph encoder’s node embedding dimension is set to 128, and the hidden size in LSTM is set to 100. We adopt the Adam optimizer with ρ = 5e− 5 and = 1e− 8, using a batch size of 32. All classifiers are trained for 10 epochs on the train set by default.
We mostly follow the parameter settings from (Jin et al., 2020) for the ancient-modern Chinese translation. The Adam optimizer is configured with ρ = 1e − 4 and = 1e − 8. And all models are trained for 50 epochs with a batch size of 108.

heading: 4.4 Results

heading: 4.4.1 Ancient Poetry Thematic Classification
text: Table 2 presents the results of ancient poetry thematic classification. We report the results in MicroF1 and Macro-F1 scores. The table is divided into three blocks, showing the results of the baseline model, vanilla GAT, and the proposed cSEN. The baseline model achieves 92.4 in Micro F1 and 90.4 in Macro F1, showing strong performance.
From the results in the first two blocks, it can be found that incorporating syntactic trees with GAT encoder brings substantial improvement, proving
the value of syntactic information for enhancing ancient Chinese understanding. Through comparing the results of employing Anchi-Bert as the sentence encoder and those obtained employing Bert-wwm, we can see that Anchi-Bert outperforms BERTwwm with a significant lead in all cases. Recall that Anchi-Bert was pre-trained on a much smaller corpus. Also, the performance of syntactic trees derived by BERT-wwm is inferior to the other three. This once more indicates the linguistic gap and syntactic incompatibility between ancient and modern Chinese.
Unsupervised syntax trees derived by AnchiBERT performs roughly the same as those produced by the Biaffine parser. Additionally, LRB is the best-performing syntax parse among all, improving the performance by 0.9 in Micro F1 and 1.0 in Macro F1. It can be partially explained by the fact that ancient poems are comprised by a few brief sentences, which are highly concise and structurally compact. This results in fewer long-range dependencies, and each token is closely dependent on the immediate preceding or succeeding token.
From the third block, it can be seen that when using Anchi-BERT as sentence encoder, cSEN brings
performance gains across all syntax trees setups, raising the top Micro and Macro F1 scores to 93.8 and 91.9, respectively. This demonstrates that: (1) cSEN’s denoising capability is effective for utilizing noisy syntactic information to improve ancient Chinese understanding; (2) cSEN can handle noise introduced by different parses, whether it is from a supervised modern Chinese parser or unsupervised derivation.

heading: 4.4.2 Ancient-Modern Chinese Translation
text: Results of the ancient-modern Chinese translation are shown in Table 3. We use BLEU (Papineni et al., 2002) and ROUGE (Lin, 2004) scores for performance evaluation. The baseline model without syntax parses achieves 37.14 in BLEU score and F-scores of 69.71, 46.24, 67.62 in ROUGE1, ROUGE-2, and ROUGE-L respectively. With single syntactic parses incorporated, all models achieve better performance in all metrics, proving that syntax can effectively improve ancientmodern Chinese translation. LRB is relatively the weakest one, slightly increasing BLEU score by 0.28, and ROUGE f-scores by 0.15, 0.12, 0.10. This might be caused by that sentences from the ancient books have more long-distance dependencies and more complicated syntactic structures that left right branch can not recover. Anchi-BERT derived syntax parses have better performance with an improvement of 0.41 in BLEU score, and 0.19, 0.29, and 0.23 in ROUGE scores. BERT-wwm derived syntax trees and trees generated by Biaffine parser have similar results. In contrast to AnchiBERT derived trees, their performance are inferior
in BLEU scores but better in ROUGE F-scores. Feeding multiple syntactic parses into the GATbased model simultaneously leads to a significant performance drop. While replacing GAT with the proposed cSEN increases performance in all metrices, with 37.73 in BLEU score and 70.27, 47.09, 68.23 in ROUGE F-scores. From the above results, we conclude that syntax parses from unsupervised derivation or modern Chinese syntax parsers introduce noise and degrade model performance. With our confidence learning, model is able to distinguish and separate informative syntactic information from noise, thus alleviating its negative effect.
Table 6 shows three ancient-to-modern Chinese translation examples produced by different models. From generations for Sent 1, we can see a common error: due to the lack of contextual information, all three models assume the surname of "the father" useing the most common Chinese surnames, such as "Li" and "Zhang". For Sent 2, the generations from the baseline model and vanilla GAT differ significantly from the human-annotated reference. They fail to recognize the relationship between the characters,such as who "其娣" refers to, thus generating tranlations that did not correspond to the facts. In contrast, with stronger denoising capability, cSEN is able to correctly encodes the information in ancient Chinese texts, thus producing higher-quality translations.

heading: 5 Exploration
text: In this section, we investigate the impact of different cSEN components and analyze the nature of different syntax parses.
First, we conduct ablation studies on cSEN. Results are reported in Table 4. Both the removal of the confidence (equivalent to vanilla GAT) and the removal of the gated mechanism lead to significant performance degradation. This indicates that both modules are crucial for effectively utilizing syntactic information from noisy parses.
Second, we experiment with different syntax tree combination setups. Table 5 lists the findings. Combining two syntax parsers onto a single dependency graph can provide richer syntactic information and produce higher performance than using alone syntax parses, no matter it is from unsupervised derivation or a supervised parser. We also explore the incorporation position of LRB features. It can be observed that the model performance suffers significantly if LRB features and graph parses
are directly merged together. This again indicates the necessity of our gated method for LRB feature integration.
Third, as illustrated in Figure Figure 3, we compare our model and baselines over different input lengths. cSEN performs better in relative longer sentences, according to the results. This supports the hypothesis that syntax helps guide longer sentence understanding as dependency reduces the distance. Because of the incompatibility between modern and ancient Chinese, unsupervised derivation is more effective than supervised parsing when compared to other syntax parsers. In most cases, cSEN yeilding better performance due to its stronger denoising capabilities.

heading: 6 Conclusions
text: In this paper, we investigate the role of syntax in improving ancient Chinese understanding. Due to lack of syntax annotation, syntax trees are obtained by unsupervised derivation and supervised modern Chinese parser. To alleviate the negative effect of noise, we propose a confidence-based syntax encoding network (cSEN). Experimental results on
two typical ancient Chinese understanding tasks show that our model can effectively distinguish informative syntactic information from noise and achieve better performance. The application of our proposed cSEN can enhance the accessibility of ancient Chinese resources by offering a scalable and consistent solution for mining semantic information of ancient Chinese texts.
Limitations
The main limitation of our study comes from the extra parameters caused by confidence calculation, in which two separate self-attention operations and Biaffine transformation are performed. Incremental parameters results in a more time-consuming training process, and a higher hardware demand for storage. To address this issue, we plan to combine parameters from different attentional transformations into shared weight matrices in our future work to reduce the model size.

heading: Acknowledgements
text: This paper was partially supported by the National Natural Science Foundation of China [No. 72074171].

heading: Future Work
text: To address this issue, we plan to combine parameters from different attentional transformations into shared weight matrices in our future work to reduce the model size