WYWEB: A NLP Evaluation Benchmark For Classical Chinese
Year: 2023

Abstract:
To fully evaluate the overall performance of different NLP models in a given domain, many evaluation benchmarks are proposed, such as GLUE, SuperGLUE and CLUE. The field of natural language understanding has traditionally focused on benchmarks for various tasks in languages such as Chinese, English, and multilingual, however, there has been a lack of attention given to the area of classical Chinese, also known as "wen yan wen (文言文)", which has a rich history spanning thousands of years and holds significant cultural and academic value. For the prosperity of the NLP community, in this paper, we introduce the WYWEB evaluation benchmark, which consists of nine NLP tasks in classical Chinese, implementing sentence classification, sequence labeling, reading comprehension, and machine translation. We evaluate the existing pre-trained language models, which are all struggling with this benchmark. We also introduce a number of supplementary datasets and additional tools to help facilitate further progress on classical Chinese NLU. The github repository is https://github.com/baudzhou/WYWEB.

heading: 1 Introduction
text: Classical Chinese, as a written form of the Chinese language, had been widely used in the Confucian cultural circle, including China, Japan, Korea, Vietnam, etc (Ye and Tian, 2013; Phong1 and Van2, 2020; Xu, 1995; Zhou, 2009; Jin, 2004). As far as we know, there are about 400 million words, 3 million ancient articles have been passed down, covering literature, art, history, philosophy, etc, half of which are of great value (Yin et al., 2018). However, in recent centuries, the use of classical Chinese has been gradually phased out and replaced by modern languages, resulting in increasing difficulty in comprehending it. Therefore, it is necessary to
∗Corresponding Author: Yin Zhang.
introduce efficient NLP technology to process, understand, and research such literature.
While pre-trained language models such as BERT (Devlin et al., 2019) and BERT-like models (Yang et al., 2019; Dong et al., 2019; Lan et al., 2019; Liu et al., 2019; He et al., 2020; Raffel et al., 2019; Wang et al., 2019c) have shown remarkable performance on English NLP benchmarks, including GLUE (Wang et al., 2019b) and SuperGLUE (Wang et al., 2019a), there are also many efforts (Cui et al., 2020a; Wei et al., 2019; Cui et al., 2021a) in Chinese NLP community, achieving significant improvement on modern Chinese NLP benchmark (Xu et al., 2020; Cui et al., 2018; Duan et al., 2019; Cui et al., 2020b).
However, since classical Chinese differs from modern Chinese in writing and grammar, these benchmarks can not be applied well to the studies in the classical Chinese domain. In order to better adapt to the understanding of classical Chinese, many tasks and datasets are required to be redesigned, such as sequence labeling and sentence pair similarity. Meanwhile, due to the performance of the model being closely related to the pre-training corpus (Qiu et al., 2020), such as scale, language, domain, etc., the existing language models pre-trained on modern Chinese corpus can not adapt well to classical Chinese tasks.
Considering that previous studies (Wang et al., 2021; Yang et al., 2021; Koichi et al., 2022) for classical Chinese have typically evaluated models on few or different NLU tasks, it is difficult to compare the performances of these models. To facilitate such research in classical Chinese, it is necessary to design a standard classical Chinese NLP evaluation benchmark.
In this paper, we introduce WYWEB (Wen Yan Wen Evaluation Benchmark), which will be open, and continually developed as much as we can. To evaluate the performance of the models of classical Chinese language representation, we create and
3294
refine nine tasks for different aspects of language understanding.
Specifically, considering the importance of breaks and pauses in sentences for the comprehension of classical Chinese, for sequence labeling, we design two novel tasks, including punctuation PUNC and named entity recognition GLNER, to evaluate word separation capability of pre-trained language models. For sentence classification, we design three novel tasks, including text category classification GJC, written time classification TLC, and emotion classification of poems task FSPC. For reading comprehension, we create a multiple choice task named WYWRC. And on the other hand, the assessment of the natural language comprehension capability of a model through previous reading comprehension tasks is challenged by the extensive use of rare vocabulary and idiomatic expressions in classical Chinese. Therefore, we design a novel reading comprehension task, IRC, from the exam papers and idiom dictionary. Meanwhile, since machine translation of classical Chinese is also a problem of great concern, we design a novel WYWMT task to study this topic. In addition, considering that some tokens in the classic Chinese language have the functions of prepositions, conjunctions and auxiliaries, and the same token has different meanings in different sentences, we design a new task, Xuci, for token comparison. More details of these tasks are described in Section 4 and Appendix A. And we describe the principles we used to design tasks and the process of data collection in Section 3.
Furthermore, to better understand the challenges provided by WYWEB, we build a baseline for each task and evaluate several pre-trained models released by the community. The experimental results demonstrate that current state-of-the-art methods are struggling with these tasks, which suggests that those tasks in WYWEB can constitute a useful testbed for developing and comparing NLP systems for classical Chinese.
The contributions of our work are summarized as follows:
• We propose and establish a novel benchmark for classical Chinese natural language understanding via redesigning, creating and collecting nine classical Chinese NLP tasks.
• To validate the challenge of this benchmark on existing pre-trained model models, we con-
duct a series of experiments with several baselines. Experimental results demonstrate these baselines are struggling with these novel tasks in classical Chinese.
• Finally, we build an online leaderboard and provide an evaluation tool set for further exploration, which will be publicly accessible as soon as possible.

heading: 2 Related Work

heading: 2.1 Benchmarks for Pre-trained Language Model
text: With the rise of the pre-training language model, pre-training a model on large corpus and finetuning them on downstream tasks becomes a general practice in the NLP community. To evaluate the ability of pre-trained language models in NLP tasks, several benchmarks are proposed for NLU tasks, such as SentEval (Conneau and Kiela, 2018), GLUE (Wang et al., 2019b) and SuperGLUE (Wang et al., 2019a), making existing models more comparable. For Chinese NLU, CLUE (Xu et al., 2020) benchmark is proposed with more than 10 tasks, including most NLP problems. To evaluate the ability of pre-trained language models in both natural language understanding and generation, CUGE (Yao et al., 2021) is proposed, which is designed as a hierarchical framework via a multilevel scoring strategy. Meanwhile, to evaluate whether language models can learn a linguistic phenomenon of Chinese, Xiang et al. (2021) develops CLiMP which covers 9 major Mandarin linguistic phenomena. QuoteR (Qi et al., 2022) is designed for the evaluation of quote recommendation methods. Moreover, CBLUE (Zhang et al., 2021) is a biomedical language understanding benchmark for Chinese, which mainly focuses on information extraction.
However, compared to modern Chinese, there has been a lack of sufficient datasets and benchmarks for classical Chinese. CCLUE 1 provides 5 NLU tasks for classical Chinese, including punctuation, NER, classification, sentiment recognition and retrieval between classical and modern Chinese. Unfortunately, this project was not finished and is no longer maintained. Therefore, a new carefully designed benchmark for classical Chinese is very crucial for current studies.
1https://cclue.top/

heading: 2.2 Corpus Datasets for Classical Chinese
text: The largest classical corpus dataset available is Daizhige (殆知阁)2, which contains about 3.3 billion tokens of classical Chinese literature, making classical Chinese corpus not low-resource. Most of pre-training related works use this dataset for model training. Ancient Chinese Corpus (ACC) 3 dataset contains the word segmented, POS-tagged data of Zuozhuan (an ancient Chinese history classical book). This dataset is widely used in ancient Chinese studies. Recently, Zinin and Xu (2020) introduces an open source corpus of Twenty-Four Histories and some other ancient books. Meanwhile, FSPC (Shao et al., 2021) and CCMP (Li et al., 2021) are proposed for ancient poem understanding. While CUGE (Yao et al., 2021) uses CCMP as a sub-task for classical poetry matching, in this work, we apply the FSPC dataset for poetry emotion recognition.

heading: 2.3 Pre-trained Models for Classical Chinese
text: In Classical Chinese pre-trained language models, SikuBERT and SikuRoBERTa (Wang et al., 2021) are pre-trained BERT/RoBERTa model on the Si Ku Quan Shu (Complete library in the Four Branches of Literature) corpus, and evaluated on 4 tasks, which are built from ACC dataset. Meanwhile, based on RoBERTa model, GuwenBERT 4 is pre-trained on Daizhige corpus with continuous training method and is evaluated on several NLU tasks. Other works (Hu Renfen, 2021; Yu et al., 2021; Yang et al., 2021) also evaluate their models on different few NLP tasks.However, since these models are not consistent in their evaluation tasks,
2https://github.com/garychowcmu/daizhigev20 3https://catalog.ldc.upenn.edu/docs/LDC2017T14/ 4https://github.com/ethan-yt/guwenbert
to compare the natural language understanding ability of different pre-trained models in classical Chinese, a standard evaluation benchmark is required.

heading: 3 WYWEB Overview
text: In this section, we introduce the principles and methods we applied during the construction process of WYWEB, and describe a brief overview of tasks in Table 1. Firstly, we describe the process of task design and the principles we follow. Then, we introduce the data selection principles in Section 3.2. Finally, we provide a description of the leaderboard and evaluation toolkit.

heading: 3.1 Task Design Principles
text: In this work, to assure that the benchmark can evaluate most aspects of pre-trained models and language phenomenons, we design evaluation tasks following best practices of other NLP benchmarks (Xu et al., 2020; Yao et al., 2021; Wang et al., 2019a,b) and suggestions from experts.
Following the principles of Xu et al. (2020), firstly, these tasks should vary in most aspects of NLP, including text classification, reading comprehension and machine translation, etc.
Secondly, these tasks should be well-defined in the academic community and easily processed for corpus collection.
Thirdly, they should be challenging but solvable. Finally, these tasks should be useful for followup studies and representative of classical Chinese natural language understanding tasks.
With the study of thousands of Chinese exam papers and requirements from academia and applications, we construct the evaluation tasks, covering most of the regular NLP tasks.
In addition to the regular tasks, we designed
several tasks specifically for classical Chinese, i.e., punctuation of sentences without punctuation marks, comparison of confusing words and written period classification. These tasks will be introduced in Section 4 and Appendix A.

heading: 3.2 Corpora Selection
text: Diversity Over Time Since classical Chinese has a very long history and evolves over time, when designing tasks, we should choose texts that cover as many periods as possible. It is supposed that it is not reasonable enough to treat an isolated article as an independent task.
Diversity Over Style The stylistic theory is an important part of Chinese traditional literary theory. Since there are great differences between different styles, such as prose, parallel prose, poetry and so on, we believe that the benchmark should cover as many styles as possible.
For instance, Wang et al. (2021) evaluate their model on ACC corpus which is built on Zuo Zhuan. However, Zuo Zhuan was written by Zuo Qiuming in East Zhou Dynasty, so the text features of Zuo Zhuan are relatively simple and unable to test the models for a variety of linguistic phenomenons of classical Chinese. Therefore, in this work, we refine datasets like this and combine them into welldefined datasets to build uniform sample sets.
To evaluate an NLU ability for classical Chinese, it is natural to handle classical Chinese text as it is. However, dealing with raw classical Chinese text without sentence segmentation is extremely difficult, which leads to the task becoming unsolvable. Therefore, besides the PUNC task itself handling sentence segmentation of classical Chinese text, segmented texts are adopted for other tasks.
When selecting candidate corpora, we apply rules such as (1) having refined punctuation marks; (2) having more than 4 words in classification tasks; (3) being originally simplified Chinese character style preferred.

heading: 3.3 Data Collection and Ethical Concerns
text: We collect data from as many channels as possible, i.e, open source projects, public websites, competition data and education institutions. Since classical Chinese sources are all works from many years ago, people could use the corpora for free. For other texts, if the copyright issue is concerned, we have been granted to use the data in this work. For example, GLNER is data from a competition for
classical Chinese, we contact the owner by email and finally get licensed.
Classical Chinese was officially and commonly used as a written language before recent times in East Asia, but now, modern languages have taken its place in these countries. In China, people learn classical Chinese at school but rarely use it in everyday life except for some poems and idioms. We could hardly collect any publicly available NLP datasets compared to modern Chinese. As a result, we design and create most of WYWEB datasets by ourselves.

heading: 3.4 Annotation
text: Annotation Process For different tasks, we apply different annotation processes. For PUNC, TLC and GJC tasks, the process is "data collecting, extracting, proofreading and final review". Specifically, since the data source is very important, these tasks are created using high quality and authoritative documents. Then paragraphs are sampled from every document with a certain small proportion, avoiding information leakage as much as possible. The annotators double-check every sample to ensure correctness of the computer work. Finally, domain experts review the whole work to get the result dataset.
For other tasks extracted from examination papers, the process is different because they need much more human workload. Given the papers (part of them are in PDF format or images), annotators copy or type the questions and do proofreading to assure the quality. Some new samples are also created by annotators to enrich the task. After the annotation work, the datasets are also sent to domain experts to carry out a final review.
Annotators This is a community-based project, and most of the annotators and reviewers are volunteer students and scholars who are interested in classical Chinese or natural language processing. The authors take on the remaining annotation tasks.
It takes us a long time to create this benchmark, so the annotators vary during the process. When selecting annotators, we make sure they have got a good language score in national college entrance examination and are familiar with classical Chinese. Some rules annotators following are: (1) dropping out confusing sentences; (2) double-checking rarely used words and dropping out sentences with uncertain rarely used words; (3) removing unnecessary symbols except specified punctuation marks.
To ensure the quality of the datasets, we asked some domain experts to do the final review.
For instance, Buddhist scriptures in classical Chinese are kind of important documents. As Buddhist scriptures are generally collated and proofread by Buddhist believers, the correctness of these texts is relatively reliable. We apply many Buddhist scripture texts in our tasks, hence the need for a review of scholars from the Buddhist Academy.
Quality checks for WYWMT Translating classical Chinese sentences to modern Chinese is challenging. We follow Guzmán et al. (2019) to filter texts collected from the internet. In addition, because classical Chinese sentences are usually short, the limitation of sample length is set to 5 to 200 characters.

heading: 3.5 Toolkit and Leaderboard
text: For the evaluation toolkit, we provide scripts implemented using PyTorch (Paszke et al., 2019) and transformers (Wolf et al., 2020), which can help the followers evaluate their models easily. Otherwise, they can upload their models to Hugging Face Model Hub 5 and contact us for the evaluated results. This toolkit is also released on WYWEB repository. Furthermore, we provide a leaderboard for the community to present the performance of each model. The leaderboard includes a general list and a sub list of each task. The results will be updated soon after the submission of models. As shown in Section E, we provide some details and screen-shots of the leaderboard.

heading: 4 Tasks
text: In this section, we describe tasks and datasets designed for specific aspects of classical Chinese NLP. These datasets, except GLNER and FSPC, are firstly created by ourselves. More details are shown in Appendix A.

heading: 4.1 Sequence Classification Tasks
text: GJC This task aims to work on the problem of ancient book classification which has been discussed since ancient times. We select a proportion of text from each category of the Daizhige project and divide them into the training set and evaluation set, where each sample is a selected paragraph from an article or book, ranging from a few dozen to hundreds of characters in length. More details are shown in Appendix A.5.
5https://huggingface.co/
TLC This task is to identify the written time of ancient books. We create the TLC dataset where each sample has a coarse-grained label (period) and a fine-grained label (Dynasty) forming a hierarchical structure. Similar to GJC task, each sample is a paragraph selected from ancient literature. More details are shown in Appendix A.4.
FSPC FSPC (Fine-grained Sentiment Poetry Corpus) is an emotion recognition task for ancient rhythmic poetry, created by THUAIPoet (九歌) group (Shao et al., 2021). Sentiments are annotated into 5 classes, i.e. negative, implicit negative, neutral, implicit positive, and positive. THUAIPoet designs a reasonable annotation mechanism to ensure annotations follow similar standards during the work process. See Appendix A.6 for details.

heading: 4.2 Sequence Labeling Tasks
text: PUNC This task is designed to perform text segmentation, i.e., adding punctuation marks to continuous ancient Chinese texts. The dataset uses some fairly authoritative texts, including historical records and Buddhist scriptures, to construct the samples. In order to reduce the complexity of the research, only a few commonly used punctuation marks are used. Each sample consists of the original text and its corresponding label sequence. Furthermore, the samples are paragraphs containing several sentences. See Appendix A.2 for details.
GLNER GLNER is a named entity recognition task created by GULIAN (2020). Texts of the dataset are selected from ancient books and some other relevant literature. There are two kinds of entities in this dataset, i.e., classical book name and other which includes human name, location name, etc. Since the entity category is of coarse grain size, it is expected to implement new labeling work to refine this dataset in the future. See Appendix A.3 for details.

heading: 4.3 Sentence Pair Tasks
text: XuCi This task is designed to determine whether two function words in a sentence pair have the same meaning and usage. The words to be compared in the samples generally consist of one or two single characters. Each sample includes fields such as the pair of sentences, the position of the function words in the sentence, and a label indicating whether they are the same or not (True or False). See Appendix A.7 for details.

heading: 4.4 Reading Comprehension Tasks
text: WYWRC Similar to the RACE dataset (Lai et al., 2017; Sun et al., 2019), this task involves providing a classical Chinese paragraph, a question, and selecting the best answer from among four options. This task is quite challenging, as the model must possess a proficient understanding of both modern and classical Chinese. For analysis purpose, we separate the samples into 10 types according to their questions and answers. More details are shown in Appendix A.1.
IRC Considering idiom comprehension is a very important part of classical Chinese learning, to evaluate the idiom comprehension ability of the model, we design and collect the IRC dataset. In IRC, given an idiom and its origin (most are in classical Chinese), the model is required to select the best explanation from four options. See Appendix A.8 for details.

heading: 4.5 Sequence to Sequence Tasks
text: WYWMT Machine translation of classical Chinese is a problem of great concern. This task is used to evaluate whether pre-trained models can effectively improve the performance of machine translation models for classical Chinese. Due to the limited number of samples, we only use WYWMT dataset for evaluation rather than fine-tuning. Furthermore, we separate this task from others and make a stand-alone leaderboard. See Appendix A.9 for details.

heading: 5 Baselines

heading: 5.1 Baseline Implementation
text: Sequence Labeling We get hidden states from the last layer of the model encoder, and pass them to a classifier to get sequence labels. See Figure 1.
Sentence Classification We get pooled output of model encoder, i.e., hidden state of [CLS] token, and pass it to a classifier to get sequence labels. See Figure 2.
Reading Comprehension We encode each option concatenated with paragraph-question, pass the hidden states to a shared classifier to get prediction score and choose the best as final answer. See Figure 3.
Token Similarity Similar to sentence classification, we encode sentence pairs and get the hidden state of the corresponding token, then we use {u ; v; |u - v|} to represent the similarity score, where we mark the vectors as u and v. See Figure 4.
Machine Translation We implement this task as sentence pair with a prefix attention mask to adapt BERT-like models. To save inference time cost, the sequence output of the target sentence is greedy decoded. Note that this implementation is untypical for the sequence-to-sequence models, and is just used to reflect the capabilities of the model itself.
All the experiments are implemented using PyTorch (Paszke et al., 2019).

heading: 5.2 Pre-trained Models
text: GuwenBERT GuwenBERT has three versions, including GuwenBERT-base, GuwenBERT-large, GuwenBERT-fs-base. While GuwenBERT-base and GuwenBERT-large are trained based on RoBERTa-wwm-ext (Cui et al., 2021b), a modern Chinese pre-trained model, and then continue trained on classical Chinese corpus, GuwenBERTfs-base is trained purely on classical Chinese corpus.
RoBERTa-classical-chinese RoBERTaclassical-chinese has two versions, including RoBERTa-classical-chinese-base-char (RoBERTaCCBC), RoBERTa-classical-chinese-large-char (RoBERTa-CCLC). This is a RoBERTa model pre-trained on classical Chinese texts, derived from GuwenBERT-base. Character-embeddings are enhanced into traditional/simplified characters (Koichi et al., 2022).
SikuBERT, SikuRoBERTa These models are pre-trained on the verified high-quality “Siku Quanshu” (Wang et al., 2021). Note that these two models are pre-trained on traditional Chinese. In the fine-tuning stage, we convert simplified Chinese corpus into traditional Chinese.
DeBERTa-base Based on the structure of DeBERTa (He et al., 2020), we pre-trained the model on DaiZhiGe corpus from scratch.
RoBERTa-wwm-ext This model is trained with BERT (RoBERTa) structure (Cui et al., 2021b) and whole word masking.
Note that there are not as many pre-training models of classical Chinese as modern Chinese. We collect all models of classical Chinese accessible to evaluate and take them as baselines. More details of these models can be found in Appendix C.

heading: 5.3 Experiment Setting
text: For the evaluation, we fine-tune the pre-trained models mentioned above. For each task, we train 3 runs, and the model with the best development score is used for testing. When the learning rate decreases to a specified small value or the performance does not improve for 5 evaluations, the training is stopped. More details of hyper-parameters are shown in Appendix D.

heading: 5.4 Human Performance
text: For all tasks, we evaluate human performance following the principle of SuperGLUE: extract 30 samples in the training phase, and then sample 100 items from the test set in the testing phase. We collect test results from three annotators and calculate the human performance. The annotators are all college students majoring in ancient Chinese. The results of human performance are shown in Table 2 and Table 3. More details are shown in Appendix B.

heading: 5.5 Benchmark Results & Analysis
text: As shown in Table 2, we present the performance of existing baseline models in classical Chinese NLU tasks. Since evaluation metrics of sequenceto-sequence tasks are different from NLU tasks, as shown in Table 3, we evaluate each model on WYWMT task independently with several metrics, including BLEU, chrF2, TER and ROUGE.
From the results, it can be seen that some regular patterns, i.e. "the bigger (model scale and batch size), the better"; "the more (data and train steps), the better" appear as described in other experiments.
DeBERTa-base (He et al., 2020) performs best on this benchmark showing that the model structure and training strategy are both effective. Note
that this model is pre-trained just according to the default settings of DeBERTa V2 English version without the convolution layer and purely on classical Chinese. Meanwhile, some techniques that have obvious effects in Chinese are not used, such as Whole Word Masking (Cui et al., 2021b), etc. All models pre-trained on classical Chinese get better scores than chinese-roberta-wwm-ext (Cui et al., 2021b) pre-trained on modern Chinese corpus. Similarly, models trained on both classical Chinese and modern Chinese perform better on tasks involving both scripts, such as WYWRC, IRC and GLNER.
For FSPC task, composed of ancient Chinese rhythmic poems, SikuRoBERTa (Wang et al., 2021) performs the best, which is pre-trained with a high-quality classical Chinese corpus of Si Ku Quan Shu rather than on Daizhige.
Since poems are different from general texts, the models could learn a better representation of ancient words on Si Ku Quan Shu instead of on Daizhige. The two large models yield similar scores to DeBERTa-base but much better than other smaller ones, however, the parameter size of the large model is 3 times larger than that of DeBERTabase.
On WYWMT task, GuwenBERT-base achieves the best score with its pre-training strategy, which initializes the parameters of the transformer model from a pre-trained model and trains the model via freezing encoder layers to translate modern Chinese knowledge to classical and updates all parameters of the model. With the strategy, the model could learn a good representation of both modern and classical Chinese and achieve the best perfor-
mance on translation tasks. Compared with human performance, all the models have a big gap with the artificial results, especially on tasks WYWRC, XuCi, and IRC, which require a lot of implicit knowledge.
One limitation of our evaluation is, the models we collected are all BERT or RoBERTa style and are lacking some variety. Furthermore, the models we evaluated maybe not achieve the best score in this baseline due to differences among them. However, they are fine-tuned with similar hyperparameters, so that the results are comparable as expected.

heading: 5.6 Task Probing
text: According to scores in Table 2, we choose the most challenging task, WYWRC for further exploration and analysis. As shown in details of Table 4 in Appendix A.1, this task has a variety of questions, which is different from traditional machine reading comprehension tasks, and is more difficult for existing models.
Following previous work of Wang et al. (2022), we assess WYWRC dataset for the 4-dimensional MRC capabilities, which are word reading, sentence reading, word understanding and sentence understanding. To adapt to classical Chinese, we make some adjustments to the metrics and establish necessary dictionaries. The main adjustment is oriented towards readability, as Chinese and English have significant differences. Instead of the Flesch-Kincaid index, which could not be suitable for Chinese, we adopt the readability index for Chinese issued by (Yuying, 2020). The final evaluation results are shown in Figure 5. As when the capability-specific value score increases, the
difficulty of the problem that the model is understanding also increases and the correlation between sentence understanding and model performance is higher (Wang et al., 2022), we can see that the proportion of high scores in the WYWRC’s sentence understanding (v4) category is relatively large, thus poses a huge challenge to the pre-trained models.
In Table 5, we probe the accuracy of the test set to find that there are significant differences in scores between the different types of samples. Since the number of type 8 is small and it is a really hard question requiring extra knowledge, it is not very meaningful to discuss type 8.
It can be seen that Type 2 is the most difficult of the categories with a score of 17.9. In this type of question, given a list of characteristics or actions of the protagonist in a passage, the test-taker is required to select one option that either conforms to or does not conform to the given information. Some of the questions require the test-taker to have some reasoning ability.
Type 5 is the easiest category of problems to solve, with an average score of 66.5. This type of problem involves segmenting sentences with contextual information, similar to the PUNC task, with the difference being that the four options are confusing to each other. Nevertheless, because the problem is relatively simple and does not require much knowledge, it is easy for the models to handle.
Type 0 is the category with the largest number of samples, but its score is not high, indicating a considerable level of challenge. This type of problem involves several statements about an article, and the task is to identify the correct or incorrect option. Solving this type of problem requires not
only understanding the content of the article, but also sometimes additional knowledge support and even reasoning. Therefore, this type of problem not only requires a good pre-trained model, but may also requires better fine-tuning methods and the injection of more knowledge to achieve better accuracy.
All models have a significant gap compared to human scores, mainly due to the additional knowledge and reasoning abilities of the testers. This suggests that while current natural language processing models have made significant progress in understanding and handling language, there is still a long way to go before they can approach the level of human language comprehension and reasoning.

heading: 6 Conclusions and Future Work
text: In this paper, we introduce a NLP benchmark for classical Chinese, which contains nine NLP tasks and datasets respectively to help researchers to evaluate and analyze NLP models. Also, we build a toolkit for reproduction and a leaderboard online for the community.
The study of ancient Chinese is a highly specialized subject, so the professionalism of this benchmark may need to be further improved. On the other hand, there is a big gap between the performance of the classical Chinese models on this benchmark with other leader-boards. Better models are needed to handle more linguistic features of classical Chinese. Furthermore, to resolve traditional and simplified character issue, traditional style tasks are meaningful to researchers. We consider it as a future work of the community.
Classical Chinese is a treasure of the entire human cultural history. We contribute this work with the hope of helping the entire community to be more prosperous. Our work will be an open, community-driven project which improves with the advancement of technology.

heading: 7 Limitations
text: In this work, we contribute an evaluation benchmark for classical Chinese NLP tasks. We did our best to create as comprehensive a well-defined task set as possible, something no one has done before. However, our work has several limitations due to lacking expertise knowledge and data.
When designing the tasks, we got a lot of inspiration from the middle school Chinese test paper. thousands of test papers are collected in order to ex-
tract data for NLP tasks. During the work process, we learn that it is difficult to extract a sufficient number of questions of a single type. The main difficulty is due to the variety of questions on the test papers and the mixture of the language of classical and modern Chinese. Finally, we create Xuci task, WYWRC task and IRC task from the test papers and related literature but failed to create solvable natural language inference tasks.
When working on some datasets which have less corpus, i.e, the Xuci task, we find it very difficult to calibrate existing samples or create new ones, resulting a small dataset size.
Meanwhile, the category rule we followed in the GJC task is not certified by authoritative experts, so this method is not completely reliable if viewed by experts of classical Chinese.
In this work, tasks for more aspects of grammar phenomenon are lacking. It’s expected that more classical Chinese experts and researchers join this work in the future to solve the above problems.
On the other hand, we lack a diagnostic dataset compared to other benchmarks. This is because similar data (NLI corpus generally) are even more difficult to retrieve. However, this benchmark works for NLP researchers even though the diagnostic dataset is missing. This issue is also expected to be solved in future work.

heading: 8 Impact Statement
text: This work aims to help in enhancing the capabilities of pre-trained models for classical Chinese language basic infrastructure. Classical Chinese is a wealth of all humanity, with a great influence worldwide. We hope to help to improve the prosperity of the classical Chinese NLP community and better mine this spiritual wealth.
Our data sources, including raw classical Chinese text and related modern text, mainly come from official and authoritative releases, so there are generally not many ethical concerns. As mentioned above, when using copyrighted texts, we have got their permission. In terms of data sets, we expect to objectively and comprehensively reflect the language characteristics of classical Chinese as much as possible, so we also try to be as general as possible when sampling, without deliberately doing content filtering.
Bias and Race Concern Classical Chinese was born in ancient times when people entered the patriarchal society, and it is inevitable to be some
gender bias. Since such content is not common, we believe that there will not be many ethical issues. Additionally, one of the most important categories in ancient texts - historical books - records a large number of wars between the central government and surrounding minority ethnic groups, so there are derogatory and insulting terms among different ethnic groups. But fusions of different ethnic groups have been more often in the history books and no contents so-called racial discrimination in the current society exist.
Energy Cost Practical pre-training language model requires large amounts of computation, so the cost and efficiency of such models should be taken into account(Brown et al., 2020). To achieve a better score on the leaderboard, many tries of pre-training may be required hence a large amount of energy consumption. So models with better efficiency are preferred for environment-friendly reasons. For instance, in this work, the DeBERTa-base model scores better than the large models which have three times more parameters.

heading: Acknowledgements
text: We thank everyone who contributes dataset to this project. We are also grateful to the annotators and scholars who have spent much of their time and effort helping with the creation of this benchmark. This work was supported by Zhejiang Provincial Natural Science Foundation of China under Grant No. LZ23F020009, the NSFC under Grant No. 62072399, Chinese Knowledge Center for Engineering Sciences and Technology, MoE Engineering Research Center of Digital Library. Thanks to Hangzhou AI Computing Center for the computing resource. Special thanks to the following companies and organizations: Hangzhou Xiaoniao AI Co., Ltd., Hangzhou Bolazhe Co., Ltd. Gulian (Beijing) media tech. Co., Ltd.

heading: A Data Details and Annotation
text: In this section, we use a "[SEP]" mark to denote separation between two parts of a sample. And we try to translate the classical sentence to English to make it easier to understand. It should be noted that the English translations of the following texts are machine-translated, which are not very accurate.

heading: A.1 WYWRC

heading: A.1.1 Details
text: As previously mentioned, reading comprehension is a crucial aspect of classical Chinese learning, and is tested annually in the college entrance exam. With the assistance of middle school teachers, we have collected thousands of examples from examination papers. This dataset is in JSON format. Statistics are show in Figure 21 Figure 22 and Table 10. All the samples are separated into 10 types according to the difference between the questions which are shown in Table 4. Furthermore, as shown in Figure 5, we carry out a competency assessment of MRC capabilities to probe the challenge of this task in many fine-grained metrics.

heading: A.1.2 Annotation
text: • Prepossess test papers, including OCR, layout
parser and then copy all reading comprehension problems;
• Annotate, including filter the sub-problems for which suitable for machine learning and correct misspelling, etc.;
• Cross double-check between annotators;
• Final review by experts.

heading: A.2 PUNC

heading: A.2.1 Details
text: This task is designed for text punctuation. Since there are not any punctuation marks in traditional Chinese literature, discriminatory of sentence punctuation is important for reading ancient books. Even though ancient Chinese researchers have made great efforts in the proofreading and sorting out of ancient books, there are still a large number of ancient books without punctuation waiting to
be solved (Qi, 2022; Li, 2002). So that punctuation task is useful for classical Chinese researchers. Therefore, all related works evaluate their models mainly on this task.
To make sure the time distribution of the corpus as uniform as possible, we select history books as source data for this task including 二十四史(the Twenty-Four Histories),春秋(The Spring and Autumn Annals), 战国策(Strategies of the Warring States Period) and so on. The corpus contains historical books from the Zhou Dynasty to the Republic of China, which cover nearly three thousand years (1046 BC to 1927). All of the books are concatenated and shuffled by paragraph, then sampled by a reasonable rate and finally split into task datasets.
This dataset is in sequence pair TSV format. Every sample is a pair of source text and label sequence as shown following. We choose eight punctuation marks as prediction target in this dataset. Statistics are show in Figure 10 Figure 11 and Table 6.
壬戌诏定科举流寓人名额蒙古色目南 人各十五名汉人二十名 OO，OOOOOOOO，O、O、OOOOO， OOOO。 On the 24th, an imperial edict was issued to establish the quota of expatriates in the imperial examination, 15 for Mongolians, 15 for colored-eyes and 20 for Han Chinese.
谢肇《北河纪》八卷《纪余》四卷除 坛西郊坎其击鼓百灵至止结作主
OOOOOOOOOO，OOOOO|OOO，OOO 。OOO，OOO。 Xie Zhaozhe wrote eight volumes of Beihe Ji and four volumes of Ji Yu. At the altar in the western suburbs, playing drums, hundreds of gods stopped here and became the leader of the alliance.

heading: A.2.2 Annotation
text: • Extract corpora from the Twenty-Four Histo-
ries and some other history books;
• Filter out low quality samples, for example,
too short or punctuation too few;
• Sample from the result corpora with a specific ratio;
• Annotator check;
• Final review by experts.

heading: A.3 GLNER

heading: A.3.1 Details
text: This dataset is in JSON format as shown below. Every sample consists two keys which are "text" and "label", and every label is represented as start index, end index and category style. Statistics are show in Figure 12 Figure 13 and Table 7.
{ "text": "谢绛 三月戊戌，知礼仪 院、兵部员外、知制诰谢绛知邓州。 十一月己酉，卒。欧文。长编：绛按 召信臣故迹，距城三里，壅湍水，注 钳庐陂，溉田，请复修之。可，罢州 人岁役。", "label": [[0, 2, "other"], [21, 23, "other"], [24, 26, "other"], [35, 36, "other"], [38, 40, "bookname"], [41, 42, "other"], [43, 46, "other"], [59, 62, "other"]] }
{ "text": "六月己未，郑居中等上哲宗 御集。壬戌，景灵宫建禧祖殿室。复 广、惠、康、贺州旧铸夹锡钱监。辛 未，湖南路提点刑狱陈义夫奏邵阳县 贼平。", "label": [[5, 8, "other"], [10, 14, "bookname"], [18, 21, "other"], [22, 24,
0 100 200 300 400 500 0
500
1000
1500
2000
2500
3000
3500
4000
Sample Length
C ou
nt

heading: A.4 TLC

heading: A.4.1 Details
text: Since ancient books have been handed down over a period of more than 2,000 years, it is a very meaningful and challenging task to identify the writing time of ancient books according to the characteristics of the text. Chang et al. (2021) propose that identifying written time of literature is help-
ful for understanding works. Being classified according to the period, ancient Chinese is generally divided into ancient (Pre-Qin and Han Dynasty), mid-ancient (Jin Dynasty to Song Dynasty) and late-ancient (Yuan, Ming, Qing Dynasty) (Wang, 2004). Furthermore, in the process of the development of classical Chinese, each dynasty has its own unique characteristics (Li et al., 2013). In such background, we collect about 300 ancient books and famous articles which have exact time of writing, and sample a reasonable number of paragraphs from the texts. This dataset is in TSV format as shown below. The three segments of a sample are Period label, Dynasty label and source text respectively. Statistics are show in Figure 14, Figure 15 Figure 16 and Table 8.
近古 [SEP]元明 [SEP]主治风痹，筋 骨不仁，功与脂同。补虚羸。 Indications of wind arthralgia, numbness of the muscles and bones, its power is the same as fat. Make up for weakness. 中古 [SEP]魏晋南北朝 [SEP]东观汉 记曰：羌什长巩便。然更盖其种也。 尚书曰：歼厥渠魁。既已袭而馆其 县。左氏传曰：凡师轻曰袭。杜预 曰：掩其不备。子以眇尔之身，介 乎重围之里；率寡弱之众，据十雉之 城。 Dongguan Han Ji said: Gong Bian, the leader of the Qiang people. But it is another cover. The book of Shang said: Destroy the head of the thief. Has attacked Fianxian and stayed in a hotel. Zuo’s biography said: "Any army with light baggage is called袭." Du Yu said: Attacking who is unprepared. With a small body, you are in the center of the encirclement, leading the weak, and defending the city of ten feet. 远古 [SEP]先秦 [SEP]齐晏桓子卒， 晏婴粗斩，苴、带、杖，菅屦，食 鬻，居倚庐，寝苫、枕草。 When father died, Yan Ying wore coarse cloth mourning clothes, made filial piety clothes, belts and walking sticks of coarse linen, wore shoes made of thatch, ate thatch, ate thatch, lived in a leaning hut, and slept on a straw mattress.
50 100 150 0
2k
4k
6k
8k
10k
Sample Length
C ou
nt

heading: A.4.2 Annotation
text: • Annotator check;
• Final review by experts.

heading: A.5 GJC

heading: A.5.1 Details
text: The Si Ku Quan Shu had formed a classification method of four parts of Jing, Shi, Zi, Ji (Confucian classics, historical records, philosophical writings, and miscellaneous works), and 40 categories. This is the authoritative method till now. The largest classical Chinese corpus dataset Daizhige extends the method to 10 collections. Since this corpus is actually the basis of most of classical Chinese NLP research, we apply this method to design our text classification task following CCLUE. This dataset is in text–category format as shown below. Statistics are show in Figure 17 Figure 18 and Table 9.
然则世所谓雅乐者，未必如古，而教 坊所奏，岂尽为淫声哉？” [SEP] 艺 藏 However, the elegant music in the society is not necessarily the same as in ancient times, but is the music played by Jiaofang all debauched music? 有丧必求牧师殓，独自入房把门 掩。[SEP]子藏 If there is a funeral, you must find the priest to be buried, entering the room alone and close the door. “梦幻空花，何劳把捉？得失是非， 一时放却。” [SEP]佛藏 "Dreaming of empty flowers, how to take the handle? Do not care about the right and wrong, and put it back at once. "" 羲，乃天皇伏羲氏也。齐驱，即并 驾。元始，万有万无之祖号。比肩， 并立之义。是足上文比喻也。学者慎 毋住相，是即舜何人也，予何人也云 尔。[SEP]道藏 Xi is also called the Emperor Fuxi. “齐 驱” means the two marched side by side. “元始”, the ancestor of all things and nothing. "比肩" means to stand side by
0 50 100 150 200 250 0
5k
10k
15k
20k
25k
30k
35k
40k
Sample Length
C ou
nt

heading: A.5.2 Annotation
text: • Final review by experts.

heading: A.6 FSPC
text: This dataset is in JSON format as shown below. The sentiment labels are of five specifications which shift from negative to positive. Statistics are show in Figure 19 Figure 20.
{ "poet": "范仲淹", "poem": "静映寒林晚未芳|人人欲看 寿阳妆|玉颜须傍韶春笑|莫斗严风与 恶霜", Quietly reflecting the cold, the forest is late and not fragrant | Everyone wants to see Shouyang makeup | Jade face must be close to spring smile | Do not fight with strong wind and cold frost. "dynasty": "宋", "sentiments": { "holistic": "implicit positive", "line1": "implicit positive", "line2": "neutral", "line3": "implicit positive", "line4": "neutral" }, "title": "和提刑赵学士探梅三绝" }, { "poet": "王维", "poem": "独在异乡为异客|每逢佳节 倍思亲|遥知兄弟登高处|遍插茱萸少 一人", Being alone and a stranger in a foreign land | I miss my relatives every time during the festival | I know my brothers climb a high place from afar | Everyone plant cornel all over the place but me "dynasty": "唐", "sentiments": { "holistic": "implicit negative", "line1": "implicit negative", "line2": "implicit negative", "line3": "neutral", "line4": "implicit negative" },
A.7 Xuci

heading: A.7.1 Details
text: Function words (Xu ci in Chinese) have no real meaning and generally cannot be used as a single
24 26 28 30 32 0
500
1000
1500
2000
2500
3000
3500
4000
4500
Sample Length
C ou
nt
Figure 20: Statistic of Sample Length of FSPC dataset. Note that this dataset only contains five-character quatrains and seven-character quatrains.
sentence element (Liu et al., 1995). They are very important in classical Chinese but easily confused. Relevant topics are part of the basic knowledge for Chinese students which appears in the college entrance exam every year. We collect sentence pairs with function words from examination papers with help of middle school teachers to construct this dataset.
This dataset is in TSV format. Statistics are show in Figure 21 Figure 22 and Table 10.
使夫邪污之气无由得接焉。[SEP]复 驾言兮焉求。[SEP] 10, 10 [SEP] 4, 4 [SEP] f so that there is no way for those evil and filthy atmospheres to reach them. [SEP]What am I driving for? 上官令民送牛羊之陕西。[SEP] 久 之，举于朝。[SEP] 7, 7 [SEP] 1, 1 [SEP] f The superior commander sent cattle and sheep to Shaanxi. [SEP] After a long

heading: A.7.2 Annotation
text: correct misspelling, etc.;
• Cross double-check between annotators;
• Final review by experts.
A.8 IRC

heading: A.8.1 Details
text: Idiom is one of the major features of Chinese culture. Most of the idioms are long-standing fixed phrases, derived from ancient classics or writings, historical stories, and oral stories. For idiom comprehension, there are other tasks (Zheng et al., 2019) ready. However, they are mainly aiming to test modern Chinese texts with idioms. To focus on classical Chinese, we implement this task。This dataset is in JSON format. Every sample consist of four fields which are "idiom", "options", "label" and "origin". The ground truth "label" is best fit of the four options. Statistics are show in Figure 23 Figure 24 and Table 11.
{ "idiom": "眼去眉来", eye to eyebrow "options": [ "火烧到眉毛。比喻事到眼前，非常 急迫。", The fire burned to the eyebrows. The metaphor is very urgent. "形容事情已到眼前，情势十分紧 迫。", Describe the matter has come to the front, the situation is very urgent. "原指眼前见到的。后形容用眉眼传 情。", It meant what was seen. After describing the use of eyebrows teasing. "形容眉眼含情示意的神态。" Describe the expression of the eyebrows showing affection. ], "label": 2, "origin": "落日苍茫，风 才定，片帆无力。还记得眉来眼去， 水光山色。" The setting sun is vast, the wind is fixed, and the sails are weak. I still remember the frowning, the water and the mountains. },

heading: A.8.2 Annotation
text: • Prepossess test papers, including OCR, layout
parser and then copy all idiom comprehension problems;
• Study the idiom dictionary to extract idiom comprehension problems;
• Annotate, including filter the sub-problems for which suitable for machine learning and correct misspelling, etc.;
• Cross double-check between annotators;
• Final review by experts.

heading: A.9 WYWMT

heading: A.9.1 Details
text: Classical Chinese is a very concise written language, so it’s not easy for everyone to understand.
Scholars often translate classical Chinese into modern Chinese with notations to make it easier for people to read. We consider it as an in-language translation or rewriting task because the source and target could share the same vocabulary and some semantic features.
This dataset is filtered and calibrated from hundreds of translated classical Chinese books collected from multiple channels. Since allusions and quotations appear frequently in classical Chinese, and these references may have a time span of thousands of years, it’s not easy to construct a very well-established dataset by ourselves.
This dataset is in sentence pair TSV format. Samples are represented as "source" and "reference" segment which are separated by "tab". Statistics are show in Figure 25 and Table 12.
共四里，又越一冈脊而下，其脊高不 及高井之半，而实为西北来过脊以趋 清秀者也。[SEP]共四里，又越一道 冈脊后下走，这个冈脊高处不到高井 的一半，但实际上是从西北前来趋向 清秀山的延伸而过的山脊。 After a total of four miles, we went down after another ridge. This ridge was less than half of the height of Gaojing, but it was actually a ridge extending from the northwest towards Qingxiu Mountain. 读性理书时，则杂以诗文各集，以歧 其趋。[SEP]在读性理书的时候，又 掺杂写诗文，走了岔路。 When I read books about ethics, I mixed it with writing poetry, so I went to a wrong road.
0 100 200 300 400 500 0
100k
200k
300k
400k
Modern Chinese Classical Chinese
Sample Length
C ou
nt
Figure 25: Statistic of Classical Sample Length of WYWMT dataset.

heading: A.9.2 Annotation
text: • Crawl classical Chinese articles with modern
translation from all channels, note that these articles are open and free for everyone;
• Text align;
• Split long paragraphs into shorter sentences;
• Filter low quality examples;
• Final review by experts.

heading: B Details of human evaluation
text: To obtain a more reliable evaluation of model performance, we chose students majoring in Classical Chinese to provide an upper bound score. This sets a higher value for the benchmark and helps researchers improve their models. The students were given access only to the train-set during the test, without any additional tools. For different tasks, set of the test examples is different. See Table 13.

heading: C Details of Models Evaluated
text: In this section, we present the details of pretrained language models we used, including guwenbert-base, guwenbert-large, guwenbertbase-fs, roberta-classical-chinese-base-char, roberta-classical-chinese-large-char, SikuBERT, SikuRoBERTa, DeBERTa-base and RoBERTawwm-ext. As shown in 14, the masking, scale, corpus, vocabulary and parameter initialization are different in each pre-trained language model.

heading: D Hyper-parameters for fine-tuning
text: As shown in Table 15, we present the hyperparameters applied in fine-tuning. For different scale of pre-trained language model, we set different learning rates. In large scale, we set learning rates with 5e-6, 8e-6, 9e-16 and 1e-5. In base scale, we set learning rates from 1e-5 to 5e-5. We set warmup to 0.1, maximum epochs to 10. For Adam, we set ϵ to 1e-6, β1 and β2 to 0.9 and 0.999 respectively. Meanwhile, we use linear for LR decay and set weight decay to 0.01.

heading: E Leader-board
text: Following other benchmark leaderboard, this leaderboard is designed containing an overall list and several sub-lists. Metrics of the tasks are listed in Table 1. Since most of the tasks are common tasks, we did not design more metrics for them. For the sequence-to-sequence task of WYWMT, due to the shortcomings of various indicators, we calculate several metrics for the prediction results to make that clearer (3).
ACL 2023 Responsible NLP Checklist

heading: A For every submission:
text: 3 A1. Did you describe the limitations of your work?
7
A2. Did you discuss any potential risks of your work? Not applicable. Left blank.
3 A3. Do the abstract and introduction summarize the paper’s main claims? 1
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? 4
3 B1. Did you cite the creators of artifacts you used? 2, 4
3 B2. Did you discuss the license or terms for use and / or distribution of any artifacts? 3.3
3 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? 2
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Not applicable. Left blank.
B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? Not applicable. Left blank.
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. 3, appendix A
C 3 Did you run computational experiments? 5
3 C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? 5.2
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? 5.3
3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? 5.5
3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? 3.5
D 3 Did you use human annotators (e.g., crowdworkers) or research with human participants? 3.4
3 D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? 3.4
3 D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? 3.4
3 D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? 3.3
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? Not applicable. Left blank.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? Not applicable. Left blank.

heading: Future Work
text: It’s expected that more classical Chinese experts and researchers join this work in the future to solve the above problems. On the other hand, we lack a diagnostic dataset compared to other benchmarks. This is because similar data (NLI corpus generally) are even more difficult to retrieve. However, this benchmark works for NLP researchers even though the diagnostic dataset is missing. This issue is also expected to be solved in future work