DEIM Forum 2023 1b-6-2

Token-Free Cross-Lingual Named Entity Recognition for ClassicalChineseZhongqing JIANG† , Zengqing WU† , and Chuan XIAO†,††† Graduate School of Information Science and Technology, Osaka University
1–5 Yamadaoka, Suita, Osaka, 565–0871 Japan†† Graduate School of Informatics, Nagoya University
Furo–cho, Chikusa–ku, Nagoya, 464–0814 Japan
E-mail: †{jiang.zhongqing,chuanx}@ist.osaka-u.ac.jp, ††wuzengqing@hus.osaka-u.ac.jp
Abstract With the support of cross-lingual transfer learning techniques, the performance of named entity recognition (NER) for low-resource languages has recently received considerable improvement. However, the performanceof existing methods still falls short on token-free low-resource target languages, such as Classical Chinese. In thispaper, we target NER for Classical Chinese, and propose a method for token-free cross-lingual NER zero-shottransfer based on the ByT5 model. We build a Modern Chinese-Classical Chinese entity alignment model with the
ByT5-based Modern Chinese NER classiﬁer to obtain the predicted labels of Classical Chinese corpus. Following,
we train a pre-trained language model for Classical Chinese and ﬁne-tune it using supervised learning methods in
combination with the prediction labels obtained. We evaluate our approach on a widely used Classical Chinese NER
dataset C-CLUE and obtain better results than various baseline methods in the token-free condition and reduce thecomputational requirements for training the model.
Key words Named Entity Recognition, Natural Language Processing, Transfer Learning, Classical Chinese

1 Introduction

language, studied in this paper as an example, the study [12]points out that there are a large number of unlabeled Classi-

Natural language processing (NLP) for low resource lan-cal Chinese texts that have not been included as part of theguages where labeled data is scarce has always been a chal-language model training, and with the help of LLM, theselenge in the ﬁeld, and the advent of pre-trained language

corpus sets are able to be used.

models has improved the situation for solving this prob-

However, despite the fact that the aforementioned methods

The latest large-scale pre-trained language models

have brought a considerable improvement in the performance(LLM), such as GPT-3 [1], BERT [6], T5 [18] and other self-of NLP tasks, restricted by the limited corpus of the target

supervised models that are pre-trained and ﬁne-tuned on

language and the feature diﬀerences between the target lan-large-scale corpora have substantially improved the perfor-

guage and the source language, there is still much space formance of NLP tasks in low-resource languages since they

improvement in the performance of existing methods on un-

do not rely on labeled data for training [15]. Building onlabeled, low-resource target languages. Existing studies ar-

this, the development of cross-lingual transfer learning tech-gue that the fact of large pre-trained language models trainedniques has led to signiﬁcant performance improvements foron general-domain texts makes them diﬃcult to transfer tomany NLP tasks, such as named entity recognition (NER),

some domain-speciﬁc texts [15], [22]. General-domain texts,

especially for those languages with low resources [8]. A num-such as English and Modern Chinese, diﬀer signiﬁcantly fromber of transfer learning methods provide some pseudo-labels

the linguistic features of Classical Chinese [12], [22]. On thefor low-resource target languages by using pre-trained cross-other hand, the small size of the extant corpus of Classi-

lingual language models of the source language, thus allow-

cal Chinese, compared to other languages, limits the perfor-

ing the target language model to learn in a zero-shot conﬁg-mance of LLM self-supervised learning.

lem.

uration [5], [8], [20]. This enables the training of models for

Further, aside from the problem of limited dataset size,

low-resource languages no longer relies on expert annotations

Classical Chinese has additional word separation require-and large-scale labeled datasets. In terms of speciﬁc applica-

ments compared to most modern languages.

tions, taking the Classical Chinese language, a low resource

to European languages such as English and German, Chi-

In contrast

nese and Japanese do not use whitespace as a boundary be-

BERT achieved better results compared to traditional mod-tween words, and Classical Chinese goes further by not hav-els [7], [22]. Some other models, such as deep neural net-

ing markers as separators between sentences, which makes

works, have also been applied in Classical Chinese NER prob-

word separation more challenging [9]. The Classical Chineselems [24]. However, as mentioned earlier, still limited by the

corpus with expert annotations for punctuation is very lim-

scale of the original corpus of Classical Chinese, the perfor-

ited, and expert annotations are subject to certain errors.

mance of LLM on Classical Chinese is still deﬁcient compared

Existing studies have shown that the accuracy of Classicalto its performance on other modern languages.

Chinese word separation aﬀects performance on other down-

Several transfer learning and contrast learning methods

stream tasks like NER [2]. The performance of existing at-have been used to train Classical Chinese language mod-tempts to pre-train and transfer learning directly on Classicalels. One research uses a contrastive learning approach toChinese without word separation and markers is limited.

enhance machine reading comprehension of both languages

Therefore, in this paper, we focus on exploring how to bet-by contrastive learning of Classical Chinese and Modern

ter implement the NER task for token-free Classical Chinese.

Chinese [13].

We argue that the transfer learning approach via Moderntransfer approach to transfer the word segmentation features

The AT-CCNER model uses an adversarial

Chinese-Classical Chinese entity alignment on the unlabeledlearned by the language model from large-scale translation

corpus can perform better than the traditional automatedto Classical Chinese, obtaining better performance than ex-

word separation combined with supervised learning methods

isting approaches [17]. It is evident from these studies thatand direct training using LLM in a zero-shot setting.

it is diﬃcult to apply LLM directly to the training of a low-

We propose a token-free cross-lingual zero-shot transfer

resource language like Classical Chinese, and that with somelearning model based on the ByT5 model [23] to process the

combination of learning from a broader corpus, such as trans-

Classical Chinese NER task. The method is based on a Mod-fer from Modern Chinese to Classical Chinese, can usefully

ern Chinese NER classiﬁer based on ByT5, a pre-trained

enhance the performance of the experiments.

token-free byte-level language model that directly processes
the raw text, combined with the entity alignment method
to obtain prediction labels for corpus of Classical Chinese.
This is followed by training a pre-trained language model forClassical Chinese and ﬁne-tuning it using supervised learning
methods with the obtained prediction labels.
We evaluate our model on the open-source Classical Chinese text dataset C-CLUE [10] and compare it with existingbaseline methods. It turns out that our model achieves thebest performance in the Classical Chinese NER task. We
also conduct ablation experiments to analyze how the components of our model aﬀect its performance.

2 Related Works

2. 2 Token-Free Language Models
The shortcomings of current research on token-free languages have been noted in some existing studies. For example, the ByT5 model, a byte-level sequence-to-sequencemodel, is based on the T5 architecture to design a Transformer architecture that processes raw text (i.e., bytes) directly, rather than language processing on sequences of words
or subwords, on language-independent, token-independent
NLP tasks [23]. This approach gives a signiﬁcant performance boost for tasks in token-free languages, and is thereason why this study employs ByT5 as the basis of our language model. Classical Chinese, as a language that does notdelimit words and sentences with punctuation and whitespace, makes the application of models with tokenizer on it

2. 1 Chinese NER Task
A lot of existing work has been done on the Chinese NERtask. Early work trained language models based on word separation, e.g., LSTM-CRF [16], Latice-LSTM [25], etc. Subsequently, some LLM-based work and the development oftransfer learning techniques have driven the progress of Chinese NER tasks. For example, the BERT-wwm model [4],which uses BERT as the framework and incorporates certain
whole word masking strategies, and the RoBERTa model[14], which adjusts the selection and experimental conﬁguration of its hyperparameters based on BERT, are widelyused as baselines for Classical Chinese NER problems. Also
based on BERT, some studies that trained on large scale
Classical Chinese corpus, such as AnchiBERT and Siku-

diﬃcult, because these steps of dividing the raw text into
tokens are usually based on whitespace and markers. In
addition, since all e-texts in Classical Chinese are enteredmanually, there is a certain amount of noise and mis-entries
in them, which can negatively aﬀect the results of both theword segmentation and the non-token-free methods. Comparatively, ByT5 excels in processing noisy text data due tofeatures such as direct processing of raw text [19]. For thesereasons, this paper argues that the ByT5 model is a good
match for the linguistic properties of Classical Chinese, it
avoids the interference of errors in intermediate steps suchas word separation on the ﬁnal results, and weakens the disadvantages of the absence of markers that make the language

Figure 1 Framework

the target language (Classical Chinese) with prediction la-model diﬃcult to apply directly.
Nevertheless, the reason we did not directly train the Clas-

bels after building a Modern Chinese - Classical Chinese en-sical Chinese language model using ByT5 is that the modeltity alignment during the translation procedure; (2) using

only shows more prominent performance with limited model

ByT5’s architecture to train a preliminary pre-trained lan-size and its performance varies on languages with diﬀerent

guage model for the target language; and ﬁnally (3) ﬁne-tune

token compression rates [23]. Relatively, we use the Mod-the pre-trained language model for the target language using

ern Chinese NER classiﬁer parameters of ByT5 as a source

supervised learning methods in combination with the predic-of transfer learning to achieve better performance by ﬁne-tion labels obtained. Eventually, the model will result in atuning. To the best of our knowledge, this paper is the ﬁrst

zero-shot token-free NER classiﬁer for the target language.

to use a token-free language model to deal with the classicalChinese NER problem.

3. 2 Entity Alignment
The target language corpus is limited by the presence of

2. 3 Cross-Lingual Pre-training Models

relatively frequent missing or diﬃcult-to-read parts, and the

The most widely used cross-lingual pre-training model inexistence of a range of heterogeneous or rare Chinese char-

existing studies is the multilingual version of BERT, which

acters, which results in a much larger character set size com-

has achieved leading results in various NLP tasks. Whereas

pared to other languages and impacts the eﬀectiveness of afor low-resource languages, some studies have trained cross-range of NLP downstream tasks, including NER. Hence, we

lingual models based on BERT on large amounts of data,

use entity alignment to address this issue. To achieve entity

such as XLM-RoBERTa, showing the conclusion that ﬁne-alignment, we ﬁrst translate the unlabelled corpus for thetuning for low-resource languages can further improve modeltarget language, align the translated source language withperformance [3]. In addition, some existing researches en-

the target language for entity alignment, and label it using

hance the eﬀect of ﬁne-tuning on NER task by entity align-

the NER classiﬁer with ByT5 for the source language. The

ment from source to target language in pre-training phase

entity alignment is performed using the exact match method.

[11], [26]. Therefore, this work try to improve the eﬀectiveness of ByT5 on Classical Chinese by using transfer learning,
entity alignment and ﬁne-tuning from Modern Chinese parameters of ByT5 model to Classical Chinese.

3. 3 Pre-training Based on ByT5
We adapt the ByT5 architecture to pre-train the corpus of
the target language. Since T5 architecture is capable of implementing major NLP downstream tasks, the trained ByT5
model is able to perform NER task after a simple ﬁne-tuning.

3 Methodology

The corpus used for training is token-free.
3. 1 Model Architecture
Figure 1 illustrates the framework of our model.

The

3. 4 Cross-Lingual Fine-tuning

model is divided into three main components, which are (1)We obtain prediction labels for the target language train-using ByT5 [23]’s token-free NER classiﬁer for the source lan-ing in the zero-shot conﬁguration by the entity alignment

guage (Modern Chinese) to label the training set corpus of

method and ByT5 NER classiﬁer, and we apply these labels
to the initially trained target language ByT5 model for super-vised learning to obtain a zero-shot token-free NER classiﬁer

4. 4 Baselines
We compare the performance of our model with the mod-for the target language.

els commonly used in Classical Chinese NER task, including

4 Experiments

BERT [6], BERT-wwm [4], RoBERTa-zh [14], Lattice-LSTM[25], etc.

4. 1 Dataset
We evaluate our model using the Classical Chinese NER
dataset C-CLUE [10]. The dataset includes six named entity
categories including person (PER), location (LOC), organization (ORG), etc. Separately, limited by the size of the C-

5 Results
Table 1 shows the experimental results of diﬀerent models
for the NER task on the C-CLUE dataset.

CLUE dataset, we use Daizhige （ 1）, a Classical Chinese text

The results show that our token-free model obtains the

corpus including 15 thousand and more texts from various

best experimental performance compared to the above mod-Classical Chinese literature, as the source of training set forels. This reﬂects that our model can better handle the char-the model. We preprocessed this training set, including re-acteristics of Classical Chinese which is diﬃcult to be seg-moving invalid tokens from non-Classical Chinese texts andmented and the raw text has no punctuation marks. The

non-UTF-8 characters that cannot be used as model inputs.

main reason for this result is that the token-free model

Then, to match the use of the ByT5 model, we convert thematches better with the Classical Chinese language in terms

raw text characters from Traditional Chinese to Simpliﬁed

of word segmentation, amount of tokens, grammatical pat-

Chinese.

terns, and other factors.
First, considering that Classical Chinese is featured by the

4. 2 Pretraining Setup
Considering the size of the dataset, our model uses thesame conﬁguration as ByT5-small [23]. The model uses atransformer encoding-decoding model with 12 encoding layers and 4 hidden layers, and 300M parameters. We pre-train
the ByT5 model for 1 million steps on token-free text with amaximum sequence length of 1024 bytes and a batch size of
220 tokens. The experiments are run on Google Colab platform with Google Cloud TPU v2-8. In all experiments, thesame network layers are set with the same parameters andperform in the same experimental environment.
For NER task, we ﬁne-tune ByT5 for 1000 steps with thesettings of a constant learning rate of 0.001 and a dropout

absence of punctuation, traditional methods, such as LSTMCRF, need to perform word segmentation ﬁrst before implementing the NER task. Existing studies show that the quality of word segmentation directly aﬀects the performanceof the NER task. Compared with languages such as Modern Chinese and Classical Japanese, word segmentation inClassical Chinese is more diﬃcult. The inevitable errors inword segmentation eventually degrade the performance of
the NER task. Therefore, this paper uses the token-free approach to avoid the problem of word segmentation. This paper further argues that the language model based on tokenfree architecture is also applicable to other downstream tasks
that are aﬀected by word segmentation.

rate of 0.1.

On the other hand, this paper argues that the advantage
4. 3 Evaluation Settingsof applying a token-free model on Classical Chinese is not

The NER task uses Precision (P), Recall (R), and F1-score

only the avoidance of the word segmentation process. Con-

as metrics. As the indicator that reﬂects the model perfor-sidering that there are many variants of Classical Chinesemance in a balanced way, F1 will be used as the primary

vocabulary and that the text has a lot of noise (e.g., humanindicator to measure the results of our experiments.

input errors, unreadable parts of the original text, etc.), theTable 1 Experimental Results on C-CLUE Dataset

dataset carries noise, and the total number of tokens is large,

MethodP

R

F1

all these factors decreases the performance of the pre-trained

BERT-Base

29.82

35.59

32.12

model. In contrast, the token-free model is thought to reduce

BERT-wwm

32.98

43.82

35.40

the interference of these problems because it does not deal

RoBERTa-zh

28.28

34.93

31.09

with characters/words, but directly with bytes. Further em-

BiLSTM-CRF

49.76

54.77

52.15

pirical studies are still needed to explore whether token-free

Lattice-LSTM

51.96

56.92

54.33

Our Model

52.21 56.94 54.47

models may also have better results in downstream tasks inother languages without punctuation.
Finally the transfer learning approach is able to deal withthe limited corpus of Classical Chinese as a low-resource language. Since the corpus set of Classical Chinese is consider-（ 1）：Available at: https://github.com/garychowcmu/daizhigev20

ably small in size even if it is unlabeled. This paper showsthat the transfer learning and entity alignment methods caneﬀectively improve the NER task performance of Classical[2]

Chinese.

6 Ablation Study
To analyze the eﬀect of diﬀerent factors on the validity of[3]

our model, we conducted ablation experiments on the model,
as shown in Table 2.
Table 2 Results of Ablation Experiments
MethodP

R

F1

Our method [ByT5+Transfer]52.21 56.94 54.47-Cross-lingual Fine-tuning [ByT5] 49.28 55.18 52.06

[4]

[5]

Eﬀectiveness of transfer learning: To test the eﬀectivenessof transfer learning, we perform experiments after removing the part of transfer learning from the model. The results[6]

show that our model with transfer learning included has better performance compared to the ByT5 model trained on the[7]

Classical Chinese corpus only.

7 Conclusion
In this paper, we propose a token-free cross-lingual zero-[8]

shot transfer learning model for the Classical Chinese NERtask. Through a series of experiments, we show that ourmodel can eﬀectively improve the performance of the NER[9]task. We ﬁnd that the token-free model, such as ByT5, canhandle the linguistic features of Classical Chinese well. In addition, the transfer learning approach can eﬀectively address
the problem of limited size of the target language dataset,[10]

and the performance of the NLP task in the target language
can be improved by transfer learning from the source language to the target language.
In future research, we will compare the models with morebaselines. Further, we will perform more experiments tovalidate the model, including validation of entity alignment[11]

methods, comparison of using architectures (e.g., replacing ByT5 with BERT, mT5, other token-free models like

[12]

Charformer [21], etc.). In addition, the generalization ability
about the ﬁndings of this paper in other languages as well as
in other NLP downstream tasks can be further discussed.

8 Acknowledgements[13]This work was supported by JSPS KAKENHI Grant Number 22H03903. We thank Dr. Ruoming Pang, Mr. Run
Peng, Prof. Keishi Tajima, and Prof. Hiroki Nomiya for[14]

their instructive suggestions.
References[1] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al.

[15]

Language models are few-shot learners. Advances in neural
information processing systems, 33:1877–1901, 2020.
Pengfei Cao, Yubo Chen, Kang Liu, Jun Zhao, and Shengping Liu. Adversarial transfer learning for chinese namedentity recognition with self-attention mechanism. In Proceedings of the 2018 conference on empirical methods innatural language processing, pages 182–192, 2018.
Alexis Conneau, Kartikay Khandelwal, Naman Goyal,
Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán,
Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin
Stoyanov. Unsupervised cross-lingual representation learning at scale. arXiv preprint arXiv:1911.02116, 2019.
Y Cui, W Che, T Liu, B Qin, Z Yang, S Wang, andG Hu. Pre-training with whole word masking for chinese
bert. arxiv preprint arxiv: 190608101. 2019.
Wietse de Vries, Martijn Wieling, and Malvina Nissim.
Make the best of cross-lingual transfer: Evidence from pos
tagging with over 100 languages. In Proceedings of the 60th
Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 7676–7685, 2022.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina
Toutanova.
Bert: Pre-training of deep bidirectionaltransformers for language understanding. arXiv preprint
arXiv:1810.04805, 2018.
Wang Dongbo, Liu Chang, Zhu Zihe, Liu Jiangfeng,
Hu Haotian, Shen Si, and Bin Li. Sikubert and sikuroberta:
Research on the construction and application of pre-training
model of sikuquanshu for digital humanities. Library Tribune, pages 1–14, 2021.
Julian Martin Eisenschlos, Sebastian Ruder, Piotr Czapla,
Marcin Kardas, Sylvain Gugger, and Jeremy Howard. Multiﬁt: Eﬃcient multi-lingual language model ﬁne-tuning.
arXiv preprint arXiv:1909.04761, 2019.
Shohei Higashiyama, Masao Utiyama, Eiichiro Sumita,
Masao Ideuchi, Yoshiaki Oida, Yohei Sakamoto, Isaac
Okada, and Yuji Matsumoto. Character-to-word attention
for word segmentation. Journal of Natural Language Processing, 27(3):499–530, 2020.
Zijing Ji, Yuxin Shen, Yining Sun, Tian Yu, and Xin
Wang. C-clue: A benchmark of classical chinese basedon a crowdsourcing system for knowledge graph construction. In Knowledge Graph and Semantic Computing:Knowledge Graph Empowers New Infrastructure Construction: 6th China Conference, CCKS 2021, Guangzhou,
China, November 4-7, 2021, Proceedings 6, pages 295–301.
Springer, 2021.
Bing Li, Yujie He, and Wenjin Xu. Cross-lingual named entity recognition using parallel corpus: A new approach using
xlm-roberta alignment. arXiv preprint arXiv:2101.11112,
2021.
Dayiheng Liu, Kexin Yang, Qian Qu, and Jiancheng
Lv.
Ancient–modern chinese translation with a newlarge training dataset.
ACM Transactions on Asianand Low-Resource Language Information Processing (TALLIP), 19(1):1–13, 2019.
Maofu Liu, Junyi Xiang, Xu Xia, and Huijun Hu. Contrastive learning between classical and modern chinese forclassical chinese machine reading comprehension. ACM
Transactions on Asian and Low-Resource Language Information Processing, 22(2):1–22, 2022.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke
Zettlemoyer, and Veselin Stoyanov.
Roberta: A robustly optimized bert pretraining approach. arXiv preprint
arXiv:1907.11692, 2019.
Qiuhao Lu, Dejing Dou, and Thien Nguyen. Clinicalt5: A
generative language model for clinical text. In Findingsof the Association for Computational Linguistics: EMNLP

2022, pages 5436–5443, 2022.
[16] Nanyun Peng and Mark Dredze. Improving named entity
recognition for chinese social media with word segmentation
representation learning. arXiv preprint arXiv:1603.00786,
2016.
[17] Yongjie Qi, Hongchao Ma, Lulu Shi, Hongying Zan, andQinglei Zhou. Adversarial transfer for classical chinese ner
with translation word segmentation. In Natural Language
Processing and Chinese Computing: 11th CCF International Conference, NLPCC 2022, Guilin, China, September
24–25, 2022, Proceedings, Part I, pages 298–310. Springer,2022.
[18] Colin Raﬀel, Noam Shazeer, Adam Roberts, Katherine Lee,
Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, andPeter J Liu. Exploring the limits of transfer learning witha uniﬁed text-to-text transformer. The Journal of Machine
Learning Research, 21(1):5485–5551, 2020.
[19] David Samuel and Milan Straka. \’ufal at multilexnorm
2021: Improving multilingual lexical normalization by ﬁnetuning byt5. arXiv preprint arXiv:2110.15248, 2021.
[20] Henning Schäfer, Ahmad Idrissi-Yaghir, Peter Horn, andChristoph Friedrich. Cross-language transfer of high-quality
annotations: Combining neural machine translation withcross-linguistic span alignment to apply ner to clinical texts
in a low-resource language. In Proceedings of the 4th Clinical Natural Language Processing Workshop, pages 53–62,
2022.
[21] Yi Tay, Vinh Q Tran, Sebastian Ruder, Jai Gupta,
Hyung Won Chung, Dara Bahri, Zhen Qin, Simon Baumgartner, Cong Yu, and Donald Metzler. Charformer: Fast
character transformers via gradient-based subword tokenization. arXiv preprint arXiv:2106.12672, 2021.
[22] Huishuang Tian, Kexin Yang, Dayiheng Liu, and Jiancheng
Lv. Anchibert: A pre-trained model for ancient chinese language understanding and generation. In 2021 International
Joint Conference on Neural Networks (IJCNN), pages 1–8.
IEEE, 2021.
[23] Linting Xue, Aditya Barua, Noah Constant, Rami Al-Rfou,
Sharan Narang, Mihir Kale, Adam Roberts, and Colin Raffel. Byt5: Towards a token-free future with pre-trained
byte-to-byte models. Transactions of the Association for
Computational Linguistics, 10:291–306, 2022.
[24] Chengxi Yan, Qi Su, and Jun Wang. Mogcn: Mixture ofgated convolutional neural network for named entity recognition of chinese historical texts. IEEE Access, 8:181629–
181639, 2020.
[25] Yue Zhang and Jie Yang. Chinese ner using lattice lstm. In
Proceedings of the 56th Annual Meeting of the Association
for Computational Linguistics (Volume 1: Long Papers),
pages 1554–1564, 2018.
[26] Ran Zhou, Xin Li, Lidong Bing, Erik Cambria, Luo Si,
and Chunyan Miao. Conner: Consistency training forcross-lingual named entity recognition. arXiv preprint
arXiv:2211.09394, 2022.