# Few-Shot Relation Extraction on Ancient Chinese Documents
## 1 Introduction
The pretraining models of BERT-large [9] with 340 M parameters are trained using 64 TPU (Tensor Processing Unit) chips for 4 days. The training requires at least 12-16 GB GPUs to address memory constraints while training with only one batch ([[https://github.com/google-research/bert](https://github.com/google-research/bert)]([https://github.com/google-research/bert](https://github.com/google-research/bert)), accessed on 16 December 2021). The need for huge computation resources makes rapid feedback or individual experiments impractical. The need for a large corpus cannot be met in research fields with limited text. BERT used 16 GB uncompressed plain text from BookCopus [10] and English WikiPedia. RoBERTa [11] (158 GB) and XLNET [12] (161 GB) used more corpus for models pretraining. However, even the complete collection SiKuQuanShu ([[http://skqs.guoxuedashi.net](http://skqs.guoxuedashi.net)]([http://skqs.guoxuedashi.net](http://skqs.guoxuedashi.net)), accessed on 16 December 2021) has merely a total of 1.5 GB plain text data for the ancient Chinese documents. Pretraining-based models are impractical due to a lack of training corpus. Moreover, the corpus cannot be collected from contemporary Chinese documents because of their different sentence structures, idioms, and words. Pretraining is difficult because of these factors. Accordingly, word representation-based models, such as Meta Networks (CNN), Prototypical Networks (CNN) [6], and MLMAN [13], were suggested. These models achieved good performance when GloVe [14] was used as the static word embedding vector. However, a large gap is still observed between humans and the models. The best result of these models lags 10% behind human performance on the FewRel test dataset. Moreover, these models require pretrained word representation (e.g., a standard GloVe model generally requires 42 billion tokens to generate a global word representation matrix). Although static word embedding models may be faster than dynamic models, such as BERT, they still require an external corpus. We examined several classical few-shot learning methods that do not have a satisfying performance to address the above problems. Then, we developed a multi-head self-attention with convolution encoder (MASCOT) and paired attention (PA) relation extractor to fill the gap. The MASCOT encoder can capture more local and global features. Our experiment has shown that this encoder has a better sentence encoding ability than convolution or transformer layer. This encoder achieves better performance than the standard GloVe-based convolution encoder, which means that its structure relieves the resource dependence to a large extent even without an external corpus for word embedding pretraining. Additionally, the paired attention layer is used to enhance instance's feature information and expressive capability. Our experimental results show that the paired attention achieved a 3.68% improvement over the base model. For the ancient Chinese documents, we tested our proposed model on the ShihChi documents and analyzed the prediction results in Section 5.5. The result shows that the relation extraction on the ancient Chinese document is useful, and researchers can analyze or plot the structured predicted data to support their research or arguments. We performed a demo analysis for ShihChi documents to demonstrate a usage for our model, which might inspire researchers using manners of this model. Finally, our contribution may be summarized as follows: First, we examined the prevailing few-shot learning methods and adopted them in the few-shot relation extraction task to find a better solution. Second, we used a local GloVe word representation and MASCOT-PA to avoid problems with the high pretraining corpus requirement. Finally, we applied our model to an ancient Chinese document relation extraction task and achieved promising results.