# Metamorphic Testing of Relation Extraction Models
## 1 Introduction  
Relation extraction (RE) is a fundamental task of natural language processing (NLP) and is also an important sub-task of information extraction. The ultimate goal of RE is to extract the relation between some entities according to a given text. Specifically, given a text and two entities (that appear in the text) as inputs, RE predicts the semantic relation (if any) holding between the two entities. Formally, given a triplet of \((s,e_{h},e_{t})\), RE determines which relation \(r\in\mathcal{R}\) holds between \(e_{h}\) and \(e_{t}\) in \(s\) or indicates \"no_relation\" (\(\phi\)). Here, \(\mathcal{R}\) is a predefined relation set which contains a variety of person-oriented or organization-oriented relation types, such as \"per:children\" (indicating that the tail entity \(e_{t}\) is the child of the head entity \(e_{h}\)) and \"org:alternate_names\" (representing that \(e_{t}\) is an alias of \(e_{h}\)). Furthermore, RE supports the construction of knowledge graph [1] and further facilitates a series of downstream applications, such as question answering systems [2], search engines [3], sentiment analysis [4], and text summarization [5]. In light of its importance, various deep learning models have been developed for RE, including models based on attention mechanisms [6], models combining external information sources [7], and pretrained language models [8]. Although many efforts have been made to advance RE models, the quality of the state-of-the-art RE models is still far from perfect [9]. For example, RE models may rely on shallow heuristics instead of learning to perform the intended task. As a result, although these models are reported to be effective on a dataset, they may perform poorly on data beyond the dataset [10]. Additionally, some RE models have been reported to have over-reliance on entity information [11], and some models even have difficulties in distinguishing relations with similar types of signatures [12]. What is worse, some models have been shown to ignore the information expressed by thecontext semantics and thus fail to understand the context correctly [13]. Therefore, there is an urgent need to evaluate RE models properly. Unfortunately, evaluating RE models is non-trivial. Existing studies mainly apply the dataset-based evaluation method. One critical issue in this method lies in the reliance on ground truth labels, which always require tedious human annotations. Furthermore, low-quality annotations may in turn affect the evaluation of RE models [13; 14]. This situation is known as the oracle problem [15] in software testing. On the other hand, dataset-based evaluation reports the accuracy of a model with respect to the evaluation dataset, which is insufficient to reveal the characteristics of the model in terms of various aspects relating to RE. To address these issues, in this study, we propose applying the technique of metamorphic testing (MT) to RE. MT [16; 17] is a property-based testing technique that is well known for alleviating the oracle problem. The key component of MT is metamorphic relation (MR), which encodes the properties of the target system via the relationship among multiple related inputs and outputs. Based on this, MT checks the relations among the related inputs and their outputs against some MRs, and thus it can be conducted without using oracles. Moreover, the testing results of MT can be interpreted with respect to the MR and accordingly reflect the performance of the target system in terms of the relevant properties. Overall, MT-based evaluation for RE can not only alleviate the oracle problem but also help to better understand the characteristics of RE models from various viewpoints. To apply MT to RE models, we identify a total number of eight MRs. These MRs focus on different aspects of the task of relation extraction. We further demonstrate the effectiveness of MT by conducting experiments on three RE models.