# NAS-CRE: Neural Architecture Search for Context-Based Relation Extraction
## 2 Related Work  
Recently, relation extraction has become a research hotspot and is a key step in building knowledge graphs. In general, there are three mainstream solutions to the information extraction problem: supervised methods [17], unsupervised methods [18], and semisupervised [19] methods. Among them, supervised methods, which use a set of samples of known classes to train the model to meet the performance requirements, perform best. Early existing methods used feature-based and kernel-based methods to extract relationships [20; 21]. With the development of neural networks, researchers have tried to use different network structures to improve the performance of relation extraction. [PERSON] et al. and [PERSON] et al. [22; 23] proposed extracting vocabulary and sentence-level features using a convolutional neural network (CNN). [PERSON] et al. and [PERSON] et al. [24; 25] added attention to better identify ambiguities in sentences. CNN is too simple, but it made a groundbreaking breakthrough in relation extraction. Later, recurrent neural networks (RNNs) appeared based on researchers' perspectives. [PERSON] et al. [26] concatenated a bidirectional RNN and used a ranking loss function to improve accuracy. [PERSON] et al. [10; 27; 28] used a variant of RNN, the LSTM network, to achieve this task. As a result, RNN has become a classic method for solving relation extraction tasks. With the development of graph convolutional networks (GCNs), many researchers have begun to use improved graphs as feature extraction layers for information extraction. For example, [PERSON] et al. and [PERSON] et al. [29; 30; 31] used incrementally generated graphs to solve the problem of information extraction. [PERSON] et al. and [PERSON] et al. [32; 33] added a graph convolutional network (GCN) to the syntax parse tree, which is conducive to relation extraction. However, the above work is based on the empirical design of the network structure, which artificially increases parameter constraints, such as specifying the use of CNN or a variant of RNN, LSTM. These designed networks ignore the impact of structural changes and may not be optimal. Therefore, these models cannot search for the optimal network structure and cannot accurately capture relationships within sentences. Most of the above models directly use sentences to generate word vectors and require a large amount of data for training to prevent the model from overfitting. However, these data require human resources to label, which is not a good use of time. In this case, pretraining on many unlabeled corpora is particularly important. As a result, different pretrained models have begun to appear in the literature. The pretrained model learns general semantic knowledge from a large corpus and migrates it to downstream tasks to improve low-resource tasks [16]. Pretraining enables the model to have generalization ability. Furthermore, a pretrained model is equivalent to a kind of regularization, which can prevent the model from overfitting downstream tasks [34]. After the first pretrained model appeared, embeddings from language models (ELMo) [35] and bidirectional encoder representations from transformers (BERT) [36; 37] were proposed to generate word vectors. However, BERT [36], which uses a [MASK] in pretraining, also has shortcomings. For example, the use of a [MASK], which is an artificial symbol, is invalid in the actual data when fine-tuning is used and results in the inconsistency of fine-tuning in the pretraining step. Another disadvantage of using a [MASK] is that it is assumed that the predicted tokens are independent of each other, which results in unmasked tokens. For information extraction tasks that need to consider other relations in a sentence, independent mechanisms cannot relate multiple entity pairs in a sentence. To solve this problem, a new method to fuse AE and the AR language models, called XLNet, learns from the bidirectional context, avoiding the drawbacks brought by the mask method in the AE language model [38].