# NAS-CRE: Neural Architecture Search for Context-Based Relation Extraction
## 3 Method
### Relational Feature Representation  
A neural network can be regarded as a black box that can fit any function. As long as there is enough training data, given a specific x, the desired y can be obtained. Using these networks, only one input can be individually processed at a time, and the previous input and the next input are irrelevant. However, some tasks require information that can better handle sequences; that is, the previous input is related to the following input. For example, when we attempt to understand the meaning of a sentence, it is not sufficient to understand each word of the sentence in an isolated manner. Instead, we need to process the entire sequence of the connected words. In a sentence, the part-of-speech prediction of the previous word for the current word is very influential. To solve some similar problems and better process the information of the sequence, RNN is utilized. RNN is one of the most commonly used models when dealing with time series problems using deep learning. There are many application fields of RNNs. As long as the problem of time sequence is considered, RNN can be used to solve it. The reason why RNN has excellent performance on time series data is that RNN will use the hidden node of the \(t-1\) time slice as the input of the current time slice in the \(t\) time slice. The RNN structure is shown in Figure 4. In this figure, \(x_{t}\) represents the input information,  
Figure 3: The principle of the permutation language model in XLNet. The number i represents the i-th token in a sentence. The blue text represents the knowledge predicted from the preceding context. \(h_{t}\) represents the hidden state, and \(A\) represents the RNN structure. Using RNN, the information from the previous time slice is also used to calculate the content of the current time slice, and the current output of the sequence is relative to the previous output. The specific concept is that the network memorizes the previous information and applies it to the calculation of the current output, and the nodes between the hidden layers become connected. The input of the hidden layer includes not only the output of the input layer but also the output of the hidden layer at the previous moment. Although RNNs are widely used, they suffer from long-distance dependencies. The reason for the long-distance dependence is that after the nodes of the neural network have undergone many stages of computation, the features of the previous relatively long time slice have been covered. LSTM is a variant of RNN that has the ability to learn long-term and short-term information. The reason why LSTM can solve the long-term dependency problem of RNNs is because LSTM introduces a gate mechanism to control the flow and loss of features. An RNN is a relatively mainstream network, but it is manually designed. Manually designing RNNs is a laborious and challenging task. NAS is a fully automatic neural structure search method that automatically learns applicable deep neural structures through algorithms [39]. When optimizing the network, NAS can use different search strategies, such as random search, Bayesian optimization, evolutionary algorithms, reinforcement learning, and gradient-based algorithms. In NAS-RNN, we use the common and effective reinforcement learning method to learn an RNN cell. NAS-RNN optimizes the network by maximizing the expected accuracy of the network on the validation set. Compared to manually designed RNNs, NAS-RNN offers the following advantages:  
1. Automatic Search: Traditional manual design methods often require extensive trial and error and significant human expertise. In contrast, NAS-RNN can automatically search for the optimal RNN architecture, alleviating the burden of manual design. By exploring various architectural spaces, it selects the structure best suited to the task, thereby enhancing the performance of the relation extraction model. 2.