# Joint Extraction of Multiple Relations and Entities from Building Code Clauses
## 2 Methodology
### Joint Extraction Model
#### 2.2.4 Subject Extractor  
The subject extractor is responsible for perceiving all candidate subject entities for the target SPO triplets. In Chinese code clauses, a domain-specific named entity is frequently composed of multiple characters. For example, the named entity \"\(\%\)\(\%\)\(\%\)\(\%\)\", comprising three characters, means \"transformer\". To identify such entities, we first identify the start positions of the candidate subject entities and then identify their end positions. Specifically, two tagging processes are employed: one for the start position and the other for the end position. When a character is tagged as a start or end position, its corresponding entity type is also labeled. In addition, the start position tagging information is used as input to the end position tagging process to achieve more accurate predictions. Figure 3, Part 3, illustrates the processing procedure of the subject extractor. First, a Bi-LSTM model is used to extract task-specific feature vectors from the task-shared feature vectors. Subsequently, a multihead self-attention model is applied to the feature vectors to tag the start position of each subject entity along with its entity type. Self-attention is a method of encoding sequences of vectors by relating these vectors to each other based on pairwise similarities [34]. By means of self-attention mechanism, a model can determine which characters are important for specific tasks. Essentially, an attention function is a mapping function that maps a query vector \(\mathbf{Q}\) and a set of key-value vector pairs (\(\mathbf{K}\), \(\mathbf{V}\)) to an output. In this case, the scaled dot-product attention is adopted (see Equation (3)). The query vector of a character is applied to the key vector of each other character in the sentence and then scaled by the dimensionality of the hidden units. Subsequently, a softmax function is utilized to compute the pairwise similarity between the query character and the key features of each other character of the sentence. The softmax is a probabilistic function that takes real numbered inputs and outputs a probability number. This function is directly imported from the TensorFlow library. Finally, the attention result can be calculated by taking the dot product of the softmax output and the value vector of each character. \[\text{Attention}(\mathbf{Q},\mathbf{K},\mathbf{V})=\text{Softmax}\left(\frac{\mathbf{Q}\mathbf{K }^{T}}{\sqrt{\mathbf{d}}}\right)\mathbf{V}, \tag{3}\]  
where \(\mathbf{Q}\in\mathbb{R}^{n\times 2d_{h}},\mathbf{K}\in\mathbb{R}^{n\times 2d_{h}}\), and \(\mathbf{V}\in\mathbb{R}^{n\times 2d_{h}}\) are the query matrix, key matrix and value matrix, respectively. \(\text{Attention}(\mathbf{Q},\mathbf{K},\mathbf{V})\) is the final attention value calculated by pairwise similarity scores and value vector. \(\mathbf{h}^{se}=[\mathbf{h}_{1}^{se},\mathbf{h}_{2}^{se},\dots,\mathbf{h}_{n}^{se}]\) denotes the task-specific features output by the Bi-LSTM layer. In this paper, \(\mathbf{Q}=\mathbf{K}=\mathbf{V}=\mathbf{h}^{se}\). \(\mathbf{d}\) is the dimensionality of the hidden units of the Bi-LSTM layer, which is equal to \(2d_{h}\). Furthermore, the multihead attention model employs multiple different linear layers to linearly map vectors \(\mathbf{Q}\), \(\mathbf{K}\), and \(\mathbf{V}\) and then focuses on obtaining the multihead attention. The intuition behind the multihead attention is that applying the attention multiple times may learn more abundant features than single attention (also known as a head) in the sentence [34]. If the multihead attention contains \(m\) heads, the \(i\)-th attention head can be calculated using Equation (4). And the final multihead attention is the concatenation of all heads as shown in Equation (5).