# Joint Extraction of Multiple Relations and Entities from Building Code Clauses
## 3 Experiment
### Experimental Results and Analysis
In addition, [PERSON]'s model [28] can predict only a fixed number of triplets, and not all triplets can be extracted; [PERSON]'s model [38] fails to extract entities consisting of multiple words, and the final results were significantly influenced by the word segmentation performance; and [PERSON]'s work [30] assumes that every pair of entities has an associated relation, causing the model to be misled by many nonrelated entity pairs. To further demonstrate the contributions of some of the components illustrated in Figure 3, one component at a time was intentionally removed to explore the impact on performance. Concretely, the word-level embedding, character-level embedding, sentence-level feature vectors, start tagging feature vectors, and joint learning framework were investigated in this way. Specifically, the start tagging feature vectors were removed when predicting the end positions, and the joint learning framework was removed by training the subject extractor and the object and predicate extractor separately, without parameter sharing. The ablation results presented in Table 6 indicate the following: (1) The integration of the character-level and word-level embedding methods is useful for capturing the prior semantic information of each character appearing in the code clauses. (2) Introducing sentence-level feature vectors is an effective means of further encoding sentence information for use in prediction. (3) Considering the predicted start positions when predicting the end positions is beneficial, as implied by the 3.5% decline in the F-measure score observed when removing the start tagging features from the end position prediction process. (4) The joint learning framework is also effective, as implied by the 6.4% drop in the F-measure score seen when the subject extractor and the object-and-predicate extractor are trained separately. \begin{table}
\begin{tabular}{c c c c c c c} \hline \hline
**Predicate Type** & **In Gold Standard** & **Extracted** & **Correctly Extracted** & **Precision** & **Recall** & **F-Measure** \\ \hline hasObject & 305 & 298 & 243 & 0.8154 & 0.7967 & 0.8060 \\ hasProperty & 406 & 388 & 357 & 0.9201 & 0.8793 & 0.8992 \\ hasFeature & 279 & 266 & 240 & 0.9023 & 0.8602 & 0.8807 \\ AccessTo & 26 & 19 & 17 & 0.8947 & 0.6538 & 0.7556 \\ Within & 42 & 44 & 38 & 0.8636 & 0.9048 & 0.8837 \\ Outside & 21 & 15 & 13 & 0.8667 & 0.6190 & 0.7222 \\ Between & 115 & 108 & 95 & 0.8796 & 0.8261 & 0.8520 \\ AdjacentTo & 30 & 23 & 21 & 0.9130 & 0.7000 & 0.7925 \\ NotLessThan & 278 & 290 & 266 & 0.9172 & 0.9568 & 0.9366 \\ NotGreaterThan & 78 & 77 & 69 & 0.8961 & 0.8846 & 0.8903 \\ LessThan & 26 & 27 & 18 & 0.6667 & 0.6923 & 0.6792 \\ GreaterThan & 27 & 27 & 19 & 0.7037 & 0.7037 & 0.7037 \\ EqualTo & 17 & 20 & 14 & 0.7001 & 0.8235 & 0.7568 \\ MultipliedBy & 24 & 17 & 16 & 0.9412 & 0.6667 & 0.7805 \\ \hline Total & 1674 & 1619 & 1426 & 0.8808 & 0.8519 & 0.8661 \\ \hline \hline \end{tabular}
\end{table} 
Table 4: Precision, recall, and F-measure results obtained through model testing. \begin{table}
\begin{tabular}{c c c c} \hline \hline
**Model** & **Precision** & **Recall** & **F-Measure** \\ \hline [PERSON] et al. [36] & 0.3748 & 0.3202 & 0.3454 \\ [PERSON] et al. [37] & 0.5039 & 0.4253 & 0.4613 \\ [PERSON] et al. [28] & 0.6704 & 0.6099 & 0.6387 \\ [PERSON] et al. [38] & 0.7098 & 0.6780 & 0.6936 \\ [PERSON] et al. [30] & 0.7440 & 0.7640 & 0.7539 \\ \hline Our Model & 0.8808 & 0.8519 & 0.8661 \\ \hline \hline \end{tabular}
\end{table} 
Table 5: Comparison of results on the test set.