# Enriching Knowledge Base by Parse Tree Pattern and Semantic Filter
## 6 Experiments
### Comparison with Previous Work  
In order to show the plausibility of the proposed method, we perform an additional experiment with a new benchmark dataset, NYT, which is generated with Freebase relations and New York Times corpus [36]. The Freebase entities and relations aligned with the sentences of the corpus in the years 2005-2006. The triples generated by this alignment are regarded as training data, and those aligned with the sentences from 2007 are regarded as test data. The training data contain 570,088 instances with 63,428 unique entities and 53 relations with a special relation 'NA' which indicates there is no relation between subject and object entities. The test data contain 172,448 instances with 16,705 entities and 32 relations including 'NA'. Note that 'NA' is adopted to represent negative instances. Thus, the triples with 'NA' relation do not deliver any information actually. Without the triples with 'NA' relation, there remain 156,664 and 6444 triples in training and test data. Table 5 shows simple statistics on the NYT dataset. The proposed method is compared with four variations of PCNN (piecewise convolutional neural network) which have used the NYT dataset for their evaluation [19; 20; 21; 22]. These models are listed in Table 6 in which ATT implies the attention method proposed by [PERSON] et al. [19], nc and cond_opt denote the noise converter and the conditional optimal selector by [PERSON] et al. [20], soft-label means the soft-label method by [PERSON] et al. [21], and ATT_RA and BAG_ATT are the relation-aware intra-bag \begin{table}
\begin{tabular}{c c c c} \hline \hline
**Relation** & \(\theta_{r}\) & **\#Patterns** & **\#New Triples with Filtering** \\ \hline _artist_ & 0.20 & 36 & 961 \\ _board_ & 0.13 & 48 & 3163 \\ _crosses_ & 0.14 & 23 & 234 \\ _deathPlace_ & 0.11 & 58 & 3448 \\ _field_ & 0.12 & 20 & 43 \\ _location_ & 0.21 & 35 & 731 \\ _publisher_ & 0.24 & 58 & 1169 \\ _religion_ & 0.15 & 19 & 382 \\ _spouse_ & 0.30 & 66 & 1970 \\ _team_ & 0.19 & 9 & 421 \\ Sum & - & 372 & 12,522 \\ \hline \hline \end{tabular}
\end{table} 
Table 4: The number of newly extracted triples by parse tree patterns and semantic filter. \begin{table}
\begin{tabular}{c c c c c} \hline \hline
**Relation** & **Lexical + Frequency Filter** & **Lexical + Semantic Filter** & **Parse Tree + Frequency Filter** & **Parse Tree + Semantic Filter** \\ \hline _artist_ & 43\% & 52\% & 92\% & 89\% \\ _board_ & 60\% & 58\% & 89\% & 80\% \\ _crosses_ & 16\% & 31\% & 46\% & 68\% \\ _deathPlace_ & 0\% & 19\% & 0\% & 21\% \\ _field_ & 42\% & 30\% & 8\% & 39\% \\ _location_ & 38\% & 53\% & 80\% & 75\% \\ _publisher_ & 46\% & 49\% & 57\% & 75\% \\ _religion_ & 4\% & 10\% & 38\% & 23\% \\ _spouse_ & 40\% & 37\% & 91\% & 87\% \\ _team_ & 35\% & 43\% & 38\% & 44\% \\ Average & 32.4\% & 38.2\% & 53.9\% & 60.1\% \\ \hline \hline \end{tabular}
\end{table} 
Table 3: Accuracies of newly extracted triples by parse tree pattern and lexical sequence pattern. \begin{table}
\begin{tabular}{c c c c c c} \hline \hline  & **\# Instances** & **\# Unique Sentences** & **\# Relations** & **\# Unique Enities** & **\# Triples** & **\# Unique Triples** \\ \hline _Train_ & 570,088 & 368,099 & 53 & 63,428 & 156,664 & 19,601 \\ _Test_ & 172,448 & 61,707 & 32 & 16,705 & 6444 & 1950 \\ \hline \hline \end{tabular}
\end{table} 
Table 5: Simple Statistics on the New York Times Corpus (NYT) dataset. attention method and inter-bag attention method proposed by [PERSON] et al. (2020). We measure the top-\(K\) precision (Precision@K) where \(K\) is 100, 200, and 300. Table 6 summarizes the performance comparison on the NYT dataset. According to this table, the proposed method achieves comparable performance to the neural network-based methods. PCNN+ATT_RA+BAG_ATT shows the highest mean precision of 84.8% while the proposed method achieves 79.2%. Thus, the difference between them is just 5.6%. The proposed method, however, is consistent against the change of \(K\). All neural network-based methods show about a 10% difference between \(K=100\) and \(K=300\).