# Enriching Knowledge Base by Parse Tree Pattern and Semantic Filter
## 6 Experiments
### Performance of Semantic Filter  
The proposed semantic filter is based on a composite similarity of WordNet-based similarity and word-embedding similarity. Thus, we compare the composite similarity with each base similarity to see the superiority of the semantic filter. In addition, many lexical sequence representations of patterns remove irrelevant patterns based on a pattern frequency. Thus, a frequency-based filter is also compared with the proposed semantic filter.  
For each relation, parse tree patterns are generated using all seed triples and the Wikipedia corpus. As a result, 47,390 parse tree patterns are generated. Thus, one relation has 4739 patterns on average. Then, four filters were applied to sort the patterns according to their similarity or frequency. Since it is impractical to investigate the correctness of 47,390 patterns manually, the correctness of top 100 patterns by each filter is checked. Figure 5 shows the average top-\(K\) accuracies of the four filters. In this figure, 'WordNet + Word Embedding' is the proposed semantic filter, 'WordNet Only' and 'Word Embedding Only' are two base filters, and 'Frequency-Based' is the frequency-based filter used in OCLIIE [9]. 'WordNet + Word Embedding' outperforms all other filters for all \(k\)'s. In addition, the difference between 'WordNet + Word Embedding' and other filters gets larger as \(k\) increases. These results imply that the proposed semantic filter preserves high-quality patterns and removes irrelevant patterns effectively.  
Among the ten relations, the results for _deathPlace_ show the lowest accuracy. As shown in Figure 5(a), the accuracies of _deathPlace_ are below 50% for all filters. In the knowledge base, the concepts _Person_ and _Location_ are usually used as the domain and the range of _deathPlace_, respectively. However, they are often used for many other relations such as _birthPlace_ and _Nationality_. Thus, even if a number of patterns are generated from sentences with _Person_ as a subject and _Location_ as an object, many of them are not related at all to _deathPlace_. For instance, a parse tree pattern  
\[[\text{{Subject}}]\leftarrow\text{nsubj}\leftarrow[\text{live}]\rightarrow\text {prep-in}\rightarrow[\text{{Object}}].\]  
is generated from a sentence _\"[PERSON] lived in Ansbach from 1830 to 1833.\"_ with a seed triple \(\langle\text{{[PERSON]}},\text{{deathPlace}},\text{{Ansbach}}\rangle\). This pattern is highly ranked in our system, but its meaning is \"_Subject_ lives in _Object_\". Thus, it does not deliver the meaning of a death location.  
When word embedding similarity is compared with WordNet-based similarity, it is more accurate than WordNet-based similarity. As seen in Figure 5, its accuracy is always higher than that of WordNet-based similarity for all \(k\)'s. However, its accuracy is extremely low with the relation _spouse_ as seen in Figure 5(b). Such extremely low accuracy happens when the similar words of a relation at the word embedding space are not synonyms of the relation. The similar words of _spouse_ in WordNet are its synonyms like _'wife'_ and _'husband'_, but those at the word embedding space are _'child'_ and _'grandparent'_. Even if _'child'_ and _'grandparent'_ imply a family relation, they do not correspond to _spouse_. Since the proposed semantic filter uses a combination of WordNet-based similarity and word embedding similarity, the problem of word embedding space is compensated by WordNet-based similarity. Figures 7 and 8 show Top-K accuracies of all relations except _deathPlace_ and _spouse_. For most relations, the semantic-based scoring achieves higher performance than the frequency-based scoring.  
Figure 5: Average top-\(K\) accuracies of semantic and frequency-based filters.  
Figure 6: Average top-\(K\) accuracies for _deathPlace_ and _spouse_.  
Figure 8: Average top-\(K\) accuracies for _location_, _publisher_, _religion_ and _team_.  
Figure 7: Average top-\(K\) accuracies for _artist_, _board_, _crosses_ and _field_.