# Constructing Uyghur Commonsense Knowledge Base by Knowledge Projection
## 4 Experiment
### Experiment
#### 4.2.2 Bilingual Semantic Similarity Scoring Model Analysis  
1. Semantic Accuracy We obtain 143,900 translated Uyghur facts for 63,400 Chinese facts in the dictionary-based translation step. By using the hand-crafted rule template, we can generate the 143,900 Uyghur-Chinese \begin{table}
\begin{tabular}{c c c c c c c} \hline \hline \multirow{2}{*}{**Method**} & \multicolumn{3}{c}{**Source (Chinese)**} & \multicolumn{3}{c}{**Translated (Uyghur)**} & \multirow{2}{*}{**Relation**} \\ \cline{2-2} \cline{4-7}  & **Head** & **Tail** & **Fact** & **Head** & **Tail** & **Fact** & **Count** \\ \hline Original & 67,464 & 85,832 & 369,687 & & 0 & 24 \\ Translation & 12,708 & 15,373 & 99,649 & 24,863 & 27,921 & 2,944,802 & 23 \\ Back-translation & 7336 & 8089 & 63,406 & 8835 & 9873 & 143,926 & 17 \\ \hline \hline \end{tabular}
\end{table} 
Table 2: Experimental results of dictionary-based entity translation. parallel sentences and score the semantic similarity of each sentence by trained bilingual semantic similarity scoring model. There is no publicly available test set for Uyghur-Chinese bilingual phrase similarity measurement, so we randomly select 2000 scored bilingual sentences, using a combination of automatic test and manual check, to get the accuracy of the semantic scoring model. Automatic Test:  
We also focus on whether our model can recognize correct bilingual phrases; in other words, we assign higher semantic similarity scores for the correct bilingual phrases. We use semantic accuracy metrics for this evaluation, which is mentioned in [25]. Formally, given a pair of correct bilingual phrases \((f,e)\) and its incorrect counterpart \((f,e^{-})\) (replaced with a non-translation target phrase) or \((f^{-},e)\) (replaced with a non-translation source phrase), the semantic accuracy (\(SAcc\)) of the bilingual phrase is defined as follows: (we take \((f,e^{-})\) for example)  
\[SAcc=\left\{ \begin{array}{ll}True&\quad\text{if }s(f,e)>s(f,e^{-})\\ False&\quad\text{otherwise}\end{array} \right. \tag{15}\]  
Manual Check:  
We use crowdsourcing to check semantic scores, detailed as follows:  
* As we use cosine similarity as a scoring metric, whose values are distributed from \(-1\) to \(1\), we set zero as the semantic similarity threshold. * Workers check the Uyghur facts with labels: (1) \"True, makes sense in every context\", (2): \"False, does not make sense, or does not make sense in some contexts\". * Each Uyghur fact is judged by three workers. * We aggregate the collected judgments by taking the median. Finally, we kept the Uyghur facts that have been through the automatic and manual test verification. Performance of the scoring model is shown in Figure 3. It can be observed that the model can achieve 94.75% accuracy for our task, which works well for many relationship templates, except for _SymbolOf_ and _Synonym_ relationship. Figure 3: Accuracy of bilingual phrase semantic similarity model for each relation type. Error Analysis  
After analyzing the semantic scoring result, we find two types of errors, as follows:  
* Unknown Word Error: Although we have pretrained word embeddings on a fairly large corpus, we also find that being an agglutinative language, Uyghur still has some words that could not be included. They affect the accuracy of the model due to random initialization. For example, the words _visible_(politician) and _visible_(president) could not get a correct embedding when training and testing, as the sentence which contains this word gets a low score. * Template Error: We define a single template for each relationship type, which works well for most facts. However, for some verbs, the dictionary-translated entities format does not match with the hand-crafted template, so it generates ungrammatical sentences, especially for Uyghur. For example, Table 3 shows the generated sentences of grammatical errors for the _Causes_ relation.