# Deep Semantic Parsing with Upper Ontologies
## 5 Results  
The harmonic mean between precision and recall (F1) is used in all experiments because it is the most commonly used metric in the WSD and SRL literature to measure classification results. The dataset consists of several well-known datasets plus the dataset that was created during this project. The following list shows all the experimental datasets:  
1. The WordNet Unified Evaluation Framework [27]. From this framework, six data sets were used: Senseval-2, Senseval-3, SemEval-07, SemEval-13, SemEval-15 and SemCor. Additionally, the system adds descriptions of WordNet synsets. 2. The Berkeley FrameNet project. An annotated example corpus based on the theory of frame semantics. 3. During this research project, _The Hound of the Baskervilles_ was manually annotated using four annotation systems (see Supplementary Materials): (1) WordNet, (2) FrameNet, (3) SUMO ontology concepts, 4) a set of objects and their references in the scene. Table 4 shows the results of our WordNet sense disambiguation system (BERT1024 k-NN) for these datasets. The semantic analysis method presented in this paper parses only nouns and verb, so the system details results per part of speech. As a benchmark for this experiment, MFS [27] and LMMS2348 (BERT) [28] were used. LMMS2348 is similar to the method proposed in this paper, but it uses additional embedding vectors. A smaller vector size of the BERT embeddings was also used (BERT-Medium512 k-NN). From this experiment, one can observe that the book dataset shows a higher F1 score compared to the Senseval dataset. This can be explained by the fact that the Senseval dataset is more diverse and was created to test the word sense disambiguation with a large set of different words. The book's dataset has a smaller vocabulary and this fact may explain some of the differences in the F1 score. Another interesting conclusion from this experiment can be drawn by noting two facts: firstly, all embedding methods outperform the MFS approach by a significant margin, and secondly, the difference between embedding methods is within a few percentage points. This explains why the system chose BERT1024 k-NN as the final step implemented in the natural language processing pipeline. Even if the LMMS2348 (BERT) method performs slightly better, it takes up more than twice the memory space and more than quadruples the processing time in our implementation. WSD has a small part in this text-to-3D framework, and the system has to consider the performance of each component to obtain reasonable processing resources for the whole framework. There is another reason why there is no need to account for a difference of a few percentage points in the F1 score to be an important argument in favor of a more complex \begin{table}
\begin{tabular}{c c c c} \hline \hline
**Method** & **POS** & **Senseval All** & **Book Dataset** \\ \hline \multirow{2}{*}{MFS} & Verb & 49.6 & 52.2 \\  & Noun & 67.5 & 69.1 \\ \hline \multirow{2}{*}{LMMS2348 (BERT)} & Verb & 64.9 & 65.2 \\  & Noun & 76.7 & 78.6 \\ \hline \multirow{2}{*}{BERT-Medium512 k-NN} & Verb & 62.5 & 62.9 \\  & Noun & 75.1 & 77.3 \\ \hline \multirow{2}{*}{BERT1024 k-NN} & Verb & 63.9 & 64.7 \\  & Noun & 76.4 & 78.1 \\ \hline \hline \end{tabular}
\end{table} 
Table 4: WordNet sense disambiguation results (F1 score). model. WordNet has over 100,000 synsets and the system needs far fewer concepts to create a few simple objects in a 3D environment. The analysis of several results of word disambiguation in which the system made a mistake showed that these results are not an error from the point of view of the upper ontology. To be more precise, the project team investigated the verb \"_increase_\" when the system made a mistake and the MFS system disambiguated correctly. There are two senses of the verb \"_increase_\" in the WordNet dictionary: (1) \"_increase%2:30:00::_\"--(become bigger or greater in amount) (2) \"_increase%2:30:02:_\"--(make bigger or more)_.