# Ontology-Based Framework for Cooperative Learning of 3D Object Recognition
## 5 Validation
### Framework Usability Evaluation  
We evaluate the usability of our framework by simulating a scenario in which a participant wants to command a service robot to look for and grasp an object. In the experiment, 20 non-expert participants were assigned to command a robot to search for different specific objects in the given scene. Figure 12 shows an example of a target object in a given scene. The goal is to find the 3D coordinates in the robot's 3D frame, based on which, an object grasping strategy can be further applied. \begin{table}
\begin{tabular}{c c} \hline \hline
**Class of Object (SUMO)** & **WordNet** \\ \hline DrinkingCup & An open Fluid\_Container that is intended to serve a Beverage to a single person. Note that this class includes both cups with handles and drinking glasses (beer\_glass, beer\_mug, champagne\_flute, coffee\_cup, coffee\_mug, drinking\_cup, flute, flute\_glass, stein). \\ \hline Bottle & A Container whose top is narrower than its bottom, which has no handle, and which is intended to store Liquids (beer\_bottle, bottle, canteen, catsup\_bottle, feeding\_bottle, flask, ketchup\_bottle, nursing\_bottle, water\_bottle, whiskey\_bottle, wine\_bottle). \\ \hline \hline \end{tabular}
\end{table} 
Table 2: Example of SUMO classes for semantic analysis using WordNet. In the experiment, the participant selects an object from among multiple objects within the scene and gives a command to the robot to look for and grasp the object. The simulation starts with the participant defining an object class based on the SUMO ontology. The system then performs semantic analysis to present a description as a label of the object from WordNet (see Table 3). The participant verifies that the label is suitable to represent the selected object. The participant can also elaborate on the physical appearance of the object by specifying a description, in order to populate the knowledge base with more knowledge that could be shared within the framework. The robot simulates the object recognition and object grasping tasks, then responds to the user as to whether the object can be found in the scene and can be grasped or not. Lastly, the participant evaluates the usability of the framework based on System Usability Scale (SUS) metrics [38]. The results show that the user satisfaction with the system had a SUS score of 75.25, as shown in Table 4. \begin{table}
\begin{tabular}{c c c} \hline \hline
**\#** & **System Usability Scale Questionnaire** & **Average** \\ \hline Q1 & I think that I would like to use this system frequently. & 3.60 \\ Q2 & I found the system unnecessarily complex. & 1.50 \\ Q3 & I thought the system was easy to use. & 3.85 \\ Q4 & I think that I would need the support of a technical person to be able to use this system. & 1.85 \\ Q5 & I found the various functions in this system were well integrated & 3.90 \\ Q6 & I thought there was too much inconsistency in this system. & 1.90 \\ Q7 & I would imagine that most people would learn to use this system very quickly. & 3.80 \\ Q8 & I found the system very cumbersome to use. & 1.65 \\ Q9 & I felt very confident using the system. & 4.00 \\ Q10 & I needed to learn a lot of things before I could get going with this system. & 2.15 \\ SUS Score & 75.25 \\ \hline \hline \end{tabular}
\end{table} 
Table 4: Usability test of web-based service application base on System Usability Scale. Figure 12: Examples of object identification a coffee mug in a target scene. This experiment allowed participants to execute queries based on a common understanding of the learned data stored by the system. Without a shared ontological knowledge base, searching for an object without a precise specific class or label specification often results in a larger than necessary search space and is time consuming. On the other hand, a precise specification of a class or label will make the search-space smaller, resulting in less time spent on the search process. Therefore, this study affirms that the proposed ontology-based framework can benefit robots' cooperative learning.