Question: Can the verbalization of axioms generated by LD2NL be easily understood by domain experts?
Evidence: We evaluated our approach in three different experiments based on human ratings. We divided the volunteers into two groups—domain experts and non-experts. The group of domain experts comprised 66 persons while there were 20 non-experts forming the second group. In the first experiment, an OWL axiom and its verbalization were shown to the experts who were asked to rate the verbalization regarding the two following measures according to gardent2017creating: (1) Adequacy: Does the text contain only and all the information from the data? (2) Fluency: Does the text sound fluent and natural?. For both measures the volunteers were asked to rate on a scale from 1 (Very Bad) to 5 (Very Good). The experiment was carried out using 41 axioms of the Koala ontology. Because of the complexity of OWL axioms, only domain experts were asked to perform this experiment.
In the second experiment, a set of triples describing a single resource and their verbalization were shown to the volunteers. The experts were asked to rate the verbalization regarding adequacy, fluency and completeness, i.e., whether all triples have been covered. The non-experts were only asked to rate the fluency. The experiment was carried out using 6 DBpedia resources.In the third experiment, the verbalization of an OWL class and 5 resources were shown to the human raters. For non-experts, the resources have been verbalized as well, while for domain experts the resources were presented as triples. The task of the raters was to identify the resource that fits the class description and, thus, is an instance of the class. We used 4 different OWL axioms and measured the amount of correct identified class instances.
Results In our first series of experiments, the verbalization of OWL axioms, we achieved an average adequacy of 4.4 while the fluency reached 4.38. In addition, more than 77% of the verbalizations were assigned the maximal adequacy (i.e., were assigned a score of 5, see Fig. FIGREF38). The maximal score for fluency was achieved in more than 69% of the cases (see Fig. FIGREF38). This clearly indicates that the verbalization of axioms generated by LD2NL can be easily understood by domain experts and contains all the information necessary to access the input OWL class expression.
Experiments on the verbalization of summaries for RDF resources revealed that verbalizing resource summaries is a more difficult task. While the adequacy of the verbalization was assigned an average score of 3.92 by experts (see Fig. FIGREF39), the fluency was assigned a average score of 3.47 by experts and 3.0 by non-experts (see Fig. FIGREF39). What these results suggest is that (1) our framework generates sentences that are close to that which a domain expert would also generate (adequacy). However (2) while the sentence is grammatically sufficient for the experts, it is regarded by non-domain experts (which were mostly linguists, i.e., the worst-case scenario for such an evaluation) as being grammatically passably good but still worthy of improvement. The completeness rating achieves a score of 4.31 on average (see Fig. FIGREF39). This was to be expected as we introduced a rule to shorten the description of resources that contain more than 5 triples which share a common subject and predicate. Finally, we measured how well the users and experts were able to understand the meaning of the text generated by our approach. As expected, the domain experts outperform the non-expert users by being able to find the answers to 87.2% of the questions. The score achieved by non-domain experts, i.e., 80%, still suggest that our framework is able to bridge the gap pertaining to understand RDF and OWL for non-experts from 0% to 80%, which is more than 91.8% of the performance of experts.
Discussion Our evaluation results suggest that the verbalization of these languages is a non-trivial task that can be approached by using a bottom-up approach. As expected, the verbalization of short expressions leads to sentences which read as if they have been generated by a human. However, due to the complexity of the semantics that can be expressed by the languages at hand, long expressions can sound mildly artificial. Our results however also suggest that although the text generated can sound artificial, it is still clear enough to enable non-expert users to achieve results that are comparable to those achieved by experts. Hence, our first conclusion is that our framework clearly serves its purpose. Still, potential improvements can be derived from the results achieved during the experiments. In particular, we will consider the used of attention-based encoder-decoder networks to improve the fluency of complex sentences.
This clearly indicates that the verbalization of axioms generated by LD2NL can be easily understood by domain experts and contains all the information necessary to access the input OWL class expression
Answer: Yes, it can.