# Graphol: A Graphical Language for Ontology Modeling Equivalent to OWL 2
## 6 User Evaluation Study
### Structure of the Study  
Here we provide the details for the evaluation studies of beginners and experts, which were conducted separately, on different dates. **Beginners:**  
1. _Introduction and brief Graphol tutorial (15 min)_: a general introduction to the purpose of the experiment, and a brief tutorial on ontologies and on the Graphol language. 2. _Brief OWLGrEd tutorial (15 min)_: a brief tutorial on the OWLGrEd language. 3. _Brief user background questionnaire (5 min)_: participants had to answer a brief questionnaire in which they were asked to provide some personal background information, as well as to rate their knowledge of conceptual modeling and ontologies on a scale from 1 to 5 (with 1 indicating extremely low and 5 extremely high expertise), to indicate how many years they had of experience with ontologies (if any), whether they were familiar with some of the more popular ontology editors and knowledge representation and conceptual modeling formalisms, and whether they had any experience with ontologies in real-life scenarios or in manually creating or editing ontologies. 4. _LUBM comprehension tasks (40 min)_: each user was asked to answer ten questions on the LUBM model they were provided. Half of the users were provided a Graphol version of the LUBM model, and half an OWLGrEd version. 5. _Pizza comprehension tasks (40 min)_: each user was asked to answer ten questions on the Pizza model they were provided. Half of the users (those which were provided the OWLGrEd version of the LUBM model) were provided a Graphol version of the Pizza model, and half an OWLGrEd version. 6. _Ex-post survey (10 min)_: after all the comprehension tasks were completed, we asked the participants to compile a short survey regarding their experience. The survey required the users to rate, on a scale from 0 to 4, the general difficulty of the comprehension tasks, the difficulty of learning the Graphol and OWLGrEd symbols, the difficulty of using Graphol and OWLGrEd to read ontologies, and, optionally, to indicate aspects of Graphol and OWLGrEd which they particularly liked, or that they would like to see improved. **Experts:**  
1. _Introduction and brief_ Graphol _tutorial (30 min)_: participants were given the same introductory tutorial on ontologies and on the Graphol language as the beginners, with the addition of some more complex features on the Graphol language which were featured in the expert questionnaire but not the beginner questionnaire. 2. _Brief user background questionnaire (5 min)_: we asked the participants to fill out the same background questionnaire given to the beginners. 3. Graphol _comprehension tasks (35 min)_: after completing the introductory part on Graphol, each user was asked to answer ten questions on the Graphol model of the Pizza ontology they were provided. 4. Graphol _editing tasks (35 min)_: we asked each participant to perform ten editing tasks on the Graphol model of the Family ontology they were provided. 5. _Ex-post survey (5 min)_: after carrying out both the comprehension and editing tasks, the users were asked to fill out a brief survey, analogous to the one given to the beginners. We now discuss some more detailed aspects of the study. All participants, in support of their tasks, were provided with documentation regarding the languages in play for that specific task. Specifically, the questionnaire included some cheat sheets which recapped the symbols of the Graphol and the OWLGrEd language (the latter only for tests carried out by beginners) and their meaning, along with some examples of the representation of some of the most common ontology expressions and assertions in the two formalisms. Additionally, users were provided with a printout of the slides of the introductory tutorials. - The order in which the tasks were presented in the questionnaires was intentionally random, i.e., not linked to the expected difficulty of each task.