# Understanding student errors in graph query formulation

Amedeo Pachera, Stefania Dumbrava, Angela Bonifati, Andrea Mauri

<figure><img src="image_1.png" /></figure>

## To cite this version:

Amedeo Pachera, Stefania Dumbrava, Angela Bonifati, Andrea Mauri. Understanding student errors in graph query formulation. ACM Transactions on Computing Education, 2025, 25 (3), pp.1-35. 10.1145/3743687. hal-05323375

HAL Id: hal-05323375

https://hal.science/hal-05323375v1

Submitted on 23 Oct 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_2.png" /></figure>
# Understanding Student Errors in Graph Query Formulation

AMEDEO PACHERA, Lyon 1 University, CNRS Liris, France

STEFANIA DUMBRAVA, ENSIIE & SAMOVAR (IP Paris), France

ANGELA BONIFATI, Lyon 1 University, CNRS Liris & IUF, France

ANDREA MAURI, Lyon 1 University, CNRS Liris, France

Query languages are the foundations of database teaching and education practices. The broad adoption of graph databases contrasts with the limited research into how they are taught. Contrary to relational databases, graph databases allow navigational queries with higher expressivity and lack an a priori schema. In this paper, we design a multi-step exploratory user study investigating these peculiarities and how they impact the student's learning process. Focusing on the widely used Cypher graph query language, we studied a new taxonomy for classifying the errors around graph queries and conducted an in-depth analysis. Our investigation highlights several learning barriers, including misunderstanding the semantics of language constructs, the confusion between data and schema, and the pattern-matching mechanism. We expect the lessons learned and the derived best teaching practices to have an impact on database education and to influence future generations of students, researchers, and practitioners working on graph database technologies.

CCS Concepts: • **Human-centered computing** → **Empirical studies in HCI**; • **Applied computing** → **Education**.

Additional Key Words and Phrases: graph query languages, education, teaching computer science

## ACM Reference Format:

Amedeo Pachera, Stefania Dumbrava, Angela Bonifati, and Andrea Mauri. 2025. Understanding Student Errors in Graph Query Formulation. *ACM Trans. Comput. Educ.* 1, 1, Article 1 (January 2025), 34 pages. https://doi.org/10.1145/3743687

## 1 Introduction

Graph databases are a novel type of NoSQL store in which data is natively represented as a graph. In particular, the most widely used model is the *property graph* one [6], i.e., a multi-labeled multi-graph with key-value properties attached to both nodes and edges. Graph databases are becoming increasingly popular and are being adopted in different domains, such as the life sciences, finance, security, logistics, and planning, to name a few [31]. Hence, it is paramount to teach students how to use this technology and, in particular, to understand where they have the most difficulties.

Many works have investigated the errors that students make when formulating relational (SQL) queries, as relational databases have been around since the 70s. In particular, these works focus on classifying these errors [2, 3, 12, 13] and investigating their root causes [23, 25] by integrating user studies with the insights from experts [24]. However, students' struggles with NoSQL and especially with graph databases remain under-researched. Works on this end focus on large-scale quantitative studies to investigate the query language features with which students struggle the most [5, 14],

Authors' Contact Information: Amedeo Pachera, Lyon 1 University, CNRS Liris, Lyon, France, amedeo.pachera@univ-lyon1.fr; Stefania Dumbrava, ENSIIE & SAMOVAR (IP Paris), Paris, France, stefania.dumbrava@ensiie.fr; Angela Bonifati, Lyon 1 University, CNRS Liris & IUF, Lyon, France, angela.bonifati@univ-lyon1.fr; Andrea Mauri, Lyon 1 University, CNRS Liris, Lyon, France, andrea.mauri@univ-lyon1.fr.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from permissions@acm.org.

© 2025 Copyright held by the owner/author(s). Publication rights licensed to ACM.

Manuscript submitted to ACM
ignoring however the reasons why these errors occur. In addition, we argue that replicating the studies done on SQL
is not sufficient to understand the difficulties students might have. Indeed, graph database query languages (such as
Cypher) differ from SQL-like query languages for multiple reasons; for instance, they have a completely different
syntax and semantics. Data stored in graph databases lacks an a priori schema (undefined or unknown) contrary to SQL
databases. Moreover, the fundamental construct of graph query languages relies on graph pattern matching as opposed
to join-based predicates and thus may need to be approached differently.

To exemplify these differences, let us consider the queries in Figure 1. To answer the following question: "Which are the names of all the actors who played in the movie The Matrix?", the user needs to approach the problem differently when using a SQL database (e.g. Figure 1(a)) or a graph database (e.g. Figure 1(b)). For instance, when writing the SQL query - Figure 1(c) - one needs to know how the data is structured (e.g., the fact that there are tables storing information regarding actors and movies) and think about the join conditions (e.g. when joining Movie, Acted_in, and Actor tables). Conversely, when writing the Cypher queries, one has to leverage navigational patterns on the graph (e.g. the edge pattern -[:ACTED_IN]-> as in Figure 1(d)). The latter figure shows a variant of the query where the structure of the data is known (e.g., the node labels Actor, Movie, and the edge label ACTED_IN). One can also express the same query without the labels, as in Figure 1(e), obtaining the same results <fnref n="1" />.

Given the above challenges associated with graph query languages, it is crucial to identify the primary barriers to
learning. Some of them are shared with SQL (as they are linked to learning how to query a database in general), while
others rise specifically from the graph data model or from the particular syntax of graph query languages. In particular,
we are interested in answering the following research questions:

* RQ1: In which aspects of the Cypher language do the students struggle the most? Here we are interested in identifying specific constructs that are more difficult for the students - like highlighted in [14] - or if there are other learning barriers.

* RQ2: What are the main causes of student errors? Here we are interested in understanding what are the reasons behind the errors. Previous work highlighted conflicts with previous knowledge and lack of practice [24], or unfamiliarity with new elements introduced by the language [4, 5, 14].

To this end, we designed a four-phase user study with students enrolled in an international master's program in France. In the study, the students had to write Cypher queries of varying complexity to solve different tasks such as schema discovery, analysis, and aggregation. Next, to better understand the causes of the participants' errors, we designed an additional post-study survey, and we performed live interviews where we asked the participants to reason about their mistakes. More details on the user study are presented in Section 4.

In this paper, we focus on Neo4j's [27] Cypher [16] language, due to its popularity [32] and support in numerous
other databases, e.g., SAP HANA [29], Amazon Neptune [10], Memgraph [22], RedisGraph [19], and AgensGraph [17].
Although our study is centered around Cypher, we argue that some of the results would apply to graph queries in
recent standard query languages and, specifically, to GQL queries, due to the inherent similarities between the two
languages. The latter standard query language has recently appeared [18] and we believe that teaching standard graph
query languages will be a compelling challenge for database education in the upcoming years [15].

In our work, we make the following main contributions.

* We build a new taxonomy of graph query errors centered on constructs of the Cypher graph query language.

<footnote n="1">¹Notice that the exact result set of the query depends on the data stored in the database. In general, removing labels from a graph query could return more results.</footnote>
<figure><img src="image_3.png" /><figcaption>Fig. 1. Example of a relational (a) and graph (b) data model storing the same information and three queries answering the same information need (e.g., list the names of the actors who played in the movie "The Matrix"). The query (c) is written in SQL while (d) the equivalent Cypher formulation. The last query (e) is the same as (d) without labels.</figcaption></figure>

* We carry out a quantitative analysis of the mistakes. We provide statistics in relation to the type of errors, the affected clauses of the query language, and previous knowledge.

* We perform a qualitative analysis focusing on the hand-written queries and highlight the error-prone constructs (WITH, COLLECT, pattern definition, and schema misunderstanding).

* We investigate the causes of the errors with an additional survey and live interviews with the students.

* We synthesize a list of best practices for database educators to improve the teaching processes and address the issues around learning graph query languages.

To the best of our knowledge, this is the first study of this kind for graph query languages. This work contributes to a better understanding of where educators should focus their efforts and, ideally, also of how the design of the graph query languages themselves can be improved.

The remainder of the paper is structured as follows. Section 2 gives a brief introduction to the property graph model and Cypher. Section 3 describes the related works. Section 4 explains the research procedure and the setting, Section 5 presents the results of the study, while Section 6 discusses the contextual factors of the errors. Section 7 delves deeper into the causes of the errors with semi-structured interviews, and discusses the implications for the education of graph query languages. Finally, Section 9 concludes.

## 2 Property Graphs and the Cypher Graph Query Language

The Property Graph model is an expressive formalism in which nodes and edges are rich objects characterized by multiple labels and a list of key-value properties (a more formal definition can be found in [11]). The schema of a property graph consists of node and edge types, formed by node and edge labels along with the list of property types, respectively. For example, Figure 1(b) shows a property graph whose schema consists of two node types, with labels MOVIE (in orange) and ACTOR (in red) between which there is an edge type with label ACTED_IN. Also, one can notice
that the ACTOR nodes have one property, i.e., name, while the MOVIE node has a set of properties, i.e., runtime, plot, year and title. The graph itself contains the data of four actors who acted in the movie "The Matrix". Cypher [16] is a popular declarative graph query language for writing queries on property graphs. While it was created within the Neo4j company [27] for their database, it was made open source with the openCypher project [1], and it is adopted by a large variety of companies - as mentioned in Section 1. The structure of a Cypher query is :

MATCH <graph pattern>
WHERE <conditions>
RETURN <result>

The core concept of Cypher - and the main difference from SQL - is the graph pattern matching, meaning that data is retrieved by specifying navigational patterns within the graph. The <graph pattern> could be a single node (e.g., (m:Movie), for retrieving all the nodes of type Movie), a relationship (e.g., (a:Actor)-[p:ACTED_IN]->(m:Movie), for retrieving the triples (actor, ACTED_IN, movie)) or a path (e.g., (a:Actor)-[p]1,3->(m:Movie), for retrieving all the paths of length three from a node of type Actor to a node of type Movie). Another key difference between Cypher and SQL is its treatment of edge directions. While SQL deals with relationships between tables without considering directional constraints, Cypher explicitly takes edge direction into account when querying relationships between nodes. This allows Cypher to differentiate between directed and undirected relationships, enabling more precise traversal and querying of graph data. Moreover, Cypher adheres to trail semantics [16], meaning that in a given path, relationships (edges) cannot be traversed more than once. This ensures that each edge in a path is unique, which is particularly important for certain types of graph queries, such as those involving traversal without revisiting the same relationship, thus avoiding potential cycles or redundant results. The second part of the query (i.e., the WHERE clause) consists of filtering and the last part consists of projecting and returning the results. For example, in Figure 1(d), to retrieve all the actors who played in the movie "The Matrix" we need to navigate all the ACTED_IN paths between an ACTOR and a MOVIE node. Then, Cypher also provides other constructs to compute aggregation (e.g., COUNT, AVG, etc.), and to perform advanced graph operations (e.g., compute the shortest path between two nodes). Other frequent Cypher constructs are WITH, COLLECT, and UNWIND. The WITH clause is used to chain multiple query parts together, allowing intermediate results to be passed from one part of the query to the next. It functions similarly to SQL's WITH or to subqueries, but in Cypher, it is also used to manage query scope, ensuring that certain aggregations or filtering operations are applied before continuing with the rest of the query. Additionally, WITH is essential for defining variables that are needed later in a query while controlling which variables are visible at each step. The COLLECT function, on the other hand, is an aggregation function that gathers multiple values into a list. This is particularly useful when querying relationships or nodes and one wants to group related data into a single collection. For instance, COLLECT can be used to accumulate all outgoing relationships from a node into a list, which can then be processed or returned as a single unit. When used alongside WITH, COLLECT enables powerful data transformations and groupings within Cypher queries. The UNWIND clause in Cypher is used to transform a list of values into individual rows. This is the inverse of the COLLECT function, which aggregates values into a list. With UNWIND, one can take a collection (such as a list of nodes or relationships) and break it down into separate, distinct elements. Each item in the list is processed as an individual row, which can then be used in subsequent parts of the query. UNWIND is particularly useful when working with collections of data where one needs to treat each element separately. For example, if a query returns a list of nodes and one needs to perform further operations on each node individually, UNWIND allows one to efficiently iterate over each element.
## 3 Related Work

Many works studied what are the most common errors students make when querying relational databases using SQL, in order to understand how educators should teach this query language and on which parts they should focus. Taipalus et al. [33] were the first to develop a unified classification of error types in SQL queries. They developed this categorization by conducting a user study with students to expand concepts related to errors previously defined by the literature [2, 3, 12, 13] identifying the following error categories: syntactical [2], logical [13], semantic [3] errors, and complications [12] (i.e., a correct query written with unnecessary constructs). The study covered both common concepts (e.g., single-table expression, grouping, nesting, etc. [21]) and more ambiguous ones (e.g., correlation names and aliases [33]).

Miedema et al. [23] went beyond error classification investigating what are the common student misconceptions while learning SQL that can lead to errors. With a think-aloud study involving 21 students, they analyzed the students’ thought processes while writing SQL queries, starting from a schema and a natural language prompt. They found students made errors mainly because of conflicting previous knowledge (of SQL or other languages), wrong or incomplete mental models, and the language itself. These findings were confirmed and expanded in a later study [24] where they integrated the perspective of experts on the reasons why errors occur. They found that most errors stemmed from prior misconceptions and insufficient mastery of the language and its use. They also found that errors come from students mismanaging the complexity of the queries they have to write. In another study [25], they highlight how students typically fail on the first try and arrive at a final solution by iterating over their previous attempts, which often leads to over-complicated or incorrect results. Similarly, Migler et al. [26] studied the learning process of 114 students over 5 weeks in designing SQL queries (introduced by a textual description and an indication of the language construct to use). They analyzed quantitative metrics, e.g., success rate, number of attempts, and time spent on a query, and used them to examine which parts of SQL are the most difficult (e.g., set operations, correlated subqueries, full outer join, and relational division).

There are not many works investigating students’ struggles with NoSQL query languages, probably due to the relative novelty of the topic and inclusion in teaching curricula. Alkhabaz et al. [5] investigated which are the most common errors of students in MongoDB. By analyzing ~500 homeworks, they highlighted that students struggled more with the new constructs that are introduced by the language. Chen et al. [14] performed a large-scale quantitative study to investigate which are the most common errors made by students while learning graph databases (in particular the Cypher query language). They analyzed over 40k homeworks where students had to write Cypher queries given the schema of the data and a textual description of the query. They found that students struggle in particular with complex pattern matching, as well as with the WITH clause. Similar findings were reported by Alkhabaz et al. [4] in a recent study comparing students’ errors in SQL, MongoDB, and Cypher.

Li et al. [20] studied if and how the knowledge of different query languages (SQL, MongoDB, and Cypher) transfers. They divided the students into two groups and taught them the query languages in different sequences, always presenting Cypher last. They found that syntax errors decreased over time, but on the other hand, not many students were able to transfer skills between languages. However, they found that teaching SQL first seems to reduce the number of semantic errors in MongoDB. Similarly, Tshukudu and Cutts [34] found that learners switching between programming languages often carry over semantic misconceptions from familiar ones, particularly during the early stages of learning.

While these studies engaged with several students and highlighted where they struggled the most, they did not explore the reasons for these errors, which is the main objective of our work.
<figure><img src="image_4.png" /><figcaption>Fig. 2. Timeline of the study</figcaption></figure>

# 4 Methods

As shown in Figure 2, the entire study was conducted over a period of six weeks. Specifically, the experiment took place in the context of the 2023/2024 master’s course on graph processing at two universities in France, the Lyon 1 University and at the ENSIIE Paris engineering school. The course was the same in both institutions, and the lectures covered the property graph model, graph schemas, and the formulation of graph queries, both simple (involving basic graph patterns) and complex (composed queries, with correlated clauses, navigational patterns involving more than three edges, and aggregates). Each of these topics was illustrated with practical examples that the students were encouraged to reproduce on their machines. At the end of the course, for three consecutive weeks, the students performed the tasks we assigned during the lab activities (Sessions 1-3). Each session lasted 3 hours. We then analyzed and further examined the results for four weeks and prepared the questions for the semi-structured interviews that we conducted during the last lab session (Session 4). The students submitted the answers using a Google Form.

## 4.1 Dataset and Questions

The questions were designed based on the real-world scenario of analyzing highly interconnected, complex genomic data. The reason behind this choice is to expose students to a realistic, specialized use case with which they did not have first-hand experience and hence which was unlikely to generate biases (i.e., some students knew about the use case more than others).

The questions were designed around the fundamental concepts needed to learn graph queries. As shown in Table 1, the questions encompass concepts such as graph pattern matching and aggregation and gradually expose students to increasingly difficult complex query formulation tasks. In addition, they target information required in real-world bio-informatics analyses, as conducted by domain experts with whom we consulted [9].

In addition, for each question, we handcrafted the corresponding correct query (i.e., the ground truth) to have an efficient and objective way to annotate the responses submitted by the participants. Given a query, there could be multiple ways to rewrite it and still obtain the same result (for example by reflecting the pattern in the MATCH clause). For this reason, to evaluate students’ queries we also compared the results.

The considered dataset [9] consists of 10 graphs comprising genome and gene sets shared across different organisms. These are called *pangenome graphs* in the bioinformatics literature [30] and provide a unifying representation amenable to cross-species comparative genomics. Figure 3 captures the schema structure of the dataset as follows. Pangenome nodes are connected to their genomic Family nodes and neighboring and similar nodes of the same label in Modules. Family nodes contain antibiotic resistance annotations. These nodes are classified as belonging to persistent, shell, and cloud Partitions, depending on whether their Genes are prevalent in all, some, or only a few corresponding genomes. Finally, Genes are part of Contigs, associated with particular Genomes, and can be part of RGPs, located in Spots.
<table><caption>Table 1. List of questions together with the related language concepts being tested. Δ = "State also how many answers it returns."</caption><thead><tr><th>Category</th><th>Question</th><th>Involved Concepts</th></tr></thead><tbody><tr><td>Schema Discovery Queries</td><td>Q1: Write a Cypher query that returns the set of all distinct node labels. Δ<br>Q2: Write a Cypher query that returns the set of all distinct relationship types. Δ<br>Q3: Write a Cypher query that returns each distinct node label together with its corresponding set of properties. Δ<br>Q4: Write a Cypher query that returns each distinct relationship type together with its corresponding set of properties. Δ<br>Q5: Write at least three additional Cypher queries that are needed for you to understand the underlying schema structure of the dataset. You can also write more than three queries in case. For each of them, Δ</td><td>MATCH, RETURN, DISTINCT<br>MATCH, RETURN, DISTINCT<br>MATCH, RETURN, DISTINCT<br>MATCH, RETURN, DISTINCT<br>MATCH, RETURN, DISTINCT, Difference between schema and data</td></tr><tr><td>Analytical Queries</td><td>Q6: Write a Cypher query to return each pangenome, together with its associated genes. Δ<br>Q7: Write a Cypher query to return each pangenome, together with its associated spots. Δ<br>Q8: Write a Cypher query to return each pangenome and its associated modules that contain at least a pair of gene families that are more than 80% identical. Δ<br>Q9: Write a Cypher query that returns pairs of inter-pangenome families that both contain annotations and that are more than 80% identical. Δ<br>Q10: Write a Cypher query that returns, for each pangenome, the neighborhood of its gene families containing annotations. Δ</td><td>MATCH, Pattern matching, RETURN, DISTINCT<br>MATCH, Pattern matching, edge direction, RETURN, DISTINCT<br>MATCH, Pattern matching, edge direction, WHERE, RETURN, DISTINCT, COUNT<br>MATCH, Pattern matching, WHERE, RETURN<br>MATCH, Pattern matching, WHERE, RETURN</td></tr><tr><td>Aggregate Queries</td><td>Q11: For every pangenome, write a Cypher query that return its name, together with its number of families that contain annotations. Δ<br>Q12: Write a Cypher query that returns all modules together with the average similarity of the gene families they are comprised of, in descending order. Δ<br>Q13: Write a Cypher query that returns the names of the modules that contain the pairs of gene families that have the minimum and, respectively, the maximum similarity (as indicated by the value of the "identity" property of the "IS SIMILAR" edge connecting them). Δ<br>Q14: Write a Cypher query that returns the top 10 gene families with the highest number of associated modules.<br>Q15: Return the names of the top 10 pairs of similar families (as defined in Q13), whose associated identity and coverage metrics surpass 0.8, together with the names of their partitions (as indicated by the value of the "subpartition" property associated to the nodes of type "Partition").</td><td>MATCH, Pattern matching, WHERE, RETURN, COUNT<br>MATCH, Pattern matching, edge direction, WHERE, RETURN, COLLECT, ORDER BY<br>MATCH, Pattern matching, edge direction, UNION, WHERE, RETURN, COLLECT, ORDER BY, LIMIT<br>MATCH, Pattern matching, WITH, RETURN, COUNT, ORDER BY, LIMIT<br>MATCH, Pattern matching, edge direction, UNION, WHERE, RETURN, ORDER BY</td></tr></tbody></table>

## 4.2 Research Procedure

The user study was composed of four sessions and it was approved by the university ethics review board. In the first one, the participants were first presented with survey questions that collected data about their previous knowledge of relational and non-relational databases (including graph-oriented ones), as well as their proficiency with programming and query languages. Then, in the same session, we asked the participants to write **schema discovery queries**. Without showing Figure 3 we asked them to write queries that would help them infer the underlying dataset schema. Among these, there were queries to retrieve the set of node and relation labels (e.g., Gene and IS_IN_FAMILY) and their properties (e.g., the name of a Gene). While we specifically gave them the description of four such queries, they also had to provide a minimum of three additional queries to help them grasp the schema structure. Finally, we asked the participants to draw the schema, as they understood it, from the results of the query they wrote. Even though the schema is not mandatory in a graph, understanding it is helpful to write complex queries. For this reason, we first tested students' ability to infer it.

In the second session, we showed the participants the schema of Figure 3 and asked them to write five **analytical queries**. These include queries about the genomic data itself, e.g., what Genes belong to a Family?
<figure><img src="image_5.png" /><figcaption>Fig. 3. Graph schema structure of the pangenome dataset.</figcaption></figure>

We also asked the students, on a voluntary basis, to participate in a third session consisting of writing five additional **aggregate queries**. These include queries that gather statistical information, e.g., counting gene families, retrieving average similarities, and ranking associations. The full list of queries asked is reported in Table 1.

After analyzing the responses from the first three parts, we categorized the errors found and designed the semi-structured interview with specific questions to understand the nature and cause of these errors. In the fourth session, we conducted live interviews with the participants to further investigate their misunderstandings and areas of confusion. The interview questions are listed in Appendix D, while the summarized answers are in the supplementary materials [28].

To write and test the queries, the participants used Neo4j Desktop Edition version 4.4.14² on their laptop and submitted their responses in a Google Form. They could make any number of attempts before submitting the queries via the form. Finally, they drew the schema using the Arrow tool<fnref n="3" />. All the questionnaires, together with the responses, are part of the supplementary materials [28].

## 4.3 Participants

In total, there were 68 participants. All of them were enrolled in a Master's program, in computer science or related topics, at the universities. Note that one participant was excluded because they did not consent to be involved in the study. We have also excluded another eight participants because they did not submit the first part of the study (that included the consent form). A total of 59 completed the first part of the study. Of them, 45 participants submitted

<footnote n="2">²https://neo4j.com/download/</footnote>

<footnote n="3">³https://www.apcjones.com/arrows/}</footnote>
answers to the second part, while the volunteers that attended the third part were 13. Finally, 14 students participated in the interviews.

The participants' ages were between 18-24 (N=42) and 25-34 (N=14) years old (three participants did not disclose their age). Generally, they claimed good knowledge of relational databases, with an average of 3.63 (± 0.83) on a five-point Likert scale, and less experience with NoSQL (2.68 ± 0.97) and graph databases (2.36 ± 1.05). Most of them knew Python (N=56) and Java (N=41), and several also stated knowing some graph query languages, mainly SQL/PGQ <fnref n="4" />(N=32) and SPARQL <fnref n="5" />(N=21). The informed consent and the knowledge assessment questions are reported in Appendices A and B.

<table><caption>Table 2. Error taxonomy.</caption><thead><tr><th>Category</th><th>Type</th><th>Explanation</th></tr></thead><tbody><tr><td>Correct</td><td></td><td>The result is correct</td></tr><tr><td rowspan="4">Complication</td><td>Added UNWIND</td><td>Student used UNWIND when it was not necessary</td></tr><tr><td>Added COLLECT</td><td>Student used COLLECT when it was not necessary</td></tr><tr><td>Added WITH</td><td>Student used WITH when it was not necessary</td></tr><tr><td>Additional information</td><td>Student returned information that were not requested (e.g. results count)</td></tr><tr><td rowspan="6">Semantic Pattern</td><td>Wrong Pattern w.r.t. Schema</td><td>The pattern does not exist in the graph schema</td></tr><tr><td>Wrong Pattern</td><td>The pattern is valid but it is not the correct one to answer the question</td></tr><tr><td>Wrong Edge Direction</td><td>The pattern is valid but the edge directions lead to a wrong result</td></tr><tr><td>Wrong Edge Label</td><td>The pattern is valid but it contains a wrong edge (in case of multiple edges between nodes)</td></tr><tr><td>Additional Node</td><td>The pattern is valid but it contains an additional node w.r.t. the correct one for the answer</td></tr><tr><td>Missing Node</td><td>The pattern is valid but it misses a node w.r.t. the correct one for the answer</td></tr><tr><td rowspan="5">Semantic Where</td><td>Wrong Property</td><td>An operator is applied to the wrong property (e.g. Node.A > 0 instead of Node.B > 0)</td></tr><tr><td>Wrong Operator</td><td>The wrong operator is applied to a property (e.g. Node.A > 0 instead of Node.A >= 0)</td></tr><tr><td>Wrong Condition</td><td>The entire condition is wrong (e.g. Node.A > 0 instead of Node.B IS NOT NULL)</td></tr><tr><td>Missing Condition</td><td>A condition is missing leading to more results</td></tr><tr><td>Additional Condition</td><td>An additional condition is applied leading to fewer results</td></tr><tr><td rowspan="8">Semantic Return</td><td>Missing Information</td><td>Some information are missing in the result (e.g. results count)</td></tr><tr><td>Missing LIMIT</td><td>LIMIT is missing leading to a wrong result</td></tr><tr><td>Missing ORDER BY</td><td>ORDER BY is missing leading to a wrong result</td></tr><tr><td>Missing DISTINCT</td><td>DISTINCT is missing in the return clause leading to a wrong result</td></tr><tr><td>Wrong Information</td><td>The result contains the wrong information wrt the question</td></tr><tr><td>Wrong LIMIT</td><td>LIMIT is used incorrectly (e.g. LIMIT 1 instead of LIMIT 10)</td></tr><tr><td>Wrong ORDER BY</td><td>The ordering is applied on the wrong objects/values</td></tr><tr><td>Wrong DISTINCT</td><td>DISTINCT is used incorrectly (e.g. DISTINCT(node.name,id) instead of DISTINCT(node.name))</td></tr><tr><td>Other</td><td></td><td>The query was formulated using Neo4j-specific procedures (i.e., via the CALL statement) rather than standard Cypher syntax.</td></tr><tr><td>SyntaxError</td><td></td><td>The query does not run</td></tr></tbody></table>

## 4.4 Error Taxonomy and Annotation of the Responses

To better understand the challenges students face when learning Cypher, we developed a tailored error taxonomy for conducting both qualitative and quantitative analyses. In contrast to the approach taken by previous studies [14], our method does not involve analyzing query output logs for syntax error classification. Instead, we are interested in analyzing the students' queries to investigate their correctness in obtaining the desired results. As explained in Section 4.2, indeed, students could try their queries multiple times and upload their final versions. Thus, to analyze the errors made by the participants, we extend the work by Taipalus et al. [33]. In particular, we adopt *Complication*, *Semantic*, and *Syntax* errors from their taxonomy, but further extend it to account for specific elements of the graph query language. First, we propose a specialization of the *Semantic* category: *Semantic Pattern* if the error is caused by a mistake in the pattern matching, *Semantic Where* if it is caused by the conditions defined (or missing) in the *WHERE* clause, and *Semantic Return* if the error occurs in the *RETURN* clause. We also propose a fine-grained annotation related to the cause of the error to better investigate the nature of the mistakes. Table 2 lists our taxonomy together with an

<footnote n="4">⁴https://pgsql-lang.org/</footnote>

<footnote n="5">⁵https://www.w3.org/TR/rdf-sparql-query/</footnote>
explanation for each label. We define the additional labels as *Missing+Object/Construct* when a missing construct (COUNT, DISTINCT, etc.) or object (node, variable, condition, etc.) leads to a wrong result, *Wrong+construct* when the construct is misused, and *Added/Additional+Object/Construct* when an additional construct or object leads to a wrong result or a complication. In case of syntax errors, the query does not run, while with a semantic one, the query is executed but returns the wrong results. Finally, *Complication* are correct queries that use unnecessary constructs. Specifically, if a query returns the same results but the pattern in the MATCH clause is written differently, it is labeled as *Correct*, while a query that needlessly uses additional constructs (e.g. WITH, COLLECT, or UNWIND) is labeled as *Complication*.

This taxonomy helps us differentiate error causes. For instance, *Semantic Pattern* errors may highlight problems due to the graph data model, while *Semantic Where* errors may be related to general query languages (including SQL). Finally, complications or errors in the *RETURN* clause may be connected to a wrong use of Cypher syntax.

Two of the authors independently examined and annotated the queries by executing them on the same database used by the participants, any disagreement was solved with a discussion with a third author. If the query did not run and the database returned an error, the query was annotated as having a *Syntax* error. If the query did not terminate or if it returned a result set different from the correct one (w.r.t the ground-truth query) it was annotated as a *Semantic* error. Then, the query was compared with the ground truth to fine-grain the annotation. In the majority of the cases, the error was evident (e.g. if only DISTINCT was missing in the RETURN clause), for the others the queries were re-run iteratively removing constructs to investigate the cause of the error (e.g. removing conditions in the WHERE clause or changing the graph pattern). A query can have multiple labels if it contains multiple independent errors (i.e., errors that pertain to different parts of the query), otherwise, only the one that causes the incorrect result is labeled. For instance, if a query contains both a wrong pattern and misses DISTINCT in the RETURN clause, it is labeled with *Semantic Pattern* and *Semantic Return*. On the contrary, if a query contains an incomplete pattern (*Semantic Pattern, Missing Node*) and it misses a condition on the missing node, it is labeled only with *Semantic Pattern*. If the query returned the correct results but contained additional useless constructs (w.r.t the ground-truth query) it was annotated as a *Complication*. Finally, if the query corresponded to the ground truth, it was annotated as *Correct*. In addition, we use the *Other* annotation in case the participants used database procedures instead of writing Cypher queries (i.e. if they used the CALL command). We chose not to focus on syntax errors, but rather on the fundamental query language principles and core construct usage that are expected to persist. The main reason is that the syntax of graph query languages will be evolving into the recent standards. Also, the students had access to Cypher's debugger and a principled analysis of such syntax errors would require real-time logging. In the case of Q5, where students were asked to write additional queries to gather information about the schema, we counted the number of queries that retrieved information about the schema. To evaluate the schemas produced by the students, we used the F1 score. Specifically, we calculated precision and recall for two aspects: node and edge labels, and properties. This distinction was made because not all students included properties in the schemas they generated. In particular, we consider as *true positive* a correctly inserted label/property, as *false positive* an incorrect label or an extra node/edge/property added compared to the correct schema, and as *false negative* a missing node/edge/property.

# 5 Quantitative Analysis (RQ1)

In this section, we present the quantitative results for schema discovery, analytical, and aggregation queries.
## 5.1 Schema Discovery Queries

Figure 4(a) shows the results of four queries (Q1-Q4) on schema discovery. Among the first set of queries, most of them are *Correct* (or have *Complications*) and there are only eight semantic errors. Some students (18) used Neo4j specific procedures, and thus we labeled the answers as *Other*. Finally, only four answers contain syntax errors. Overall, the results highlight that Q1 and Q2 were more straightforward for students, with a higher number of correct responses. In contrast, Q3 and Q4 proved to be more challenging, reflected by the higher number of *Complications* and *Semantic Errors*. Table 3 summarizes the in-depth analysis of the *Complications* and the *Semantic Errors*. The majority of *Complications* consist of the additional use of `COLLECT` and `WITH`, where they account together for almost 80% of the *Complications* - while all the semantic errors occurred in the return clause of the queries, mainly due to a missing `DISTINCT` statement. From Figure 5, instead, we can see that the most problematic queries were Q3 and Q4. The first one was often over-complicated by the participants, whereas the second one was more susceptible to *Semantic Errors*.

<figure><img src="image_6.png" /><figcaption>Fig. 4. Amount of the different types of errors performed by the participants in the schema-discovery queries (a) in the analytical queries (b) and in the aggregation queries (c).</figcaption></figure>

<figure><img src="image_7.png" /><figcaption>Fig. 5. Amount of the different types of complications (a) and semantic return errors (b) performed by the participants in the schema-discovery queries.</figcaption></figure>

### 5.1.1 Additional Queries and Schema Derivation.
As described in Section 4.4, we labeled the responses to Q5 according to how many useful queries the participants wrote. The results were: 29 responses had zero useful queries; 19 responses had one, indicating some degree of usefulness; seven responses had two, showing a moderate level of usefulness; and
<table><caption>Table 3. Number of error categories and type occurred writing schema discovery queries. We report the absolute number, the percentage of occurrence both within the category and with respect to the whole set of errors.</caption><thead><tr><th>Category</th><th>Type</th><th>Count</th><th>Count /<br>Category Total</th><th>Count / Total</th></tr></thead><tbody><tr><td rowspan="4">Complications</td><td>Added COLLECT</td><td>49</td><td>0.422</td><td>0.395</td></tr><tr><td>Added WITH</td><td>40</td><td>0.345</td><td>0.323</td></tr><tr><td>Added UNWIND</td><td>27</td><td>0.233</td><td>0.218</td></tr><tr><td>Total</td><td>116</td><td>1</td><td>0.935</td></tr><tr><td rowspan="4">Semantic<br>Return<br>Errors</td><td>Missing DISTINCT</td><td>6</td><td>0.75</td><td>0.048</td></tr><tr><td>Missing Information</td><td>1</td><td>0.125</td><td>0.008</td></tr><tr><td>Wrong Information</td><td>1</td><td>0.125</td><td>0.008</td></tr><tr><td>Total</td><td>8</td><td>1</td><td>0.065</td></tr><tr><td>Total</td><td></td><td>124</td><td>1</td><td>1</td></tr></tbody></table>

four responses had three, indicating that they were highly useful. The high number of responses with zero useful queries may indicate a fundamental misunderstanding of how to utilize the schema to gather relevant information.

We also asked the participants to draw the graph schema using the knowledge gathered during the schema discovery queries. We computed the F1 score of the schema labels and the schema properties. Concerning the schema labels, the average F1 score is 0.929±0.164 while for the schema properties, the average F1 score is 0.453±0.485. Some students (26) did not write the properties in the schemas, lowering the average. Finally, to investigate if there was a relation between writing a correct query and the quality of the drawn schemas, we computed the Pearson correlation between the results of the schema discovery queries (number of correct answers and semantic errors as well as the results of Q5) and the F1 scores, but we did not find any statistically significant correlation (p > 0.05).

## 5.2 Analytical Queries

Figure 4(b) shows the results for the first part of the analytical queries (Q6-Q10), highlighting the types of errors and correct responses. This section of our study focused on more advanced queries, requiring students to analyze the provided graph schema.

Correct responses were notably few, with only 9 occurrences, and Complications were present 27 times, as shown in Table 4. These correct responses and complications were mainly concentrated in the first two queries, underscoring the increased complexity that students faced as they progressed through the more advanced queries. The over-use of COLLECT was still frequent, particularly in Q7 (13 cases) and Q6 (11 cases) as shown in Figure 6(a). Notably, there were no complications involving WITH and UNWIND in this phase. For completeness, an extended version of Table 4, including these clauses and the other from the taxonomy, is in the supplementary materials. Overall, the results from this part of the study show that while students may grasp the basics of Cypher queries, they face significant challenges when it comes to more advanced and nuanced aspects of graph querying.

One of the most significant findings was the prevalence of Semantic Where Errors, which were the most common (82/233). This indicates that students struggled significantly with formulating correct conditional statements within their queries. The most frequent mistake was due to a missing condition in the WHERE clause (49/82), followed closely by conditions on wrong properties (31/82). This suggests a fundamental difficulty in understanding how to accurately filter and retrieve data based on specific criteria within the graph. Additionally, Semantic Return Errors were notably high (88/233). These errors involved issues with specifying the correct data to be returned from a query. A frequent problem was the absence of the DISTINCT keyword, with 16 such instances in Q6 and Q14 in Q7, as shown in Figure 6(d). This highlights a challenge in comprehending how to effectively extract and present the desired results from the graph database. Semantic Pattern Errors were also prominent (63/233). These errors involved mistakes in identifying and
<figure><img src="image_8.png" /><figcaption>Fig. 6. Amount of the different types of complications (a), semantic pattern errors (b), semantic where errors (c) and semantic return errors (d) performed by the participants in writing analytical queries</figcaption></figure>

utilizing the correct patterns to match and retrieve data within the graph. For instance, in Q8, there were 17 instances of *wrong patterns* and 10 instances of *wrong edge direction* as shown in Figure 6(b). Even with the schema available, some students tried to use non-existent patterns, while others struggled with defining the pattern correctly. This indicates that as the complexity of the tasks increased, students found it increasingly difficult to construct and apply the appropriate graph patterns.

## 5.3 Aggregate Queries

Figure 4(c) presents the outcomes of aggregate queries (Q11-Q15), categorizing the results into correct responses and various types of errors. As specified in Section 4, 13 students participated in the final part of the study, as their involvement was voluntary. To ensure a fair analysis of this part of the study, we isolated the results of the 13 students in the previous queries to ensure we did not encounter *survivor bias* - i.e., verifying the students who did the final part were not exceptionally different from the rest. We verified that the distribution of correct responses and semantic errors was not statistically different from the distribution of the entire cohort. The tests for the survivor bias are in the additional material. This part focused on aggregate queries, which required students to construct intricate patterns and use aggregating clauses. Table 5 summarizes the results. The *Semantic Pattern Errors* were the most prevalent, with
<table><caption>Table 4. Number of error categories and type occurred writing analytical queries. We report the absolute number, the percentage of occurrence both within the category and with respect to the whole set of errors.</caption><thead><tr><th>Category</th><th>Type</th><th>Count</th><th>Count/ Category Total</th><th>Count / Total</th></tr></thead><tbody><tr><td rowspan="3">Complications</td><td>Added COLLECT</td><td>26</td><td>0.963</td><td>0.096</td></tr><tr><td>Additional Information</td><td>1</td><td>0.037</td><td>0.004</td></tr><tr><td>Total</td><td>27</td><td>1</td><td>0.1</td></tr><tr><td rowspan="6">Semantic Pattern Errors</td><td>Wrong Pattern w.r.t. Schema</td><td>18</td><td>0.29</td><td>0.067</td></tr><tr><td>Wrong Pattern</td><td>21</td><td>0.339</td><td>0.078</td></tr><tr><td>Missing Node</td><td>4</td><td>0.065</td><td>0.015</td></tr><tr><td>Additional Node</td><td>5</td><td>0.081</td><td>0.015</td></tr><tr><td>Wrong Edge Direction</td><td>14</td><td>0.226</td><td>0.052</td></tr><tr><td>Total</td><td>62</td><td>1</td><td>0.230</td></tr><tr><td rowspan="4">Semantic Where Errors</td><td>Wrong Property</td><td>31</td><td>0.34</td><td>0.115</td></tr><tr><td>Missing Condition</td><td>49</td><td>0.538</td><td>0.181</td></tr><tr><td>Wrong Condition</td><td>3</td><td>0.033</td><td>0.011</td></tr><tr><td>Wrong Operator</td><td>8</td><td>0.088</td><td>0.030</td></tr><tr><td rowspan="5">Semantic Return Errors</td><td>Total</td><td>91</td><td>1</td><td>0.337</td></tr><tr><td>Missing DISTINCT</td><td>34</td><td>0.378</td><td>0.126</td></tr><tr><td>Missing Information</td><td>13</td><td>0.144</td><td>0.048</td></tr><tr><td>Wrong Information</td><td>41</td><td>0.456</td><td>0.152</td></tr><tr><td>Wrong DISTINCT</td><td>2</td><td>0.022</td><td>0.007</td></tr><tr><td rowspan="2">Total</td><td>Total</td><td>90</td><td>1</td><td>0.333</td></tr><tr><td></td><td>270</td><td>1</td><td>1</td></tr></tbody></table>

students committing 21 such errors out of 53. Despite the complexity of the tasks, there were 19 Correct responses, suggesting that some students were able to successfully write advanced queries. Then, 15 out of 53 errors were Semantic Return Errors, indicating difficulties in specifying the correct data to be returned from the queries. The majority of them correspond to missing information in the results, suggesting a possible misunderstanding of the question. A considerable number of errors (12) is given by the incorrect use of LIMIT. Interestingly, the incidence of Semantic Where Errors decreased to three, compared to earlier parts of the study, while the number of Correct responses and Complications both increased. Although it is difficult to directly compare these results with those from previous sections due to the smaller sample size, this trend might suggest that the primary difficulties students face are related to the WHERE clause of the queries.

From Figure 7 we can see that Q12 and Q13 were the most difficult queries. In Q12, the participants had difficulties in writing the graph pattern, while in Q13, the participants made mistakes on what the query should return.

## 6 Qualitative analysis (RQ1)

In this section, we dive into a qualitative analysis of the errors and complications identified in students' Cypher queries. While the quantitative data provided an overview of the frequency and types of errors, a qualitative approach allows us to explore the underlying reasons and contextual factors contributing to these mistakes. By examining specific examples from the students' work, we aimed to uncover patterns and insights that may inform better instructional strategies and targeted interventions. We provide detailed explanations of these errors, supported by representative student query examples (errors marked in red). This approach helps highlight the specific challenges students face and the common misconceptions that lead to these errors.
<figure><img src="image_9.png" /><figcaption>Fig. 7. Amount of the different types of complications (a), semantic pattern errors (b), semantic where errors (c) and semantic return errors (d) performed by the participants in writing aggregate queries.</figcaption></figure>

## 6.1 The use of COLLECT

The misuse of the COLLECT keyword was the cause of many *Complication* (75/147). Interestingly, we noticed that students tend to over-engineer simple queries. For instance, Q3 and Q4 ask to list the node and edge labels together with the associated properties. An example of a recurrent *Complication* we detected is:

```sql
MATCH (n)
RETURN labels(n) AS NodeLabel, COLLECT(DISTINCT keys(n)) AS Properties
```

Then, we often found that the incorrect use of COLLECT in *Semantic Return* errors was often coupled with a missing DISTINCT. For example, in the following query (Q6) DISTINCT was required to return unique genes, but 16 out of 37 students used only COLLECT:

```sql
MATCH (p:Pangenome)<-[:IS_IN_PANGENOME]-(f:Family)<-[:IS_IN Family]-(g:Gene)
RETURN p, COLLECT(g) as genes
```
<table><caption>Table 5. Number of error categories and type occurred in writing aggregate queries. We report the absolute number, the percentage of occurrence both within the category and with respect to the whole set of errors.</caption><thead><tr><th>Category</th><th>Type</th><th>Count</th><th>Count/Category Total</th><th>Count / Total</th></tr></thead><tbody><tr><td rowspan="3">Complications</td><td>Added WITH</td><td>2</td><td>0.5</td><td>0.038</td></tr><tr><td>Additional Information</td><td>2</td><td>0.5</td><td>0.038</td></tr><tr><td>Total</td><td>4</td><td>1</td><td>0.075</td></tr><tr><td rowspan="6">Semantic Pattern Errors</td><td>Wrong Pattern w.r.t Schema</td><td>3</td><td>0.143</td><td>0.057</td></tr><tr><td>Wrong Pattern</td><td>1</td><td>0.048</td><td>0.019</td></tr><tr><td>Missing Node</td><td>1</td><td>0.048</td><td>0.019</td></tr><tr><td>Additional Node</td><td>3</td><td>0.143</td><td>0.057</td></tr><tr><td>Wrong Edge Direction</td><td>13</td><td>0.619</td><td>0.245</td></tr><tr><td>Total</td><td>21</td><td>1</td><td>0.396</td></tr><tr><td rowspan="3">Semantic Where Errors</td><td>Missing Condition</td><td>2</td><td>0.667</td><td>0.038</td></tr><tr><td>Wrong Operator</td><td>1</td><td>0.333</td><td>0.019</td></tr><tr><td>Total</td><td>3</td><td>1</td><td>0.057</td></tr><tr><td rowspan="4">Semantic Return Errors</td><td>Missing Information</td><td>14</td><td>0.560</td><td>0.264</td></tr><tr><td>Wrong Information</td><td>1</td><td>0.040</td><td>0.019</td></tr><tr><td>Missing LIMIT</td><td>10</td><td>0.400</td><td>0.189</td></tr><tr><td>Total</td><td>25</td><td>1</td><td>0.472</td></tr><tr><td>Total</td><td></td><td>53</td><td>1</td><td>1</td></tr></tbody></table>

## 6.2 The use of WITH

The second most popular *Complication* - in agreement with previous works [4, 14] - is due to the use of WITH, often paired with UNWIND (27/147). An example of a recurrent *Complication* we detected is:

```
MATCH (n)
WITH DISTINCT labels(n) AS NodeLabels
RETURN NodeLabels, count(NodeLabels) AS NumberOfDistinctLabels
```

The WITH clause allows chaining query parts and affects variables in scope. In this example, though, there is no need to use WITH to carry the scope of the variable *n* into the final part of the query. We found the same application of WITH in combination with UNWIND as in the following example:

```
MATCH (n)
WITH LABELS(n) AS labels
UNWIND labels AS label
RETURN DISTINCT label, COUNT(DISTINCT label) AS num labels
```

## 6.3 Missing and Wrong Conditions

The majority of *Semantic Where* errors were caused by a missing condition (51/94) in the WHERE clause. Analyzing the answers given by the students, we found two main types of missing condition errors: the first one is related to dealing with complex patterns, and the second one concerns filtering. When querying a graph, it could happen that patterns are unintentionally mapped to cycles.

Given the following pattern :

```
(p1:Pangenome)-[]-(f1:Family)-[]-(p2:Pangenome)
```
the trail semantics of Cypher [16] ensures that an edge is traversed only once during the pattern matching, and thus
nodes p1 and p2 are different. This is no longer true when dealing with longer or more complex patterns. An example
consists of the following student's answer to Q9:

```sql
MATCH (p1:Pangenome)-[:IS_IN_PANGENOME]-(f1:Family)
-[r1:IS SIMILAR]-(f2:Family)-[:IS_IN_PANGENOME]-(p2:Pangenome)
WHERE r1的身份 >= 0.8
AND f1 annotation IS NOT NULL
AND f2 annotation IS NOT NULL
RETURN f1.name, f2.name
```

In this example, nodes p1 and p2 could be the same, and so, it is important to specify `p1.taxid <> p2.taxid` (the nodes are distinct) in the `WHERE` clause to get the correct number of results.

As for the second case of *Missing Condition*, an example consists of this answer to Q10:

```sql
MATCH (p:Pangenome)-->(f:Family)-[ss:NEIGHBOR]-(g:Family)<--(p:Pangenome)
WHERE g annotation IS NOT NULL
RETURN DISTINCT p.name, f.name
```

In this case, the students did not add the condition `f annotation IS NOT NULL`. The question was "Write a Cypher query that returns, for each pangenome, the **neighborhood of its gene families containing annotations**" and thus we traced back the cause of this error to a language misunderstanding or a question incomprehension.

We also found mistakes due to *Wrong Condition* in the `WHERE` clause: caused by applying it on the wrong property of a node, or by applying the wrong logical operator in a condition. We found respectively 31 and 9 of these errors. An example consists of this answer to Q8:

```sql
MATCH (p:Pangenome)<--(f1:Family)-->(m:Module)
MATCH (m:Module)<--(f1:Family)-[sim]-(f2:Family)
WHERE sim identities > 0.8
WITH p, collect(DISTINCT m.name) as modules
RETURN p.name as pangenome name, modules
```

In this case, the right query would have `WHERE sim.coverage>=0.8`; thus, both the property and operator were wrong.

## 6.4 Pattern Definition

*Wrong Pattern* definition was the cause of 83 over 399 errors. In particular, 21 consisted of a pattern that did not exist in the dataset. An example is this answer to Q6, in which they used `:ASSOCIATED_WITH` which is not present in the graph:

```sql
MATCH (p:Pangenome)-[:ASSOCIATED_WITH]->(g:Gene)
RETURN p, collect(g) AS Genes
```

Even if the students were given the correct schema before answering the analytical queries, some of the participants (6 out of 54) still tried to use non-existent patterns with the `OPTIONAL MATCH` clause. For example:

```sql
MATCH (p:Pangenome) OPTIONAL MATCH (p)-[:IS_IN_PANGENOME]->(s:Spot)
RETURN p, COLLECT(s) AS associated spots
```

Moreover, 27 of the *Semantic Pattern Errors* consist of *Wrong Edge Direction*. For instance, in Q12 some students wrote:
```sql
MATCH (m:Module)-[:IS_IN_MODULE]-(f:Family)-[s:IS_similar]-(f2:Family)
RETURN m.name AS Module_Name, AVG(sidentity) AS avg_similarity
ORDER BY avg_similarity DESC
```

To correctly answer this question, the right pattern to use is:

```
(m:Module)<-[:IS_IN_MODULE]-(f:Family)<-[s:IS_similar]-(f2:Family)
```

This is a fundamental difference with respect to SQL, where there is not the concept of direction of relationships. When defining a graph pattern instead, expliciting edge directions reduce the number of matched subgraphs. Furthermore, 13 of the errors consist of a correct pattern but incomplete (Missing Node) or too specific (Additional Node). For example, this answer to Q9:

```sql
MATCH (f:Family)-[ss:IS_similar]->(g:Family)
WHERE ss.coverage >= 0.8
AND f annotation IS NOT NULL
AND g annotation IS NOT NULL
RETURN distinct f.name, g.name
```

is missing :Pangenome in the pattern:

```
(p1:Pangenome)--(f1:Family)-[r1:IS_similar]-(f2:Family)--(p2:Pangenome)
```

and thus producing more results, while the following answer to Q12 has a too specific pattern, producing fewer results:

```sql
MATCH (m:Module)-[:IS_IN_MODULE]-(:Family)-[r:IS_similar]-(:Family)-[:IS_IN_MODULE]-(m)
WITH m, AVG(r的身份) AS avg_similarity
RETURN m.name AS module_name, avg_similarity
ORDER BY avg_similarity DESC;
```

Finally, we detected 22 errors in which the students used a wrong pattern to write the query. To illustrate, consider the below student answer to Q10.

```sql
MATCH (p:Pangenome)<-[:IS_IN_PANGENOME]-(f:Family)
WHERE f annotation IS NOT NULL
WITH p, f MATCH (f)-[]-(g:Gene)-[]-(f2:Family)
WITH p, f, collect(f2) AS neighbors
RETURN p, f annoation AS annotation, neighbors
```

In this case, while the pattern is valid (i.e., exists in the database), it is not the correct one to answer the question.

## 6.5 Schema Misunderstanding

In our analysis of additional queries aimed at discovering the schema (Q5), we observed a diverse range of approaches and varying levels of misunderstanding. Out of 59 students, 18 focused on counting nodes, properties, and frequencies of edges and patterns within the graph. This approach suggests a gap in comprehension regarding the purpose of schema discovery, or an inability to distinguish between the different levels of queries required. Moreover, 23 students misunderstood the question and repeated their previous queries. The remaining students focused on understanding the edge directions and the connectivity between a pair of nodes. Only five of the students asked queries related to disconnected nodes. This indicates a lower recognition of the importance of understanding isolated elements within the graph, which can be crucial for identifying schema completeness and data integrity issues. On the other hand, analyzing
<figure><img src="image_10.png" /><figcaption>Fig. 8. Examples of errors in schema drawn by the students, duplicating the node to represent a loop (a), multiple labels on the edge (b), and double edges (c).</figcaption></figure>

the schemas drawn by the students, concerning the labels (and properties, when specified) the students generally performed well. Concerning the errors, we found three particular cases, illustrated in Figure 8. The first example in Figure 8(a) shows an isolated error where a student duplicated the node :Family instead of using self-relationships, while Figure 8(b) shows a case in which the student put two labels on the same edge instead of drawing two different edges (this happened with three participants). The last isolated case corresponds to Figure 8(c): in this case, the student did not investigate the edge directions in Q5 and drew two edges for every relationship, one for each orientation.

## 6.6 Syntax Errors

As expected, since the students could retry their queries as much as they wanted, we found only 12 *Syntax Errors*. The majority of them consist of typos, while two of them are noteworthy. Namely, *Invalid input Count(*)*, caused by using `*` inside the `COUNT` operator like in SQL, while in Cypher the universal match is done by using unlabeled nodes, and *PatternExpressions are not allowed to introduce new variables*, caused by trying to create new variables in a pattern, like in the following example (an answer to Q9):

```sql
MATCH (f1:Family), (f2:Family)
WHERE f1.pangenomeId <> f2.pangenomeId
AND f1 annotation <> "" AND f2 annotation <> ""
AND size((f1)-[:CONTAINS]->(g1:Gene)) * 0.80
< size((f2)-[:CONTAINS]->(g2:Gene))
RETURN f1 AS family1, f2 AS family2, COUNT(*) AS total_results
```

## 7 Investigation on the cause of the errors (RQ2)

After conducting a comprehensive analysis, both quantitatively and qualitatively, we investigated the causes of the errors committed by the students. To do so, first we carried out a correlation study between the errors made by the students and their self-reported prior knowledge collected in the first phase of the user study to see whether we could observe an effect as in the previous works related to SQL databases [24]. Second, we conducted additional semi-structured interviews on specific concepts that caused most problems in the previous phases. In particular, we identified four main
aspects to investigate: WITH, COLLECT, the graph pattern formulation, and the schema. After conducting a comprehensive
analysis, both quantitatively and qualitatively, we investigated the causes of the errors committed by the students. This
interview included specific questions aimed at uncovering the underlying reasons behind the errors. More precisely,
we posed questions about the role of Cypher operators, as well as examples of queries with and without errors to
check if students were able to recognize and understand them. We asked about the WITH and COLLECT statements,
which represent the main causes of complications during the entire study. We also focused on the role of the schema
and pattern composability. The precise questions are reported in the supplementary material. Participation was on
a voluntary basis and open to anyone who did at least one session of the study, and 14 students opted to participate.
While the questions were in English (and an English proficiency certification is required to be enrolled in the masters),
the participants could answer either in French or in English.

To ensure a fair analysis of this part of the study, we isolated the results of the students in the previous queries to
ensure we did not encounter survivor bias - i.e., verifying the students who did the final part were not exceptionally
different from the rest. We verified that the distribution of correct responses and semantic errors was not statistically
different from the distribution of the entire cohort. Finally, we analyzed the results of the interviews, identifying three
main reasons for errors: confusion with previous knowledge, challenges with understanding the role of the schema,
and misunderstanding of graph query elements.

## 7.1 Correlation Analysis

We used the Spearman correlation coefficient to compare the students’ errors with their self-reported knowledge of Relational Databases, NoSQL Databases, and Graph Databases. Comparing the categories of errors made in the schema discovery queries, we found a positive correlation between *Complications* and NoSQL knowledge (0.250) and a negative correlation with *Correct* responses (-0.274). No significant correlations were found with the other types of database knowledge. This indicates that students with more NoSQL knowledge write more over-complicated queries. Interestingly, a further correlation analysis showed that these complications include COLLECT and WITH statements. For the analytical queries, we observed a positive correlation between relational database knowledge and *Semantic Return Errors* (0.303) as well as *Semantic Where Errors* (0.295). Relational DB knowledge was also correlated with the increase of *Semantic Pattern Errors* in aggregation queries (0.290). This could indicate that relational knowledge conflicts with some aspects related to graph query (as discussed in more detail in Section 7.5).

Interestingly, the number of *Correct* was also negatively correlated to previous graph database knowledge (-0.268).
However, 20 out of the 59 students who made such errors claimed previous exposure to the Semantic Web Stack, in
particular to RDF and SPARQL. While similar in functionality to property graphs and Cypher, these do not provide the
same level of support for defining attributes on properties and handling n-ary relations.

## 7.2 Understanding the overuse of WITH

Concerning the use of WITH, in the interviews we propose two queries asking the students to select which one is correct and to explain the difference between them:

// both queries are correct because they return the same elements.

1. MATCH (n)
   RETURN DISTINCT labels(n), collect(keys(n)) AS properties
<table><caption>Table 6. Spearman Correlation between the declared previous knowledge about Relational Databases, NoSQL Databases and Graph Databases and the type of errors. Between brackets are the corresponding *p* values. Bold values indicate a significant correlation (*p* < 0.05).</caption><thead><tr><th colspan="6">Schema Discovery Queries</th></tr><tr><th></th><th>Correct</th><th>Complication</th><th>Semantic Pattern</th><th>Semantic Where</th><th>Semantic Return</th></tr></thead><tbody><tr><td>Relational DB</td><td>-0.105 (0.431)</td><td>0.019 (0.886)</td><td>-</td><td>-</td><td>-0.078 (0.558)</td></tr><tr><td>NoSQL DB</td><td><strong>-0.274 (0.036)</strong></td><td><strong>0.250 (0.048)</strong></td><td>-</td><td>-</td><td>0.015 (0.908)</td></tr><tr><td>Graph DB</td><td>-0.224 (0.088)</td><td>0.215 (0.102)</td><td>-</td><td>-</td><td>-0.120 (0.366)</td></tr></tbody><thead><tr><th colspan="6">Analytical Queries</th></tr><tr><th></th><th>Correct</th><th>Complication</th><th>Semantic Pattern</th><th>Semantic Where</th><th>Semantic Return</th></tr></thead><tbody><tr><td>Relational DB</td><td>-0.080 (0.547)</td><td>-0.122 (0.356)</td><td>-0.067 (0.617)</td><td><strong>0.295 (0.023)</strong></td><td><strong>0.303 (0.020)</strong></td></tr><tr><td>NoSQL DB</td><td>-0.192 (0.143)</td><td>-0.176 (0.183)</td><td>-0.013 (0.922)</td><td>0.239 (0.069)</td><td>0.151 (0.255)</td></tr><tr><td>Graph DB</td><td><strong>-0.268 (0.040)</strong></td><td>-0.136 (0.306)</td><td>0.127 (0.336)</td><td>0.213 (0.105)</td><td>0.145 (0.272)</td></tr></tbody><thead><tr><th colspan="6">Aggregation Queries</th></tr><tr><th></th><th>Correct</th><th>Complication</th><th>Semantic Pattern</th><th>Semantic Where</th><th>Semantic Return</th></tr></thead><tbody><tr><td>Relational DB</td><td>0.120 (0.365)</td><td>0.115 (0.385)</td><td><strong>0.290 (0.026)</strong></td><td>-0.135 (0.306)</td><td>0.083 (0.533)</td></tr><tr><td>NoSQL DB</td><td>-0.026 (0.843)</td><td>0.015 (0.913)</td><td>0.041 (0.758)</td><td>-0.109 (0.409)</td><td>-0.069 (0.604)</td></tr><tr><td>Graph DB</td><td>-0.100 (0.452)</td><td>-0.015 (0.913)</td><td>0.004 (0.977)</td><td>-0.052 (0.693)</td><td>-0.099 (0.454)</td></tr></tbody></table>

## 2. MATCH (n)

WITH DISTINCT labels(n) AS labels, n

RETURN labels, collect(keys(n)) AS properties

Even if both queries are correct, six students chose the second one saying that using WITH DISTINCT is the only way to reduce the redundancy in the result. Four students opted for the first, considering the use of WITH "useless", while only four students correctly selected both queries. This disagreement highlights confusion in their understanding of the construct. The answer we expected is that both queries are correct, but the use of WITH in the second one is not necessary. As explained in Section 2, the WITH operator chains subqueries. To better investigate how this was understood by the students, we asked them to explain its use in Cypher. The two main answers were: to chain subqueries (five students) and to reuse the variables (nine students), showing that this construct indeed creates confusion among the students. To understand the root of this misconception, we asked them where they find something similar to WITH in other programming languages or tools. Their answers include: AS (7) or WITH in SQL (3), "is reminiscent of programming languages in general" (2), the Pandas Python library (2), LINQ in C# (1) and promises in JavaScript (1). Since several students mentioned concepts related to other programming languages and specifically the AS statement in SQL (used to rename variables) this explains the confusing answers obtained in the interview and may suggest that previous knowledge indeed influences participants' understanding.

## 7.3 Understanding the overuse of COLLECT

As we described in Section 6, the misconceptions that involve the use of COLLECT are the most frequent among all the complications. To better investigate the student understanding of this operator, we asked which of the following queries is correct to return each distinct node label together with its corresponding set of properties:

// correct

### 1. MATCH (n)

RETURN DISTINCT labels(n) AS labels, keys(n) AS properties
// correct

2. MATCH (n)
   RETURN DISTINCT labels(n) AS labels, collect(distinct keys(n))
   AS properties

// wrong

3. MATCH (n)
   RETURN DISTINCT labels(n) AS labels, collect(keys(n))
   AS properties

Even though the first two queries are correct, six students chose the first, four students chose the second, three students chose the second and the third, and one student chose the third. When asked to motivate the answer, we noticed that the students who selected query number three thought that COLLECT turns a list into a set, removing the duplicates. To investigate this misconception, we also asked whether the following queries, both intended to retrieve each pangenome and its associated spots, were correct.

// wrong

1. MATCH (s:Spot)<-[:IS_IN_SPOT]-(:RGP)<-[:IS_IN_RGP]-(g:Gene)
   -[:IS_IN Family]->(:Family)-[:IS_IN_PANGENOME]->(p:Pangenome)
   RETURN p.name as pangenome, COLLECT(s.name) AS spots

// correct

2. MATCH (sp:Spot)<-[:IS_IN_SPOT]-(?)<-[:IS_IN_RGP]-(?)
   -[:IS_IN FAMILY]->(?)-[:IS_IN_PANGENOME]->(p:Pangenome)
   RETURN distinct p.name, sp.name

Nine out of 14 students correctly selected the second query as the correct one, while the others stated that COLLECT was needed to remove the duplicates.

To understand why this misconception occurred, we asked the students to explain the use of COLLECT in Cypher. In total, 10 out of 14 students reported that COLLECT aggregates the results in a single list, and thus it makes the return more readable. The remaining four students instead misunderstood the function of COLLECT as shown previously.

Finally, we asked the students which other programming languages or tools have a similar construct to COLLECT. The students mentioned several SQL aggregators with similar behaviors as COLLECT. This suggests the majority of the students understood what COLLECT does - it is indeed similar to the SQL constructs mentioned - but they used it when it was not necessary to improve the readability of the results.

We found one last case of cause for the complications with COLLECT due to text incomprehension. During the interview, seven students stated that they used COLLECT because the questions asked, "...return the list of ...".

## 7.4 Understanding the comprehension of the use of the schema

As we described in Section 6, the results revealed that students had difficulty understanding the role of the graph schema. Specifically, the students struggled to grasp how the schema should be derived. This was evident from the formulated
queries, which frequently revolved around simple data retrieval operations, such as counting and frequencies, rather than investigating edge directions and node connectivity.

However, the results from the interviews showed a notable improvement. We asked students what the role of the schema is and why it is important. Students replied that "It is a structure defining the organization of data and the relationships between different entities in a database", "It's used to ensure data consistency and integrity, and to facilitate their management and retrieval", and similar explanations. Indeed, all of the 14 students demonstrated a clear grasp of its role, by stating that it shows the different types of nodes and relations, along with their attributes, allowing users to better formulate queries. These answers suggest that, over the course of the experiment, students were able to bridge the gap in their understanding.

## 7.5 Understanding the misconceptions when working with complex patterns

One of the critical challenges identified in this study was the significant number of misconceptions students had when working with complex patterns. As reported in Section 6.4, we recorded 84 *Semantic Pattern Errors*, indicating widespread difficulty in accurately defining and utilizing graph patterns. Moreover, a substantial portion of these errors was due to *Missing Conditions*, which were often tied to the formulation of the pattern itself.

To further investigate this issue, in the interview we asked the students if they believed it was necessary to explicitly define the inequality between nodes in a pattern. We asked whether, given the following patterns, the two nodes with the same label f1 and f2 could be the same.

1. MATCH (p:Pangenome)<--()-->(m:Module)<--(f1:Family)<-[r:IS SIMILAR]-(f2:Family)

2. MATCH (p:Pangenome)<-[:IS_IN_PANGENOME]-(f1:Family)-[:IS_IN_MODULE]->(m:Module), (p:Pangenome)<-[:IS_IN_PANGENOME]-(f1:Family)-[sim:IS SIMILAR]->(f2:Family)

Two students indicated that the nodes are different, while three others suggested adding the condition f1>f2. Five students responded that the answer depends on the domain, noting that "a family cannot be similar to itself" but that, in general, it is possible.

Only one student said that looking at the schema, there is only one edge between two families and thus, the nodes are different. To answer this question correctly, students need to understand the trail semantics of Cypher described in Section 2, which stipulates that an edge can be traversed only once in a single pattern. This rule is crucial for constructing valid and efficient queries. However, the survey responses revealed that many students were unaware of this principle, leading to errors in their pattern definitions.

## 7.6 Discussion (RQ2)

The user study identified several common errors made by students when formulating Cypher queries. To address RQ2, the findings can be categorized into three key areas: errors related to general query languages, errors related to the graph data model, and errors specifically related to Cypher.

*Errors related to learning a query language.* As reported in previous works [3, 23, 33], a source of errors shared between every query language is related to the WHERE clause, where students struggle with logical expressions. The Cypher syntax for filtering is indeed similar to the one of SQL, and the *Semantic Where* error is the most prominent one - as shown in Table 4. Moreover, consistent with previous works in [23, 34], we found that the student's programming experience influences the understanding of query languages. In our case, it manifests mainly as a tendency to overuse
WITH in order to chain query operations, a pattern more common in procedural SQL and imperative programming paradigms. Indeed, in SQL, WITH is used for defining Common Table Expressions, which allow complex queries to be broken down into reusable subqueries, chaining operations across multiple query steps. In Python, the WITH statement is commonly used for resource management (e.g., file handling), ensuring that operations, e.g., opening/closing files, are executed within a defined scope.

*Errors related to the graph data model.* A distinct class of errors was directly tied to the novelty of the graph data model for most students. Unlike relational databases, where schemas are explicit and often dictate query structure and guide its formation, graph databases emphasize flexible, navigational access patterns. This conceptual shift proved challenging. Many students initially struggled to recognize how the schema could influence their queries, which resulted in incorrect pattern formulations and over-complicated queries. This misunderstanding was rooted in their unfamiliarity with graph query languages and their lack of experience working with graph structures. Another area of difficulty revolved around graph pattern matching, particularly when students were required to define complex patterns. This issue was again attributed to a lack of understanding of the graph-based logic. This is also reflected in the correlation with Relational Database Knowledge shown in Table 6, highlighting that many students faced difficulties in transitioning from traditional database paradigms to a graph-focused approach, leading to errors rooted in both prior knowledge interference and misunderstandings of the new language.

*Errors related to Cypher language features.* Several errors were linked to students’ difficulties in understanding the specific features and syntax of the Cypher language, confirming previous works [4, 14]. However, unlike Chen et al., [14] we were able to identify the specific constructs, such as the WITH clause and aggregation functions (e.g., COLLECT) that are frequent sources of complications. The usage of COLLECT to remove duplicates from lists was a direct result of students’ incorrect grasp of the function’s purpose. Additionally, students overused COLLECT attempting to enhance readability, even if unnecessary. This particular misuse did not have a clear underlying cause but suggests a potential misunderstanding of when COLLECT is required and a possible desire to align Cypher queries with perceived norms of clarity from other programming languages.

## 7.7 Best Practices for Teaching (Future) Graph Query Languages

The distribution of errors and the scarcity of correct answers showcased in our study further emphasize the need for targeted instructional support to help students comprehend the complexities of graph queries and overcome the aspects that proved to be challenging.

To address these issues related to Cypher we recommend the following best practices.

*Emphasize schema understanding early on.* A key takeaway is the importance of understanding the schema from the results of the schema discovery queries (see Section 5.1) and the students’ difficulties in using complex patterns (see Section 5.2 and Section 7.4). To address this, instructors should highlight the role of the schema from the beginning of the course, teach techniques for deriving and understanding schemas, and provide examples and exercises that require students to derive schemas before writing queries. Understanding the schema can also help overcome language comprehension issues, as it provides a clearer context for query construction and expected results.

*Clarify the use of COLLECT and WITH clauses.* From Section 5.1, we observed that many complications arise from the additional use of COLLECT and WITH. To address this, one should provide detailed explanations and examples of
how these constructs are used, illustrate their output (e.g., lists of nodes, relationships), and their efficient processing. Practical exercises would help reinforce these concepts and ensure students understand how to aggregate and transform data within queries.

*Enhance instruction on complex pattern matching.* Complex patterns pose a significant challenge, as seen in Sections 5.2 and 5.3. The scarcity of correct answers and the distribution of errors highlight the need for more targeted instructional support. Our findings suggest a fundamental gap in students’ understanding of constructing and manipulating complex patterns within graph databases. To bridge this gap, it would be useful to extensively illustrate the semantics of Cypher’s pattern matching, highlighting the differences between SQL and relational algebra, as well as to provide exercises of increasing complexity, using real-world scenarios to explain how complex queries are formulated and executed. Incorporating insights from this error analysis could help to develop in-depth examples targeting the main aspects students struggled with. Given that the recent GQL standard for native graph querying is rooted in Cypher, we argue that ensuring a firm understanding of Cypher’s fundamental pattern-matching principles is a crucial first step. In light of the language’s convergence to supporting full GQL expressiveness, future pedagogical efforts should also highlight the practical needs that have shaped graph query language evolution through dedicated case studies. By implementing these best practices, educators can provide more effective instructional support, helping students to better understand and master advanced graph query languages. This approach not only addresses current challenges but also prepares students for the complexities they will encounter in real-world applications of graph databases.

## 8 Limitations

We identified the following limitations in our study. The participants of our cohort included students of two international master programs in France universities that followed a specific course in graph processing. While this may not be a representative sample of the people learning Cypher, our results confirm and corroborate some of the findings in a few available studies on graph query languages [4, 14].

Also, not all the students participated to the interviews, and, while we did not find any significant differences in the responses of the previous part of the study between the interviewees and the rest of the participants, we acknowledge that we could have missed some other insights.

We also recognize the genomic domain is complex and that it could have further impacted the student's ability to write the queries. However, in our qualitative analysis, this aspect didn't emerge, and in the interviews, none of the students highlighted this as a problem. Another limitation is related to the fact that none of the students were native English speakers, and this could lead to language incomprehension. We observed some errors of this type that lead to using wrong operators (e.g., > instead of >=), returning an incomplete result, or overcomplicating the query. However, these represent a small fragment of the total cases and, in the interview, this issue was not mentioned. We noticed that the other reasons reported in Section 7 were more predominant.

## 9 Conclusion

In this paper, we presented the first to the best of our knowledge - comprehensive study on student errors in graph query formulation. To investigate which aspects of the Cypher language the students struggle with the most and what are the main causes of the errors, we performed a user study asking them to write schema discovery, analytical, and aggregation Cypher queries.
We completed the user study with additional face-to-face interviews to understand the cause of students' misconceptions. The extensive analysis of the results shows interesting trends in the error distributions depending on the complexity of the graph queries and the type of errors.

To address the limitations presented in Section 8, we plan to develop a set of tools that can be easily used to perform this user study in different institutions and with participants having more diverse backgrounds. Moreover, future studies should utilize other domains, closer to the experience of the participants. Another direction for future investigation is to consider more expressive Cypher features that are being integrated to increase coverage of the GQL standard, e.g., path conditions, path variables, subqueries, disjunctions, and optional. Indeed, open-source and commercial databases now using Cypher are gradually adopting features from the standard GQL query language, driving the need to further investigate the novel components of the language. In addition, schema and constraint languages, such as PG-Schema [7] and PG-Keys [8] can be incorporated in the study, by extending the schema discovery part.

## Acknowledgments

The work was partially funded by the Data4Health ANR/CPJ Lyon 1, ANR-21-CE48-0015 VeriGraph, and ANR-24-CE25-1109 VERDI grants. We thank all participants and anonymous reviewers of this work.

## References

[1] [n. d.]. *openCypher*. https://opencypher.org/

[2] Alireza Ahadi, Vahid Behbood, Arto Vihavainen, Julia Prior, and Raymond Lister. 2016. Students' Syntactic Mistakes in Writing Seven Different Types of SQL Queries and Its Application to Predicting Students' Success. In *Proceedings of the 47th ACM Technical Symposium on Computing Science Education* (Memphis, Tennessee, USA) (SIGCSE '16). Association for Computing Machinery, New York, NY, USA, 401–406. https://doi.org/10.1145/2839509.2844640

[3] Alireza Ahadi, Julia Prior, Vahid Behbood, and Raymond Lister. 2016. Students' Semantic Mistakes in Writing Seven Different Types of SQL Queries. In *Proceedings of the 2016 ACM Conference on Innovation and Technology in Computer Science Education* (Arequipa, Peru) (ITiCSE '16). Association for Computing Machinery, New York, NY, USA, 272–277. https://doi.org/10.1145/2899415.2899464

[4] Ridha Alkhabaz, Zepei Li, Sophia Yang, and Abdussalam Alawini. 2023. Student's Learning Challenges with Relational, Document, and Graph Query Languages. In *Proceedings of the 2nd International Workshop on Data Systems Education: Bridging Education Practice with Education Research* (Seattle, WA, USA) (DataEd '23). Association for Computing Machinery, New York, NY, USA, 30–36. https://doi.org/10.1145/3596673.3596976

[5] Ridha Alkhabaz, Seth Poulsen, Mei Chen, and Abdussalam Alawini. 2021. Insights from Student Solutions to MongoDB Homework Problems. In *Proceedings of the 26th ACM Conference on Innovation and Technology in Computer Science Education V. 1* (Virtual Event, Germany) (ITiCSE '21). Association for Computing Machinery, New York, NY, USA, 276–282. https://doi.org/10.1145/3430665.3456308

[6] Renzo Angles. 2018. The Property Graph Database Model. In *AMW (CEUR Workshop Proceedings, Vol. 2100)*. CEUR-WS.org.

[7] Renzo Angles, Angela Bonifati, Stefania Dumbrava, George Fletcher, Alastair Green, Jan Hidders, Bei Li, Leonid Libkin, Victor Marsault, Wim Martens, Filip Murlak, Stefan Plantikow, Ognjen Savkovic, Michael Schmidt, Juan Sequeda, Slawek Staworko, Dominik Tomaszuk, Hannes Voigt, Domagoj Vrgoc, Mingxi Wu, and Dusan Zivkovic. 2022. PG-Schema: Schemas for Property Graphs. CoRR abs/2211.10962 (2022). https://doi.org/10.48550/ARXIV.2211.10962 arXiv:2211.10962

[8] Renzo Angles, Angela Bonifati, Stefania Dumbrava, George Fletcher, Keith W. Hare, Jan Hidders, Victor E. Lee, Bei Li, Leonid Libkin, Wim Martens, Filip Murlak, Josh Perryman, Ognjen Savkovic, Michael Schmidt, Juan F. Sequeda, Slawek Staworko, and Dominik Tomaszuk. 2021. PG-Keys: Keys for Property Graphs. In *SIGMOD '21: International Conference on Management of Data, Virtual Event, China, June 20-25, 2021*, Guoliang Li, Zhanhuai Li, Stratos Idreos, and Divesh Srivastava (Eds.). ACM, 2423–2436. https://doi.org/10.1145/3448016.3457561

[9] Jérôme Arnoux, Angela Bonifati, Alexandra Calteau, Stefania Dumbrava, and Guillaume Gautreau. 2024. Integrating Complex Pangenome Graphs. In *ICDEW*. IEEE, 350–354.

[10] AWS. 2023. Amazon Neptune. https://aws.amazon.com/neptune/.

[11] Angela Bonifati, George Fletcher, Hannes Voigt, and Nikolay Yakovets. 2018. *Data Models*. Springer International Publishing, Cham, 3–14. https://doi.org/10.1007/978-3-031-01864-0_2

[12] Stefan Brass and Christian Goldberg. 2006. Semantic errors in SQL queries: A quite complete list. *Journal of Systems and Software* 79, 5 (2006), 630–644. https://doi.org/10.1016/j.jss.2005.06.028 Quality Software.

[13] R. B. Buitendijk. 1988. Logical Errors in Database SQL Retrieval Queries. *Comput. Sci. Econ. Manage.* 1, 2 (jun 1988), 79–96. https://doi.org/10.1007/BF00427157
[14] Mei Chen, Seth Poulsen, Ridha Alkhabaz, and Abdussalam Alawini. 2021. A Quantitative Analysis of Student Solutions to Graph Database Problems. In *Proceedings of the 26th ACM Conference on Innovation and Technology in Computer Science Education V. 1* (Virtual Event, Germany) (ITiCSE '21). Association for Computing Machinery, New York, NY, USA, 283–289. https://doi.org/10.1145/3430665.3456314

[15] Alan D. Fekete and Uwe Röhm. 2022. Teaching about Data and Databases: Why, What, How? *SIGMOD Rec.* 51, 2 (2022), 52–60.

[16] Nadime Francis, Alastair Green, Paolo Guagliardo, Leonid Libkin, Tobias Lindaaker, Victor Marsault, Stefan Plantikow, Mats Rydberg, Petra Selmer, and Andrés Taylor. 2018. Cypher: An Evolving Query Language for Property Graphs. In *SIGMOD Conference. ACM*, 1433–1445.

[17] Bitnine Global Inc. 2023. AgensGraph. https://bitnine.net/agensgraph/.

[18] ISO/IEC. 2024. ISO/IEC 39075:2024 Information technology — Database languages — GQL. https://www.iso.org/standard/76120.html.

[19] Redis Labs. 2017. RedisGrap. https://oss.redislabs.com/redisgraph/.

[20] Zepei Li, Sophia Yang, Kathryn Cunningham, and Abdussalam Alawini. 2023. Assessing Student Learning Across Various Database Query Languages. In *2023 IEEE Frontiers in Education Conference (FIE)*. 1–9. https://doi.org/10.1109/FIE58773.2023.10343409

[21] Jim Melton and Alan Simon. 2001. *SQL:1999: Understanding Relational Language Components*. Morgan Kaufmann Publishers Inc., San Francisco, CA, USA.

[22] Memgraph. 2023. Memgraph. https://memgraph.com/.

[23] Daphne Miedema, Efthimia Aivaloglou, and George Fletcher. 2021. Identifying SQL Misconceptions of Novices: Findings from a Think-Aloud Study. In *Proceedings of the 17th ACM Conference on International Computing Education Research* (Virtual Event, USA) (ICER 2021). Association for Computing Machinery, New York, NY, USA, 355–367. https://doi.org/10.1145/3446871.3469759

[24] Daphne Miedema, George Fletcher, and Efthimia Aivaloglou. 2022. Expert Perspectives on Student Errors in SQL. *ACM Trans. Comput. Educ.* 23, 1, Article 11 (dec 2022), 28 pages. https://doi.org/10.1145/3551392

[25] Daphne Miedema, George Fletcher, and Efthimia Aivaloglou. 2022. So Many Brackets! An Analysis of How SQL Learners (Mis)Manage Complexity during Query Formulation. In *Proceedings of the 30th IEEE/ACM International Conference on Program Comprehension* (Virtual Event) (ICPC '22). Association for Computing Machinery, New York, NY, USA, 122–132. https://doi.org/10.1145/3524610.3529158

[26] Andrew Migler and Alex Dekhtyar. 2020. Mapping the SQL Learning Process in Introductory Database Courses. In *Proceedings of the 51st ACM Technical Symposium on Computer Science Education* (Portland, OR, USA) (SIGCSE '20). Association for Computing Machinery, New York, NY, USA, 619–625. https://doi.org/10.1145/3328778.3366869

[27] Neo4j. 2023. Neo4j. https://neo4j.com/.

[28] Amedeo Pachera, Stefania Dumbrava, Angela Bonifati, and Andrea Mauri. 2024. *Supplementary material*. https://github.com/pake97/Understanding-Student-Errors-in-Graph-Query-Formulation

[29] Marcus Paradies. 2017. Graph pattern matching in SAP HANA. https://tinyurl.com/ycxu54pr.

[30] Ye Peng, Shanmei Tang, Dan Wang, Huanzi Zhong, Huijue Jia, Xianghang Cai, Zhaoxi Zhang, Minfeng Xiao, Huanming Yang, Jian Wang, Karsten Kristiansen, Xun Xu, and Junhua Li. 2018. MetaPGN: a pipeline for construction and graphical visualization of annotated pangenome networks. *Gigascience* 7, 11 (Nov. 2018).

[31] Sherif Sakr, Angela Bonifati, Hannes Voigt, Alexandru Iosup, Khaled Ammar, Renzo Angles, Walid Aref, Marcelo Arenas, Maciej Besta, Peter A Boncz, et al. 2021. The future is big graphs: a community view on graph processing systems. *Commun. ACM* 64, 9 (2021), 62–71.

[32] solidIT. 2023. DB-Engines Ranking. https://db-engines.com/en/ranking_trend/graph+dbms.

[33] Toni Taipalus, Mikko Siponen, and Tero Vartiainen. 2018. Errors and Complications in SQL Query Formulation. *ACM Trans. Comput. Educ.* 18, 3, Article 15 (aug 2018), 29 pages. https://doi.org/10.1145/3231712

[34] Ethel Tshukudu and Quintin I. Cutts. 2020. Semantic Transfer in Programming Languages: Exploratory Study of Relative Novices. In *ITiCSE*. ACM, 307–313.

# A Informed Consent

## Researchers

Removed for anonymity purposes.

## Purpose of the study:

You are invited to participate in a research study conducted by the University. This study is aimed at understanding how students comprehend various aspects of the Cypher language, identifying common misconceptions. This will require approximately 90 minutes. The findings will be used to improve Cypher teaching methods and potentially enhance the language for better accessibility.
**Study Procedure**

You will be tasked with writing solutions to different queries using the Neo4j desktop edition to test and submit answers.
Correct answers will be submitted via a Google form, while other attempts should be documented in a separate file. At
the beginning and end of the study, surveys may be required. Please avoid personal identifiers in surveys.

**Benefits and Risks of Participation**

Participation poses no risks beyond daily life or typical online activities. While there may be no direct benefits, the data
could aid in designing better courses and more comprehensible query languages.

**Withdrawal Procedures**

Participation is voluntary, and you may withdraw at any time.

**Data Collection and Processing**

We will only collect data that you submit. The analysis results and collection methodology may be published in research
papers or technical reports, aiming for open access. Data access rights will be supervised by the researchers.

**Participant Rights**

If you have concerns about the study, please contact the researchers listed at the beginning of this informed consent
form.

**Consent Options**

* Accept

* Don't Accept

B Demographic Information and Knowledge Self-Assessment

**Demographic Information**

* Age Range: 18-24, 25-34, 35-44, 45-54, 55+, Prefer not to say

**Knowledge Assessment**

* **Programming Languages:** List the programming languages you know, separated by commas.

* **Relational Database Knowledge:** Rate on a scale of 1 (never heard of it) to 5 (professional level)

* **NoSQL Database Knowledge:** Rate on a scale of 1 (never heard of it) to 5 (professional level)

* **Graph Structure Knowledge:** Rate on a scale of 1 (never heard of it) to 5 (professional level)

**Graph Query Language Proficiency**

* OpenCypher, GraphQL, AQL, SQL/PGQ, SPARQL, None
## C Queries Ground Truth

In this Section, we report the ground truth answer to the questions of our study. We manually write the correct query to answer each question. Note that there are multiple solutions to obtain the correct results. We did not directly compare our ground truth with the students' answers to label them, instead, we used the following queries and guidelines.

### C.1 Schema Discovery Queries

* Q1: Write a Cypher query that returns the set of all distinct node labels. State also how many answers it returns.
```
MATCH (n)
RETURN DISTINCT labels(n)
//ANSWERS: 9
```

* Q2: Write a Cypher query that returns the set of all distinct relationship types. State also how many answers it returns.
```
MATCH ()-[r]->()
RETURN DISTINCT type(r)
//ANSWERS: 10
```

* Q3: Write a Cypher query that returns each distinct node label together with its corresponding set of properties. State also how many answers it returns.
```
MATCH (n)
RETURN DISTINCT labels(n) AS label, properties(n) AS properties
//ANSWERS: 18
```

* Q4: Write a Cypher query that returns each distinct relationship type together with its corresponding set of properties. State also how many answers it returns.
```
MATCH ()-[r]->()
RETURN DISTINCT type(r) AS type, properties(n) AS properties
//ANSWERS: 10
```

* Q5: Write at least three additional Cypher queries that are needed for you to understand the underlying schema structure of the dataset. You can also write more than three queries in case. For each of them, state also how many answers it returns.
```
MATCH (m)-[l]->(n)-[k]->(:Genome)
RETURN DISTINCT type(k), type(l), labels(n), labels(m)
//ANSWERS: 1

MATCH (m)-[l]->(n)-[k]->(:Family)
RETURN DISTINCT type(k), type(l), labels(n), labels(m)
//ANSWERS: 6

MATCH (m)-->(n)-->(:Family)
RETURN DISTINCT labels(m), labels(n)
//ANSWERS: 2
```

### C.2 Analytical Queries

* Q6: Write a Cypher query to return each pangenome, together with its associated genes. State also how many answers it returns.
```
MATCH (g:Gene)-[:IS_IN Family]->(f:Family)-[:IS_IN_PANGENOME]->(p:Pangenome)
RETURN distinct p.name, g.name
//ANSWERS: 5477
```

* Q7: Write a Cypher query to return each pangenome, together with its associated spots. State also how many answers it returns.
```
MATCH (sp:Spot)<-[:IS_IN_SPOT]->()<-[:IS_IN_RGP]->()-[[:IS_IN_family]->()
-[[:IS_IN_PANGENOME]]->(p:Pangenome)
```
RETURN distinct p.name, sp.name
//ANSWERS: 656

* Q8: Write a Cypher query to return each pangenome and its associated modules that contain at least a pair of gene families that are more than 80% identical. State also how many answers it returns.
 ```sql
MATCH (p:Pangenome)<--()-->(m:Module)<--(f1:Family)<-[r:ISSIMILAR]-(f2:Family)
WHERE f1 <> f2 AND r.coverage >= 0.8
RETURN p.name, COUNT(DISTINCT(m.name))
//ANSWERS: 2 - Enterobacter.cloacae (236 modules)
//and Acinetobacter.baumannii (278 modules)
```

* Q9: Write a Cypher query that returns pairs of inter-pangenome families that both contain annotations and that are more than 80% identical. State also how many answers it returns.
  ```sql
MATCH (p1:Pangenome)-[:IS_IN_PANGENOME]-(f1:Family)-[r1:IS SIMILAR]-(f2:Family)
-[:IS_IN_PANGENOME]-(p2:Pangenome)
WHERE r1 identity >= 0.8
AND p1.taxid <> p2.taxid
AND f1 annotation IS NOT NULL
AND f2 annotation IS NOT NULL
RETURN f1.name, f2.name
//ANSWERS: 14
```

* Q10: Write a Cypher query that returns, for each pangenome, the neighborhood of its gene families containing annotations. State also how many answers it returns.
  ```sql
MATCH (f1:Family)-[:NEIGHBOR]-(f2:Family) [[:IS_IN_PANGENOME]]->(p:Pangenome)
WHERE f1 annotation IS NOT NULL
AND f2 annotation IS NOT NULL
RETURN p, f1, f2
//ANSWERS: 153
```

## C.3 Aggregation Queries

* Q11: For every pangenome, write a Cypher query that return its name, together with its number of families that contain annotations. State also how many answers it returns.
  ```sql
MATCH (p:Pangenome)-[:IS_IN_PANGENOME]-(f:Family)
WHERE f annotation IS NOT NULL
RETURN p.name, count(f)
//ANSWERS: ("Enterobacter.cloacae": 110, "Acinetobacter.baumannii": 92)
```

* Q12: Write a Cypher query that returns all modules together with the average similarity of the gene families they are comprised of, in descending order. State also how many answers it returns.
  ```sql
MATCH (m:Module) <-[j]-(a:Family)<-[k:IS SIMILAR]-(f:Family)
RETURN m.name, COLLECT(f.name), AVG(r的身份) AS idnt
ORDER BY idnt DESC
//ANSWERS: 486
```

* Q13: Write a Cypher query that returns the names of the modules that contain the pairs of gene families that have the minimum and, respectively, the maximum similarity (as indicated by the value of the "identity" property of the "IS SIMILAR" edge connecting them). State also how many answers it returns.
  ```sql
MATCH (m1:Module)<-[:IS_IN_MODULE]-(f:Family)-[sim:IS SIMILAR]->(f2:Family)
-[:IS_IN_MODULE]->(m2:Module)
RETURN m1.name, m2.name, sim的身份
ORDER BY sim的身份 DESC
LIMIT 1
UNION ALL
MATCH (m1:Module)<-[:IS_IN_MODULE]-(f:Family)-[sim:IS SIMILAR]->(f2:Family)
-[:IS_IN_MODULE]->(m2:Module)
RETURN m1.name, m2.name, sim的身份
ORDER BY sim的身份 LIMIT 1
//ANSWERS: Modules 2 and 177 with similarities 40 and, respectively, 0.8
```
* Q14: Write a Cypher query that returns the top 10 gene families with the highest number of associated modules. State also how many answers it returns.

```sql
MATCH (f:Family)-[:IS_IN_MODULE]- (m:Module)
WITH f, count(m) AS num
RETURN f, num
ORDER BY num DESC
LIMIT 10
//ANSWERS: 10
```

* Q15: Return the names of the top 10 pairs of similar families (as defined in Q13), whose associated identity and coverage metrics surpass 0.8, together with the names of their partitions (as indicated by the value of the "subpartition" property associated to the nodes of type "Partition"). State also how many answers it returns.

```sql
MATCH (p:Partition)-<[:HAS_PARTITION]- (f1:Family)-[r:IS SIMILAR]-> (f2:Family)
-[:HAS_PARTITION]-> (p2:Partition)
WHERE r的身份 >=0.8 AND r. coverage >=0.8
RETURN f1.name, f2.name, p(partition, p2 partition, r身份)
ORDER BY r身份 DESC
LIMIT 10
//ANSWERS: 10
```

## D Semi-Structured Interview

* Q1: Explain your understanding of the SCHEMA (in general). What is it and what is it used for?

* Q2: What is the use of the `COLLECT` keyword in Cypher?

* Q3: In which other programming language/tool is there something similar to `COLLECT`? Does it remind you of something you've used before?

* Q4: What is the use of the `WITH` keyword in Cypher?

* Q5: In which other programming language/tool is there something similar to `WITH`? Does it remind you of something you've used before?

* Q6: Given this question: Write a Cypher query that returns each distinct node label together with its corresponding set of properties. Which of the following queries is correct?

- MATCH (n) RETURN DISTINCT labels(n) AS labels, keys(n) AS properties

- MATCH (n) RETURN DISTINCT labels(n) AS labels, collect(distinct keys(n)) AS properties

- MATCH (n) RETURN DISTINCT labels(n) AS labels, collect(keys(n)) AS properties

* Q7: Explain the differences between the queries provided in Q6.

* Q8: Given the same question in Q6, explain the difference between the following queries:

- MATCH (n) RETURN DISTINCT labels(n), collect(keys(n)) AS properties

- MATCH (n) WITH DISTINCT labels(n) AS labels, n RETURN labels, collect(keys(n)) AS properties

* Q9: Explain the differences between the queries provided in Q8.

* Q10: Given this question: Write a Cypher query to return each pangenome, together with its associated genes. Which of the following queries is correct?

- MATCH (p:Pangenome)- (f:Family)- (g:Gene) RETURN DISTINCT p.name, g.name;

- MATCH (g:Gene)-[:IS_IN FAMILY]-> ([:Family]-[:IS_IN_PANGENOME]-> (p:Pangenome) RETURN p.name AS pangenomeName, COLLECT(g.name) AS genes)

- MATCH (gene:Gene)-[:IS_IN FAMILY]-> (family:Family)-[:IS_IN_PANGENOME]-> (pangenome:Pangenome) RETURN pangenome, gene

* Q11: Explain the differences between the queries provided in Q10.
* **Q12:** Given a similar query structure as in **Q10**, are both of the following queries correct?

- MATCH (s:Spot)<-[:IS_IN_SPOT]-(:RGP)<-[:IS_IN_RGP]-(g:Gene)-[:IS_IN Family]->(:Family)
- [:IS_IN_PANGENOME]->(p:Pangenome) RETURN p.name AS pangenome, COLLECT(s.name) AS spots

- MATCH (sp:Spot)<-[:IS_IN_SPOT]-()-<[:IS_IN_RGP]-()-[:IS_IN Family]->()-[:IS_IN_PANGENOME]
-> (p:Pangenome) RETURN DISTINCT p.name, sp.name

* **Q13:** Explain the answer to the previous question.

* **Q14:** If a query has the return statement `RETURN a, COLLECT(b)`, would you assume that the names of b are unique? Motivate your answer.

* **Q15:** Given this question: Write a Cypher query to return each pangenome and its associated modules with gene families more than 80% identical. Which of the following queries is correct?

- MATCH (p:Pangenome)<-()-(m:Module)<- (f1:Family)<- [r:IS_similar]-(f2:Family) WHERE f1 <> f2 AND r.coverage >= 0.8 RETURN p.name

- MATCH (p:Pangenome)<-[:IS_IN_PANGENOME]-(f:Family)-[:IS_IN_MODULE]->(m:Module), (p:Pangenome)<-[:IS_IN_PANGENOME]-(f:Family)-[sim:IS_similar]->(f2:Family) WHERE sim.coverage >= 0.8 RETURN p.name AS Pan-genome

* **Q16:** Explain the differences between the queries provided in **Q15**.

* **Q17:** Given a query pattern with two nodes of the same label, for example:

- MATCH (p:Pangenome)<-()-(m:Module)<- (f1:Family)<- [r:IS_similar]-(f2:Family)

- MATCH (p:Pangenome)<-[:IS_IN_PANGENOME]-(f1:Family)-[:IS_IN_MODULE]->(m:Module), (p:Pangenome)<-[:IS_IN_PANGENOME]-(f1:Family)-[sim:IS_similar]->(f2:Family) WHERE sim identity >= 0.8 RETURN p.name AS Pangenome

Could f1 and f2 be the same node? Motivate your answer.

* **Q18:** Given a query to return inter-pangenome family pairs with annotations that are over 80% identical, which query is correct?

- MATCH (p:Pangenome)<-[:IS_IN_PANGENOME]-(f:Family) MATCH (p2:Pangenome)<-[:IS_IN_PANGENOME]-(f2:Family) MATCH (f)-[s:IS_similar]->(f2) WHERE s的身份 >= 0.8 AND p <> p2 AND f IS NOT NULL AND f2 IS NOT NULL RETURN f.name, f2.name

- MATCH (p1:Pangenome)-[:IS_IN_PANGENOME]-(f1:Family)-[r1:IS_similar]-(f2:Family)-[:IS_IN_PANGENOME]-(p2:Pangenome) WHERE r1的身份 >= 0.8 AND p1.taxid <> p2.taxid AND f1 annotation IS NOT NULL AND f2 annotation IS NOT NULL RETURN f1.name, f2.name

* **Q19:** Explain the differences between the queries provided in **Q18**.

## E Table extended versions

Table 7 is the extended version of Table 4 of the paper, reporting the *complications* and the *semantic return errors* performed by the students in the **schema discovery queries**. This extended table shows that none of the errors were related to advanced query concepts (as expected from the type of questions) such as the `LIMIT` and `ORDER BY` clauses. Moreover, none of the errors consists of a *semantic where error* or a *semantic pattern error*.

Table 8 is the extended version of Table 5 of the paper, reporting the *complications* and the *semantic errors* performed by the students in the **analytical queries**. In this phase, the *complications* caused by the overuse of `WITH` and `UNWIND` are absent. None of the students used a *Wrong Edge Label* when defining the pattern of the queries. Additional Condition
errors were also absent in this phase. Finally, as for the schema discovery queries, none of the students struggled with the LIMIT and ORDER BY clauses.

Table 9 is the extended version of Table 6 of the paper, reporting the *complications* and the *semantic errors* performed by the students in the **aggregation queries**. In this last phase, none of the *complications* was due to the use of `COLLECT` and `UNWIND`. As for the analytical queries, none of the students used a *Wrong Edge Label* when defining the pattern of the queries. For what concerns the *semantic where errors*, none of the students put a condition on a *wrong property* or used a *wrong* or *additional condition*. Finally, among the *semantic return errors*, *Wrong DISTINCT* and *Missing DISTINCT* are absent. Differently from the previous phases, some of the errors (10) are due to `LIMIT` on the wrong variable, while none of them concerns the `ORDER BY` clause.

<table><caption>Table 7. Number of error categories and type occurred writing schema discovery queries. We report the absolute number, the percentage of occurrence both within the category and with respect to the whole set of errors.</caption><thead><tr><th>Category</th><th>Type</th><th>Count</th><th>Count /<br>Category Total</th><th>Count / Total</th></tr></thead><tbody><tr><td rowspan="5">Complications</td><td>Added COLLECT</td><td>49</td><td>0.422</td><td>0.395</td></tr><tr><td>Added WITH</td><td>40</td><td>0.345</td><td>0.323</td></tr><tr><td>Added UNWIND</td><td>27</td><td>0.233</td><td>0.218</td></tr><tr><td>Additional information</td><td>0</td><td>0</td><td>0</td></tr><tr><td>Total</td><td>116</td><td>1</td><td>0.935</td></tr><tr><td rowspan="8">Semantic<br>Return<br>Errors</td><td>Missing DISTINCT</td><td>6</td><td>0.75</td><td>0.048</td></tr><tr><td>Missing Information</td><td>1</td><td>0.125</td><td>0.008</td></tr><tr><td>Wrong Information</td><td>1</td><td>0.125</td><td>0.008</td></tr><tr><td>Missing LIMIT</td><td>0</td><td>0</td><td>0</td></tr><tr><td>Missing ORDER BY</td><td>0</td><td>0</td><td>0</td></tr><tr><td>Wrong LIMIT</td><td>0</td><td>0</td><td>0</td></tr><tr><td>Wrong ORDER BY</td><td>0</td><td>0</td><td>0</td></tr><tr><td>Wrong DISTINCT</td><td>0</td><td>0</td><td>0</td></tr><tr><td></td><td>Total</td><td>8</td><td>1</td><td>0.065</td></tr><tr><td colspan="2">Total</td><td>124</td><td>1</td><td>1</td></tr></tbody></table>

<table><caption>Table 8. Number of error categories and type occurred writing analytical queries. We report the absolute number, the percentage of occurrence both within the category and with respect to the whole set of errors.</caption><thead><tr><th>Category</th><th>Type</th><th>Count</th><th>Count/<br>Category Total</th><th>Count / Total</th></tr></thead><tbody><tr><td rowspan="5">Complications</td><td>Added COLLECT</td><td>26</td><td>0.963</td><td>0.096</td></tr><tr><td>Added WITH</td><td>0</td><td>0</td><td>0</td></tr><tr><td>Added UNWIND</td><td>0</td><td>0</td><td>0</td></tr><tr><td>Additional Information</td><td>1</td><td>0.037</td><td>0.004</td></tr><tr><td>Total</td><td>27</td><td>1</td><td>0.1</td></tr><tr><td rowspan="7">Semantic<br>Pattern<br>Errors</td><td>Wrong Pattern w.r.t. Schema</td><td>18</td><td>0.29</td><td>0.067</td></tr><tr><td>Wrong Pattern</td><td>21</td><td>0.339</td><td>0.078</td></tr><tr><td>Missing Node</td><td>4</td><td>0.065</td><td>0.015</td></tr><tr><td>Additional Node</td><td>5</td><td>0.081</td><td>0.019</td></tr><tr><td>Wrong Edge Direction</td><td>14</td><td>0.226</td><td>0.052</td></tr><tr><td>Wrong Edge Label</td><td>0</td><td>0</td><td>0</td></tr><tr><td>Total</td><td>62</td><td>1</td><td>0.230</td></tr><tr><td rowspan="6">Semantic<br>Where<br>Errors</td><td>Wrong Property</td><td>31</td><td>0.34</td><td>0.115</td></tr><tr><td>Missing Condition</td><td>49</td><td>0.538</td><td>0.181</td></tr><tr><td>Wrong Condition</td><td>3</td><td>0.033</td><td>0.011</td></tr><tr><td>Wrong Operator</td><td>8</td><td>0.088</td><td>0.030</td></tr><tr><td>Additional Condition</td><td>0</td><td>0</td><td>0</td></tr><tr><td>Total</td><td>91</td><td>1</td><td>0.337</td></tr><tr><td rowspan="7">Semantic<br>Return<br>Errors</td><td>Missing DISTINCT</td><td>34</td><td>0.378</td><td>0.126</td></tr><tr><td>Missing Information</td><td>13</td><td>0.144</td><td>0.048</td></tr><tr><td>Wrong Information</td><td>41</td><td>0.456</td><td>0.152</td></tr><tr><td>Wrong DISTINCT</td><td>2</td><td>0.022</td><td>0.007</td></tr><tr><td>Missing LIMIT</td><td>0</td><td>0</td><td>0</td></tr><tr><td>Missing ORDER BY</td><td>0</td><td>0</td><td>0</td></tr><tr><td>Wrong LIMIT</td><td>0</td><td>0</td><td>0</td></tr><tr><td></td><td>Total</td><td>90</td><td>1</td><td>0.333</td></tr><tr><td colspan="2">Total</td><td>270</td><td>1</td><td>1</td></tr></tbody></table>
<table><caption>Table 9. Number of error categories and type occurred in writing aggregate queries. We report the absolute number, the percentage of occurrence both within the category and with respect to the whole set of errors</caption><thead><tr><th>Category</th><th>Type</th><th>Count</th><th>Count/Category Total</th><th>Count / Total</th></tr></thead><tbody><tr><td rowspan="5">Complications</td><td>Added COLLECT</td><td>0</td><td>0</td><td>0</td></tr><tr><td>Added WITH</td><td>2</td><td>0.5</td><td>0.038</td></tr><tr><td>Added UNWIND</td><td>0</td><td>0</td><td>0</td></tr><tr><td>Additional Information</td><td>2</td><td>0.5</td><td>0.038</td></tr><tr><td>Total</td><td>4</td><td>1</td><td>0.075</td></tr><tr><td rowspan="7">Semantic Pattern Errors</td><td>Wrong Pattern w.r.t. Schema</td><td>3</td><td>0.143</td><td>0.057</td></tr><tr><td>Wrong Pattern</td><td>1</td><td>0.048</td><td>0.019</td></tr><tr><td>Missing Node</td><td>1</td><td>0.048</td><td>0.019</td></tr><tr><td>Additional Node</td><td>3</td><td>0.0143</td><td>0.057</td></tr><tr><td>Wrong Edge Direction</td><td>13</td><td>0.619</td><td>0.245</td></tr><tr><td>Wrong Edge Label</td><td>0</td><td>0</td><td>0</td></tr><tr><td>Total</td><td>21</td><td>1</td><td>0.396</td></tr><tr><td rowspan="5">Semantic Where Errors</td><td>Wrong Property</td><td>0</td><td>0</td><td>0</td></tr><tr><td>Missing Condition</td><td>2</td><td>0.667</td><td>0.038</td></tr><tr><td>Wrong Condition</td><td>0</td><td>0</td><td>0</td></tr><tr><td>Wrong Operator</td><td>1</td><td>0.333</td><td>0.019</td></tr><tr><td>Additional Condition</td><td>0</td><td>0</td><td>0</td></tr><tr><td>Total</td><td>3</td><td>1</td><td>0.057</td></tr><tr><td rowspan="8">Semantic Return Errors</td><td>Missing DISTINCT</td><td>0</td><td>0</td><td>0</td></tr><tr><td>Missing Information</td><td>14</td><td>0.56</td><td>0.264</td></tr><tr><td>Wrong Information</td><td>1</td><td>0.040</td><td>0.019</td></tr><tr><td>Wrong DISTINCT</td><td>0</td><td>0</td><td>0</td></tr><tr><td>Missing LIMIT</td><td>0</td><td>0</td><td>0</td></tr><tr><td>Missing ORDER BY</td><td>0</td><td>0</td><td>0</td></tr><tr><td>Wrong LIMIT</td><td>10</td><td>0.4</td><td>0.189</td></tr><tr><td>Wrong ORDER BY</td><td>0</td><td>0</td><td>0</td></tr><tr><td>Total</td><td>25</td><td>1</td><td>0.472</td></tr><tr><td colspan="2">Total</td><td>53</td><td>1</td><td>1</td></tr></tbody></table>