# Robust Text-to-Cypher Using Combination of BERT, GraphSAGE, and Transformer (CoBGT) Model
## 4 Results
### Evaluation  
We compare our method with some other popular sequence-to-sequence model such as GPT2 [33] or T5 [32]. To evaluate these models on our dataset, we set up the following experiment: the input to each model consists of the schema information (schema field) and the question, while the output is a Cypher query. So, the input of those models will be as follows: \"With this schema: [_schema_]. Convert to Cypher query: [_question_]\". After training them on the training set, we evaluate their performance by measuring accuracy on the test dataset.  
We evaluate the performance of the proposed model using the Logical Form (LF) metric, also known as Exact Set Match Accuracy (EM). This metric is determined by comparing the predicted Cypher query to the ground-truth Cypher query. The calculation is as follows:  
\[Score_{LF}(\hat{Y},Y)=\begin{cases}1,\ \hat{Y}=Y\\ 0,\ \hat{Y}\
eq Y\end{cases} \tag{4}\]  
where \(\hat{Y}\) is the predicted Cypher query and \(Y\) is the ground truth Cypher query.  
\[EM=\frac{1}{N}{\sum_{n=1}^{N}Score_{LF}(\hat{Y}_{n},Y_{n}) \tag{5}\]  
Figure 9: A sample instance of text-to-Cypher dataset.  
where \(N\) is the number of instances in the dataset.  
The flowchart illustrating our evaluation process is shown in Figure 10.  
First, the input natural language question passes through the Key-Value Extraction module, which extracts key values (such as entity names or numbers) from the question. Next, the question is tokenized into individual words or terms and POS-tagged (we use the spaCy library). These tokens are then fed into the GraphSAGE model along with the schema. GraphSAGE processes the schema to identify relevant relationships and properties. To retain the most relevant relations and properties, matching scores are calculated using a specific formula (as Formula (3)).  
Finally, we use the Exact Match Calculation (EM) to compare the generated Cypher query against the ground-truth query, calculating the exact match accuracy.  
We simulate our experiment on the hardware as follows:  
* Intel(R) Core(TM) i9-10900X CPU @ 3.70 GHz;
* NVIDIA GeForce GTX 4090 24 GB;
* Memory 128 GB.  
In Table 1, we present the results of the proposed model and compare it with Text-to-Text Transfer Transformer (T5) and Generative Pretrained Transformer 2 (GPT2) models.  
In Table 2, we present the result of our proposed model in some study cases: without key extraction module (GraphSAGE-Transformers), without relation-properties extraction module (BERT-Transformers), and without both modules (only Transformers).  
\begin{table}
\begin{tabular}{c c} \hline \hline
**Method** & **EM** \\ \hline Proposed model & 87.1\% \\ T5 (base) & 48.06\% \\ GPT2 (standard) & 51.29\% \\ \hline \hline \end{tabular}
\end{table}
Table 1: Comparison of our method with T5 and GPT2.  
Figure 10: Evaluation workflow.  
In Table 3, we present some predictions output from the proposed model, with the question as the input to the model and the ground truth from the testing dataset.  
From Table 4, we can observe that our proposed model achieves a faster processing time than T5 and GPT-2, indicating that our model also has lower complexity or a smaller structure.