SPARKLE: Enhancing SPARQL Generation with Direct KG Integration in Decoding
Authors: Jaebok Lee, Hyeonjeong Shin


Reviewer paper summaries: Text-to-SPARQL parsers often generate non-executable queries if they generate identifiers that do not exist in the knowledge base (KB) or if they generate (h,r,t) triples that do not exist in the KB. SPARKLE presents a constrained decoding approach for improving the performance of text-to-SPARQL parsers without modifying the underlying model. They propose 2 steps to constrain the parser:
1. *Entity and Relation Constraints*: During decoding, a trie is maintained to restrict the parser to only generate tokens that belong to valid KB identifiers (entity and relation names). This Trie structure is maintained in-memory to allow fast inference.
2. *Pruning*: When the head entity of a KB triple has been generated, it automatically restricts valid relations that can follow. Similarly, when a valid entity and relation have been generated, it restricts the tail entities that can follow (including variables). Thus, by referencing the KB, SPARKLE further prunes the possible continuations for a parse.

This strategy for constrained decoding is paired with beam search to explore multiple candidate parses. Overall, this process prevents the parser from generating non-executable queries.

Experiments show that the baseline BART parser shows significant performance gains on 2 KBQA datasets (SimpleQuestions and LCQuad 1.0). Moreover, while the approach does not lead to state-of-the-art performance on WebQSP, the constrained decoding shows gains over unconstrained decoding. Note that all 3 datasets function over different underlying knowledge bases.

By design, SPARKLE can handle scenarios where the KB is updated with new entities and edges. This is demonstrated by a small case study on DBPedia where SPARKLE is shown to be able to utilize new entities and edges without any changes to the base parser.

Since SPARKLE simply performs constrained decoding over the base parser, it does not cause significant slowdown and is amenable to batching.

This paper presents SPARKLE, an end-to-end framework designed to convert natural language questions into SPARQL queries. The framework addresses the challenges of traditional multi-stage Knowledge Base Question Answering (KBQA) systems, which often suffer from error propagation and inefficiency. SPARKLE innovatively integrates the structure of the knowledge base into the decoding process, ensuring the generation of valid SPARQL queries and allowing for adaptability to changes in the knowledge base without the need for retraining. SPARKLE is demonstrated its fast inference speed and ability to adapt through diverse experiments.
Summary of strengths: - By design, the method can be extended to new knowledge bases.
    - Caveat: The KB and related Tries must fit in the processor memory. Lookups to disk will slow down the system.
- By design, SPARKLE can easily integrate updates to the knowledge base without needing to retrain the parser
- The impact of the two proposed constraints is properly ablated
    - Entity/relation constraints contribute significantly to final performance. The effect of pruning is relatively small but present
- The proposed approach runs with minimal impact on inference time and can be batched
    - In fact, all baselines that produce candidate parses can benefit from SPARKLE entity and relation constraints (although this hasn't been established). Better candidates (that are more likely to be executable) are sure to lead to better system performance when coupled with other techniques.

S1. This paper introduces a novel end-to-end approach SPARKLE that directly incorporates knowledge base structure into the query generation process. This is a significant departure from traditional multi-stage models, offering a more streamlined and potentially more accurate solution.
S2. The framework achieves state-of-the-art results on the SimpleQuestions-Wiki dataset, showcasing its effectiveness. Additionally, it performs competitively on LCQuAD 1.0, indicating robustness across different datasets.
S3. SPARKLE can adapt to update in the knowledge base without requiring retraining. This adaptability is crucial for real-world applications where knowledge bases are frequently updated.
S4. SPARKLE is designed for fast inference, capable of processing batches of queries efficiently. This makes it suitable for real-time applications, a significant advantage over slower, multi-stage approaches.
S5. This paper includes thorough evaluations across multiple datasets and detailed analysis, enhancing the credibility of the results and providing a clear picture of the model's strengths and limitations.
Summary of weaknesses: 1. It is unclear to what extent and how often pruning can be applied. The results show strong performance on simple queries (chains of RDF triples). We immediately see diminishing returns on more complex queries (WebQSP)
    - Most of the gain in the system seems to stem from constraining the entity and relation identifiers to valid identifiers from the KB
    - Since pruning can only be applied when the predicted RDF triple starts with an entity identifier (and not a variable), pruning will not be triggered for many queries. When the program requires more than 2 hop relations from the question entity, pruning will only trigger on the first hop.
    - This would also limit the benefit of SPARKLE on complex SPARQL parsing datasets like CFQ, GrailQA, etc
2, Experiments on adaptive inference could be stronger
    - The first experiment uses just 2 queries to demonstrate that SPARKLE can utilize new entities and new edges. However, this doesn't show it can predict a new relation type. Predicting a new relation identifier would require the parser to put probability on tokens that it did not see in training. Using a new edge does not require the parser to do anything differently; the new edge will be handled by the constraint Trie.
    - Table 4 provides stronger support for the claim. The authors should further separate examples with unseen entity identifiers and unseen relation identifiers. Unseen entities identifier would still match entities that occur in the questions. Unseen relation identifiers would be harder to use at inference time.
3. [Minor] As mentioned in the paper, constrained decoding has been applied effectively in the past.
    - In this sense, SPARKLE is a strict subset of constraints studied in past work PICARD [1] for text-2-SQL parsing.
    - PICARD was designed for SQL parsing and cannot be easily extended to SPARQL parsing without extensive effort. However, entity & relation constraints in SPARKLE essentially adapt the lexing constraints of PICARD. Similarly, when SPARKLE can use pruning, it adapts some of the remaining PICARD constraints.

[1] PICARD: Parsing Incrementally for Constrained Auto-Regressive Decoding from Language Models. Torsten Scholak, Nathan Schucher, Dzmitry Bahdanau. EMNLP 2021

W1. While SPARKLE is the best on SimpleQuestions-Wiki, its performance on the WebQSP dataset is not that strong. This variability suggests that the model may have limitations in handling certain types of queries or specific knowledge base structures.
W2. This paper lacks a more in-depth analysis of failure cases, which can help understand why SPARKLE underperforms in certain scenarios could provide insights for further improvements.
W3. The comparisons in this paper are more focused on multi-stage approaches. It’s necessary to add a more detailed comparative analysis with other state-of-the-art end-to-end models.
W4. The process in this paper of integrating knowledge base structure into the decoding process, including the conversion of arbitrary identifiers to human-readable formats, is complex, which might hinder replication and extension by other researchers.
Comments, suggestions and typos: Clarifications
---
1. How often is pruning triggered on the three datasets considered? It will not be triggered if the predicted triple (h, r, t) in the parse starts with a variable as the head entity. This would be affected by how SPARQL queries are annotated in each dataset.
    - The authors can further segregate system performance into examples where pruning was triggered and examples where it was not.
2. How does tail-pruning (EQ (3)) work when the predicted relation is a one-to-many relation?
3. Line 518: It is unclear why the database dump matters when training the parser. Isn't the parser trained to predict the SPARQL program given the query? i.e., Isn't the parser training independent of the KB?

Suggestions
---
1. From anecdotal evidence, T5 models (encoder-decoder) perform better on semantic parsing than BART models (decoder-only). The poor performance of the BART parser could cause the discrepancy in WebQSP results (Table 2). I would recommend that the authors use a T5 model (possibly one comparable in size) like the DECAF and RnG-KBQA baselines

C1. Including more examples and visualizations of generated SPARQL queries, both valid and invalid, could help illustrate the model's advantages and limitations.
C2. Conducting ablation studies to isolate the impact of different components (e.g., constrained decoding, real-time knowledge base linkage) would provide valuable insights into the model's performance.
Overall assessment scores: 2.5

3