## 1 Introduction  
The evolution of the Semantic Web has facilitated the creation and storage of structured knowledge [1; 2; 3; 4; 5; 6]. With the rapid development of the Semantic Web, various types of knowledge bases (KBs) have been proposed, such as DBpedia [7], Wikidata [8], and FreeBase [9]. Knowledge Base Question Answering (KBQA) systems, which have garnered considerable attention from researchers, enable users--particularly those without programming expertise--to effortlessly interact with Knowledge Bases [10; 11; 12; 13]. Central to KBQA is Complex Question Answering (CQA) which navigates queries containing multiple subjects, compound relations, or numerical operations, and is vital in applications like Apple's Siri and Microsoft's Cortana [14]. The fundamental challenge posed by this task is the translation of the user's natural language queries into a specialized knowledge base query language, such as SPARQL. A typical KBQA system consists of the following parts [11]: (1) Entity Linking (EL), (2) Relation Linking (RL), and (3) Query Building (QB). Figure 1 provides an illustrative example of this pipeline. Once entity linking and relation linking have been completed,the query building module coalesces this information into a SPARQL query. This paper places particular emphasis on the QB component, which is responsible for translating user queries into SPARQL queries when the gold entity and relation are already identified. Generating SPARQL queries is challenging due to the inherent differences in syntax and structure between natural language and formal query languages used in knowledge bases. Traditional rule-based and template-filling methods have limitations in handling complex and variable natural language inputs, often suffering from poor generalization capabilities [10; 11]. In recent years, methods based on neural networks, particularly those utilizing sequence-to-sequence models, have been increasingly adopted for this task. The most recent work employing pre-trained models like T5 [15] and BART [16], as well as BERT with Pointer-Generation Networks (PGN) [17], have significantly outperformed those traditional methods. However, despite achieving around 90% accuracy, state-of-the-art models still suffer from a significant number of \"Triple Flip\" errors, as pointed out in [11]. This type of error refers to the reversal of subject and object positions in the SPARQL triples generated by the model. For instance, given a correct SPARQL query  
SELECT (COUNT (?vr0) AS?vr1){wd:Q18813 wdt:P1574?vr0} where wd:Q18813 and wdt:P1574 are the Internationalized Resource Identifiers (IRI) for entity New Testament and relation exemplar of. The model may predict  
SELECT (COUNT (?vr0) AS?vr1) {?vr0 wdt:P1574 wd:Q18813} instead, where the positions of the subject and object are flipped. In this work, we address the triple flip error by introducing the **TSET** (Triplet **Structure** Enhanced T5) model, which exploits a novel pretraining stage positioned between the general T5 pretraining and the task-specific fine-tuning for Text-to-SPARQL parsing. The core of TSET is to enhance the model's understanding of triple-structure information within SPARQL queries. To achieve this, we incorporate a Triple Structure Correction (TSC) objective into the model's training regimen. This new objective is combined with the conventional Masked Language Modeling (MLM) objective to yield a more robust pretraining of the T5 model. Following the pretraining, we apply the TSET model to Text-to-SPARQL tasks using task-specific labeled data. To further improve the model's performance, we introduce a semantic transformation approach. In this approach, we map the IRI encoding inside the initial input to the corresponding literal during the proposed  
Figure 1: An Overview of the KBQA Pipeline: The user’s question will undergo processing through entity linking, relation linking, and SPARQL query building steps. The executed result will then be returned to the user. pretraining.