## 5 Experiment
### Ablation Study
#### 5.3.2 Pretraining Objectives  
We also conducted additional experiments to analyze the relative contributions of the proposed pretraining objectives. Firstly, we explored the impact of removing either the TSC or MLM pretraining objective. As shown in Table 5, removing the TSC objective led to an almost 1-point decrease in the QM score, while removing the MLM objective resulted in a performance loss of 0.41.  
Further, we examined the TOC (Triple Order Correction) and SOC (Sentence Order Correction) objectives, which represent constrained and expanded variants of the proposed TSC objective, respectively. The TOC objective selectively randomizes the subject and object components of triples, whereas the SOC objective expands this permutation to encompass the entirety of a SPARQL query. Table 6 presents samples of pretraining input texts derived from the application of TOC, SOC, and TSC during the pretraining phase, illustrating the distinct permutations used. As is shown in Table 5, TOC+MLM objectives resulted in performance close to that of the TSC+MLM objectives but with a 0.18 performance drop. We could also find that the performance of SOC+MLM was significantly subpar in comparison to the TSC+MLM objectives.  
\begin{table}
\begin{tabular}{l l} \hline
**Type** & **Example Questions** \\ \hline Rank & What open cluster has the largest radius? \\ \hline Center & What is the total solar radiation reflected off of Saturn? \\ \hline Left-subgraph & Who married to of actor of Bepanaah? \\ \hline Right-subgraph & Which spot came from the Ebola virus? \\ \hline Boolean double one\_hop right subgraph & Did Pope [PERSON] work in both Rome and Munich? \\ \hline Boolean one\_hop right subgraph & Did [PERSON] play as a midfielder? \\ \hline Boolean with filter & Does the BMW M20B20 have a torque equal to 160? \\ \hline Simple question left & What is the name of the opera based on Twelfth Night? \\ \hline Simple question right & What is the capital of the Hamburg region? \\ \hline Statement\_property & As of 2009, how many people lived in Somalia? \\ \hline String matching simple contains word & What is the game name starts with z? \\ \hline String matching type + relation & What temple that belongs to the World Heritage starts with letter P? \\ \hline Two intentions right subgraph & What is the cause and place of [PERSON]’s death? \\ \hline \end{tabular}
\end{table}
Table 4: Different questions according to the type of LC-QuAD 2.0 dataset.  
\begin{table}
\begin{tabular}{l c c} \hline
**Approach** & **QM** & **Performance Drop** \\ \hline TSET & 92.04 & - \\ w/o ST & 90.14 & \(-1.90\) \\ w/o TSC & 91.07 & \(-0.97\) \\ w/o MLM & 91.63 & \(-0.41\) \\ \hline TOC instead of TSC & 91.60 & \(-0.44\) \\ TOC+MLM & 91.86 & \(-0.18\) \\ SOC instead of TSC & 91.32 & \(-0.72\) \\ SOC+MLM & 91.48 & \(-0.56\) \\ \hline \end{tabular}
\end{table}
Table 5: The ablation for different pretraining objectives. TOC and SOC are different variants of TSC. They perform random permutations either only on the subject and object within a triplet or on the entire SPARQL query.  
In conclusion, these experiments demonstrate the effectiveness of the proposed TSC objective. The larger structural permutations, as in the SOC objective, are unsuitable for this task. Hence, we conclude that the TSC objective, with its more targeted and smaller-scale structure permutation, is more appropriate for enhancing the model's understanding of triple-structure information in SPARQL queries.