## 3 Proposed Method
### Reinforcement Learning with Support Tasks  
After supervising fine-tuning, we apply reinforcement learning with GRPO optimization policy to refine the fine-tuned baseline SLM. This stage includes prompt construction, a group of response generating, criteria checking, calculating the objective function, and model updating. This learning process is illustrated in Figure 3. \begin{table}
\begin{tabular}{|p{227.6 pt}|} \hline \multicolumn{2}{|p{227.6 pt}|}{Convert text to Cypher query based on this schema:} \\ The schema: \\ (Schema) \\ The text: \\ (Question) \\ \hline \end{tabular}
\end{table} 
Table 1: The example input prompt and output. \begin{table}
\begin{tabular}{|p{227.6 pt}|} \hline Node properties: \\ - Business \\ - address: STRING \\ - location: POINT \\ - city: STRING \\ - name: STRING \\ - User \\ - name: STRING \\ - userId: STRING \\ - Review \\ - date: DATE \\ - text: STRING \\ - Category \\ - name: STRING \\ The relationships: \\ (:Business)-:[IN \_CATEGORY]->(:Category) \\ (:User)-[:WROTE]->(:Review) \\ (:Review)-[:REVIEWS]->(:Business) \\ \hline \end{tabular}
\end{table} 
Table 2: Schema example. Prompt construction:  
In contrast to the supervised fine-tuning stage, we combine the schema context, input question, and a set of guidelines to construct the input, which is then fed into the fine-tuned baseline SLM. The guideline is introduced to support tasks, enabling the model to extract key-value pairs and relationship triples, in addition to generating a Cypher query. The guideline specifies the format that the model should follow. First, any reasoning-related terms are placed inside the \"reasoning\" tag. Within this tag, the model identifies and specifies the key-value pairs to extract from the input, which are useful for generating the Cypher query. These are placed inside the \"key_value\" tag. Next, in the \"relationship\" tag, the model selects the relevant relationship triples based on the schema. Finally, in the \"answer\" tag, the model generates the Cypher query corresponding to the input request. The guideline is illustrated, and the full input prompt is shown in Tables 3 and 4. \begin{table}
\begin{tabular}{|l|} \hline Response in the following format:  
Group of response generation and criteria checking:  
After the prompt is fed into the model, it generates a group of responses. The fine-tuned baseline SLM can be configured with a temperature parameter, which affects the randomness of the generated results. The temperature value ranges from 0 to 1, where a higher value increases randomness. In our experiments, we set the temperature to 0.6, allowing the SLM to generate a diverse set of responses for a single input request. For each response, we calculate a reward by applying the criteria checking. Pseudo code for reward function:  
_reward = 0 # This is reward value output # This is predicted output from model_  
**# 1. Format Checking**  
_format_result = check_format(output) # We can check using regular_  
**# expression**  
_If: format_result == True_  
_Then: reward += 1 Else: reward --= 1_  
**# 2. Answer Checking**  
_output_query = extract_cypher_query(output)_  
_If: output_query == ground truth Cypher query_  
_Then: reward += 1_  
_Else: reward --= 1_  
**# 3. Key-Value Checking**  
_list_key_value = extract_key_value(output)_  
_If: list_key_value \(\supseteq\) ground truth key-value_  
_Then: reward += 1_  
**Else: reward --= 1_**  
**# 4. Relationship Triple Checking**  
_output_triples = extract_relationship_triples(output)_  
_If: output_triples \(\supseteq\) ground truth triples_  
_Then: reward += 1_  
_Else: reward --= 1_  
_Return reward_  
We implement four reward functions for criteria checking, which assign a reward score to each response. These functions include the following:  
* Format checking: This function verifies if the output adheres to the specified guideline format. It ensures that the model extracts key-value pairs and relationship triples when generating the response;