## 6 Experimental Results and Discussion
### Manual Evaluation of in-Context Triples by Different Methods  
In order to further evaluate whether the in-context triples retrieved by the proposed method are more effective for reasoning, in this experiment, we randomly select 200 query triples from the test set of FB15k-237, and invite experts to evaluate the effectiveness of the extracted triples for reasoning. Specifically, five experts in the field of natural language processing are invited and are required to score the effectiveness of the extracted in-context triples for reasoning based on the corresponding query triple, with a scoring range of 0-10, where a higher score represents higher effectiveness. Furthermore, for the convenience of comparison, we also utilize the triples connected to the query entities and ask the experts to score them as well. Table 5 shows an example. It displays the scores given by five experts on the in-context triples extracted by three different methods. \begin{table}
\begin{tabular}{c c c c c c c c} \hline \hline
**Method** & **Extracted in-Context Triples** & **Ex. 1** & **Ex. 2** & **Ex. 3** & **Ex. 4** & **Ex. 5** & **Average** \\ \hline \multirow{8}{*}{directly connected in-context triples} & 1. ([PERSON], occupation, scientist) & & & & & \\  & 2. ([PERSON], has won prize, & & & & & \\  & Nobel Prize) & & & & & & \\  & 3. ([PERSON], has friend, & & & & & \\  & [PERSON]) & & & & & & \\  & 4. ([PERSON], has gender, male) & & & & & & \\  & 5. ([PERSON], year of birth, 1961) & & & & & & \\ \hline \multirow{8}{*}{in-context triples} & 1. ([PERSON], has friend, & & & & & \\  & [PERSON]) & & & & & & \\  & 2. ([PERSON], graduated From, & & & & & \\  & Heidelberg University) & & & & & & \\  & 3. (Heidelberg University, is located in, & & & & & \\  & Heidelberg) & & & & & & \\  & 4. (Heidelberg, is located in, Germany) & & & & & & \\  & 5. ([PERSON], graduated from, & & & & & \\  & Massachusetts Institute of Technology) & & & & & & \\ \hline \multirow{8}{*}{in-context triples by logical rules and LLMs feedback} & 1. ([PERSON], works at, & & & & & \\  & University of Colorado Boulder) & & & & & & \\  & 2. (University of Colorado Boulder, & & & & & \\  & is located in, Colorado) & & & & & \\  & 3. ([PERSON], was born in, Palo Alto) & & & & & \\  & 4. (Palo Alto, is located in, California) & & & & & \\  & 5. ([PERSON], graduated from, & & & & & \\  & Massachusetts Institute of Technology) & & & & & \\ \hline \hline \end{tabular}
\end{table} 
Table 5: An example of manual scoring. The query triple is ([PERSON], lives in country,? ), then the in-context triples extracted by three different methods are displayed and scored by five experts manually. In Table 5, the query triple is ([PERSON], lives in country,? ), and the in-context triples extracted by the different methods are scored manually. Considering the in-context triples in row 1, we can see that all the five triples cannot help for the reasoning of the given query. Then, considering the in-context triples in row 3, the triples about the work place can provide useful clues for the reasoning. As a consequence, the scores in row 1 are quite low, and the scores in the last row are the highest. The statistics for the overall scores are shown in Table 6.  
Table 6 shows the average score of each expert for the effectiveness of the in-context triples extracted by different methods. \"Ex. 1\" is an abbreviation of \"Expert 1\", and the \"average\" in the last column is the average score of the former five columns. In the first row, the triples which are directly connected by the query entities are retrieved as in-context triples. In the second row, the in-context triples are obtained by the extractor, which is trained only by logical rules-guided triples. In the last row, the in-context triples are retrieved by our proposed method. We can see that the average scores of rows 2 and 3 are higher than those of row 1; these results demonstrate the effectiveness of refining the in-context triples. Finally, the scores of row 3 are the best.