Relevant or Random: Can LLMs Truly Perform Analogical Reasoning?
Authors: Chengwei Qin, Wenhan Xia, Tan Wang, Fangkai Jiao, Yuchen Hu, Bosheng Ding, Ruirui Chen, Shafiq Joty
Accepted at: acl25

Relevant or Random: Can LLMs Truly Perform Analogical Reasoning?

Chengwei Qin*†*, Wenhan Xia*†*, Tan Wang†*, Fangkai Jiao†, Yuchen Hu†,
Bosheng Ding†, Ruirui Chen♦, Shafiq Joty†♦
*The Hong Kong University of Science and Technology (Guangzhou) • Princeton University
†Nanyang Technological University ♦Salesforce Research
♦Institute of High Performance Computing (IHPC),
Agency for Science, Technology and Research (A*STAR), Singapore

Abstract

Analogical reasoning is a unique ability of humans to address unfamiliar challenges by transferring strategies from relevant past experiences. One key finding in psychology is that compared with irrelevant past experiences, recalling relevant ones can help humans better handle new tasks. Coincidentally, the NLP community has also recently found that self-generating relevant examples in the context can help large language models (LLMs) better solve a given problem than hand-crafted prompts. However, it is yet not clear whether relevance is the key factor eliciting such capability, i.e., can LLMs benefit more from self-generated relevant examples than irrelevant ones? In this work, we systematically explore whether LLMs can truly perform analogical reasoning on a diverse set of reasoning tasks. With extensive experiments and analysis, we show that self-generated random examples can surprisingly achieve comparable or even better performance on certain tasks, e.g., 4% performance boost on GSM8K with random biological examples. We find that the accuracy of self-generated examples is the key factor and subsequently design two novel methods with improved performance and significantly reduced inference costs. Overall, we aim to advance a deeper understanding of LLM analogical reasoning and hope this work stimulates further research in the design of self-generated contexts.

1 Introduction

A hallmark of human intelligence is that they can solve novel problems by drawing analogy from relevant past experiences, a concept known as analogical reasoning in cognitive science (Vosniadou and Ortony, 1989). As indicated by the name, recalling previously acquired relevant experiences can facilitate humans to better tackle new tasks, whereas irrelevant ones are rarely beneficial and can even be distracting (Gentner and Smith, 2012). For instance, when faced with a novel math problem about determinants (e.g., calculating the value of a given fourth-order determinant), humans can resolve this by reflecting upon the methodology employed to ascertain the value of a third-order determinant, whereas biological knowledge (e.g., how the human body regulates its temperature) can generally be considered irrelevant.

With the recent advancements in scaling up model size and data, LLMs have demonstrated impressive zero-shot and few-shot performance across various reasoning tasks, especially, through advanced prompting methods like chain-of-thought (CoT) (Wei et al., 2022). Compared to common approaches such as zero or few-shot CoT (Zhou et al., 2022; Kojima et al., 2022; Zhang et al., 2023a), Yasunaga et al. (2024) introduce LLM analogical reasoning, i.e., LLMs self-generate examples rel-

![Illustration of LLM analogical reasoning in Yasunaga et al. (2024). LLMs are prompted to self-generate relevant examples as context before solving the new problem.](page_1012_370_367_246.png)

Figure 1: Illustration of LLM analogical reasoning in Yasunaga et al. (2024). LLMs are prompted to self-generate relevant examples as context before solving the new problem.
evant to the query as context to better solve new problems; see Fig. 1 for an example. However, it remains unclear whether relevance is the key to eliciting such capability in LLMs. While several studies explore the influence of the relevance of demonstrations in in-context learning (ICL) and CoT (Liu et al., 2022; Kim et al., 2022; Lyu et al., 2023; Chen et al., 2023; Yang et al., 2023; Wang et al., 2023a; Alkhamissi et al., 2023; Yasunaga et al., 2024; Luo et al., 2024), none of them investigate whether self-generated relevant examples consistently outperform irrelevant ones in LLM analogical reasoning.

In this paper, to systematically assess the capability of LLMs to perform analogical reasoning, we conduct a series of ablation experiments on a variety of reasoning tasks including problems from GSM8K (Cobbe et al., 2021), MATH (Hendrycks et al., 2021), and BIG-Bench Hard (BBH) (Suzgun et al., 2022). Furthermore, we evaluate the generalizability of our findings to other reasoning tasks, e.g., GPQA (Rein et al., 2024), in Section 4.3. With extensive experiments, we aim to address the following two research questions:

• Q1. Are self-generated relevant examples more beneficial to LLMs than random ones?
• Q2. If not, what is the pivotal factor for LLMs’ performance in analogical reasoning?

To answer these questions, we empirically analyze the analogical reasoning abilities of GPT-3.5 (turbo), GPT-4o-mini, the Llama series (Touvron et al., 2023), and Qwen 2.5 (Yang et al., 2024) models. Surprisingly, experimental results show that prompting LLMs to self-generate random examples can achieve comparable or even better performance on certain tasks which is not in line with the key claim of analogical reasoning in Gentner and Smith (2012), indicating that LLMs cannot always perform analogical reasoning. As for Q2, we point out through controlled experiments that the key factor is the accuracy of self-generated examples. Informed by these findings, we design two approaches that can outperform existing methods with significantly reduced inference costs. Specifically, we ask LLMs to randomly generate a few problems and manually verify their correctness, then use this fixed set of problems as in-context learning demonstrations for all test samples. Consistent observations across different model types and scales consolidate the conclusions. We summarize the major contributions of our work below:

• To the best of our knowledge, we, for the first time, extensively assess the ability of LLMs to perform analogical reasoning and explore their counterintuitive behavior on certain tasks.
• With extensive experiments and analysis, we demonstrate the effectiveness and limitations of different types of self-generated contexts.
• Building on the findings, we propose two novel ICL-based approaches that improve performance while significantly reducing inference costs.

2 Related Work

This work mainly explores whether LLMs can truly perform analogical reasoning. In light of this, we review two lines of research that form the basis of this work: chain-of-thought prompting and LLM analogical reasoning.

2.1 Chain-of-Thought Prompting

Chain-of-thought (CoT) prompting induces LLMs to generate intermediate reasoning steps before generating the final answer (Wei et al., 2022), greatly improving the reasoning capabilities of LLMs. Typical CoT prompting approaches include few-shot CoT (Wei et al., 2022; Zhou et al., 2022; Wang et al., 2022b; Li et al., 2022; Wang et al., 2022a), taking several labeled demonstrations of the reasoning process, and zero-shot CoT, comprising only instructions like “Let’s think step by step” (Kojima et al., 2022; Zelikman et al., 2022; Zhang et al., 2023a). Other ongoing research on CoT has also explored (i) optimizing the demonstration selection (Fu et al., 2022; Li and Qiu, 2023; Qin et al., 2024), (ii) optimizing the quality of reasoning chains (Khot et al., 2022; Chen et al., 2022; Shinn et al., 2023; Besta et al., 2024), and (iii) CoT in smaller models (Magister et al., 2022; Ho et al., 2022; Fu et al., 2023; Ranaldi and Freitas, 2024).

2.2 LLM Analogical Reasoning

While few-shot CoT can provide more detailed reasoning guidance, it requires labeled examples which can be unavailable for a new task. To tackle this problem, Yasunaga et al. (2024) propose analogical prompting to guide LLMs to self-generate relevant exemplars as few-shot demonstrations, which is similar to analogical reasoning, i.e., humans can address new problems by drawing analogy from relevant past experience (Vosniadou and Ortony, 1989; Holyoak, 2012). LBS3 (Luo et al.,
Prompt: self-generate relevant examples
Your task is to tackle mathematical problems. When presented with a math problem, recall relevant problems as examples. Afterward, proceed to solve the initial problem.

# Initial Problem: [The target problem]

# Instructions:
Make sure that your response follows the instructions below.

## Analogous Problems:
Offer five diverse examples of math problems that are relevant or analogous to the initial problem. For each problem, elaborate on the solution and conclude with the ultimate answer (enclosed in 'boxed{}'). For each problem:
- After "Q:", describe the problem
- After "A:", explain the solution and enclose the ultimate answer in 'boxed{}'.

## Solve the Initial Problem:
Q: Copy and paste the initial problem here.
A: Explain the solution and enclose the ultimate answer in 'boxed{}' here.

Prompt: self-generate random examples
Your task is to tackle mathematical problems. When presented with a math problem, recall random problems as examples. Afterward, proceed to solve the initial problem.

# Initial Problem: [The target problem]

# Instructions:
Make sure that your response follows the instructions below.

## Random Problems:
Randomly offer five diverse examples of math problems. For each problem, elaborate on the solution and conclude with the ultimate answer (enclosed in 'boxed{}'). For each problem:
- After "Q:", describe the problem
- After "A:", explain the solution and enclose the ultimate answer in 'boxed{}'.

## Solve the Initial Problem:
Q: Copy and paste the initial problem here.
A: Explain the solution and enclose the ultimate answer in 'boxed{}' here.

Figure 2: Example prompts for GSM8K (mathematical reasoning). Top: The original prompt used in Yasunaga et al. (2024) for self-generating relevant math problems. Bottom: The prompt designed for self-generating random math problems. We mark the differences between these two prompts in blue and green respectively.

2024) explores curriculum learning which can better reflect human learning habits. In this work, we step forward to explore the intrinsic principle of LLM analogical reasoning. Specifically, we aim to investigate whether LLMs can authentically exhibit such reasoning capabilities and determine the extent to which the relevance of self-generated examples contributes to enhancing this process.

3 Methodology

Our analysis is based on the analogical prompting approach outlined in Yasunaga et al. (2024). Specifically, for a given target problem \( x \), analogical prompting introduces instructions like:

# Problem: [x]
# Relevant problems: Recall five relevant and diverse problems. For each problem, describe it and explain the solution.
# Solve the initial problem:

The goal is to induce LLMs to self-generate relevant examples, aiding them to solve the target problem via in-context learning. To ensure better performance and efficiency, several key technical decisions are made in Yasunaga et al. (2024):
• The self-generated examples should be relevant and diverse, achieved through a specially designed instruction.
• Generate relevant problems and the solution to the initial problem in one pass.
• 3 to 5 self-generated examples perform the best.

In this work, we leverage similar prompts\(^1\) to guide LLMs to generate different types of irrelevant examples as context; see Fig. 2 for example prompts:

• N/A: generate problems that are N/A (not applicable) to the initial problem.
• Random\(_{same}\): randomly generate examples of the same problem type (\(e.g.\), math).
• Random\(_{diff}\): randomly generate examples of different problem types (\(e.g.\), any type except math).
• Random\(_{bio}\): randomly generate biological problems.

Yasunaga et al. (2024) demonstrate that self-

\(^1\)Since our work aims to comprehensively explore and analyze the intrinsic principle of LLM analogical reasoning proposed in Yasunaga et al. (2024), we should follow the original design of the instructions to have a fair comparison and reliable analysis.
generating relevant examples can consistently outperform zero-shot CoT and few-shot CoT (hand-crafted examples or retrieved top- \( k \) most similar training samples) on different tasks. Therefore, we do not include these two methods in our work. Interested readers can refer to the corresponding results and analysis in Yasunaga et al. (2024). In addition, we show prompts for different methods on all datasets in Appendix A.1.

4 Experiment

4.1 Experimental Setup

We construct the evaluation suite based on diverse reasoning-intensive tasks, including mathematical reasoning and other reasoning (\( e.g. \), logical and temporal reasoning) tasks:

• Mathematical reasoning. We work with two commonly used datasets, GSM8K (Cobbe et al., 2021) and MATH (Hendrycks et al., 2021). For each dataset, we randomly sample 500 examples from the original test set and run experiments three times with different random seeds (resulting in different test samples).

• Other reasoning. Following Yasunaga et al. (2024), we evaluate five reasoning tasks in BIG-Bench Hard (BBH) (Suzgun et al., 2022): temporal sequences (temporal reasoning), logical deduction five objects and reasoning about colored objects (logical reasoning), formal fallacies (deductive reasoning) and word sorting (symbolic reasoning). For each task, we use all test samples for evaluation and run experiments three times with different random seeds.

We mainly use GPT-3.5 (gpt-3.5-turbo) as the LLM (see Appendix A.3 for more results with GPT-4o-mini) and obtain all outputs from it with the temperature set to 0. We ask the LLM to self-generate 5 examples for GSM8K, 3 examples for MATH and BBH following Yasunaga et al. (2024).

4.2 Main Results

We now address the research questions asked in §1 with empirical results.

Q1. Are self-generated relevant examples more beneficial to LLMs than random ones?

The results averaged over all random seeds are reported in Table 1 and Table 2; more detailed results for every seed are shown in Appendix A.2.

Figure 3: Comparison of all methods at different difficulty levels on MATH. Level 1 represents the easiest and level 5 is the hardest. ‘relevant’ clearly performs worse than other approaches at all difficulty levels.

• Self-generated relevant examples achieve the best average performance on BBH. From the results in Table 1, we can observe that the superiority of self-generated relevant examples is empirically substantiated on BBH. Specifically, using relevant examples, denoted by ‘relevant’, outperforms other approaches on temporal and logical reasoning tasks. While it performs worse than ‘N/A’ on deductive and symbolical reasoning, it can still improve the accuracy by **1.3%** on average compared to ‘N/A’.

However, the results on mathematical reasoning tasks are quite counterintuitive as described below:

• Relevant examples do not guarantee better performance. Different from BBH, all types of self-generated irrelevant examples consistently outperform relevant ones on both mathematical reasoning datasets, showing that LLMs cannot yet perform analogical reasoning on these tasks. Interestingly, when we use randomly generated biological examples (\( e.g. \), how the process of photosynthesis occurs in plants), they can yield about **2.5%** better results on average compared to generating relevant math problems. Besides, ‘N/A’ achieves the best average result as it is the second-best on both datasets.

Problems in MATH span various subjects and difficulty levels. To investigate whether the inferior performance of relevant examples on MATH is accidentally caused by certain categories, we further report the accuracy across different subjects and difficulty levels in Table 3 and Fig. 3. The consistent performance gap between ‘relevant’ and other methods across different problem categories
<table>
  <tr>
    <th>Method</th>
    <th>Temporal sequences</th>
    <th>Logical deduction five objects</th>
    <th>Reasoning about colored objects</th>
    <th>Formal fallacies</th>
    <th>Word sorting</th>
    <th>Average</th>
  </tr>
  <tr>
    <td>Relevant</td>
    <td><b>60.0</b></td>
    <td><b>51.2</b></td>
    <td><b>76.7</b></td>
    <td>51.2</td>
    <td>76.9</td>
    <td><b>63.2</b></td>
  </tr>
  <tr>
    <td>N/A</td>
    <td>57.5</td>
    <td>45.3</td>
    <td>75.5</td>
    <td><b>53.3</b></td>
    <td><b>77.7</b></td>
    <td>61.9</td>
  </tr>
  <tr>
    <td>Random<sub>same</sub></td>
    <td>53.1</td>
    <td>48.8</td>
    <td>73.5</td>
    <td>52.4</td>
    <td>74.1</td>
    <td>60.4</td>
  </tr>
  <tr>
    <td>Random<sub>diff</sub></td>
    <td>44.3</td>
    <td>44.8</td>
    <td>72.4</td>
    <td>51.2</td>
    <td>69.2</td>
    <td>56.4</td>
  </tr>
  <tr>
    <td>Random<sub>bio</sub></td>
    <td>57.1</td>
    <td>49.5</td>
    <td>76.1</td>
    <td>50.8</td>
    <td>74.9</td>
    <td>61.7</td>
  </tr>
</table>

Table 1: Accuracy (%) of different methods on five reasoning tasks in BBH. Bold indicates the best results. Self-generated relevant examples achieve the best average performance. Detailed results for different seeds are reported in Appendix A.2.

<table>
  <tr>
    <th rowspan="2">Method</th>
    <th colspan="2">Task</th>
    <th rowspan="2">Average</th>
  </tr>
  <tr>
    <th>GSM8K</th>
    <th>MATH</th>
  </tr>
  <tr>
    <td>Relevant</td>
    <td>71.5</td>
    <td>33.3</td>
    <td>52.4</td>
  </tr>
  <tr>
    <td>N/A</td>
    <td>75.5</td>
    <td>36.1</td>
    <td><b>55.8</b></td>
  </tr>
  <tr>
    <td>Random<sub>same</sub></td>
    <td>75.1</td>
    <td><b>36.3</b></td>
    <td>55.7</td>
  </tr>
  <tr>
    <td>Random<sub>diff</sub></td>
    <td><b>76.3</b></td>
    <td>34.1</td>
    <td>55.2</td>
  </tr>
  <tr>
    <td>Random<sub>bio</sub></td>
    <td>75.3</td>
    <td>34.6</td>
    <td>54.9</td>
  </tr>
</table>

Table 2: Accuracy (%) of different methods on two mathematical reasoning tasks. Self-generated irrelevant examples are consistently better than relevant ones. Table 13 in Appendix A.2 reports detailed results for different seeds.

demonstrates the inherent flaws of relevant examples, indicating that mathematical reasoning tasks exhibit different analogical reasoning paradigms from BBH.

It might present challenges to prompt LLMs to accurately generate specific types of demonstrations. Therefore, given the unexpected results on mathematical reasoning tasks, one may wonder:

Q1-1. Are self-generated examples really relevant or irrelevant to the query?

To quantitatively measure the relevance between the generated examples and the query, we compute the average cosine similarity between them. Following Zhang et al. (2023a), we use the sentence transformer (Reimers and Gurevych, 2019) to encode all samples. For each method, the reported result is averaged across three seeds (see Appendix A.4 for the decomposition of relevance).

As observed from Table 4, relevant examples are much more semantically similar to the query than irrelevant ones and the relevance score of ‘relevant’ is more biased towards ‘oracle’ rather than ‘random’ or ‘N/A’, demonstrating that LLMs indeed follow instructions to generate specific types of demonstrations. Furthermore, we calculate the average similarity score between self-generated relevant examples and queries for BBH (0.46), which is slightly lower than the score of mathematical reasoning tasks (0.48). This result demonstrates that the difference in analogical reasoning performance between BBH (Table 1) and mathematical reasoning (Table 2) is not because LLMs can generate more relevant examples for BBH.

We provide a case study in Table 6 to delve deeper into the demonstrations of different methods. As we can notice, the example generated by ‘relevant’ is more related to the query as they both involve the mathematical concept ‘number bases’. In contrast, examples such as ‘What is the value of x in the equation 2x + 5 = 10?’ (N/A) or ‘How do you bake chocolate chip cookies?’ (Random<sub>diff</sub>) are less relevant to the query. This comparison highlights once again that relevance may not be the key factor for analogical reasoning performance on mathematical reasoning tasks. To understand better the underlying reasons for the counterintuitive results, we then ask the following question:

Q2. If relevance is not the key factor, what is more important for the accuracy of analogical reasoning?

Looking back at Table 6, an interesting observation is that the self-generated relevant example appears to be more difficult to solve than the irrelevant ones, regardless of whether they are math problems or not. Consequently, the accuracy of relevant examples may be lower. To verify this, we conduct a pilot experiment on MATH. Specifically, we randomly select 50 samples for different types of generated math problems, i.e., Relevant, N/A and Random<sub>same</sub>, and manually evaluate their ac-
<table>
  <tr>
    <th>Method</th>
    <th>Precalculus</th>
    <th>Intermediate Algebra</th>
    <th>Algebra</th>
    <th>Prealgebra</th>
    <th>Counting & Probability</th>
    <th>Geometry</th>
    <th>Number Theory</th>
  </tr>
  <tr>
    <td>Relevant</td>
    <td>10.4</td>
    <td>9.8</td>
    <td>51.8</td>
    <td>56.8</td>
    <td>22.1</td>
    <td>24.2</td>
    <td><b>37.0</b></td>
  </tr>
  <tr>
    <td>N/A</td>
    <td>9.1</td>
    <td>15.7</td>
    <td><b>55.5</b></td>
    <td><b>61.0</b></td>
    <td><b>28.7</b></td>
    <td><b>25.8</b></td>
    <td>34.2</td>
  </tr>
  <tr>
    <td>Random<sub>same</sub></td>
    <td>12.3</td>
    <td><b>17.6</b></td>
    <td>54.4</td>
    <td>60.6</td>
    <td>25.4</td>
    <td><b>25.8</b></td>
    <td>34.9</td>
  </tr>
  <tr>
    <td>Random<sub>diff</sub></td>
    <td><b>13.0</b></td>
    <td>14.1</td>
    <td>52.7</td>
    <td>56.8</td>
    <td>26.2</td>
    <td>24.2</td>
    <td>33.6</td>
  </tr>
  <tr>
    <td>Random<sub>bio</sub></td>
    <td><b>13.0</b></td>
    <td>12.2</td>
    <td>53.0</td>
    <td>59.2</td>
    <td><b>28.7</b></td>
    <td><b>25.8</b></td>
    <td>32.2</td>
  </tr>
</table>

Table 3: Accuracy (%) across different subjects in the MATH dataset. Self-generated irrelevant examples outperform relevant ones on 6 out of 7 subjects.

<table>
  <tr>
    <th>Method</th>
    <th>GSM8K</th>
    <th>MATH</th>
    <th>Average</th>
  </tr>
  <tr>
    <td>Relevant</td>
    <td><b>0.54</b></td>
    <td><b>0.41</b></td>
    <td><b>0.48</b></td>
  </tr>
  <tr>
    <td>N/A</td>
    <td>0.19</td>
    <td>0.28</td>
    <td>0.24</td>
  </tr>
  <tr>
    <td>Random<sub>same</sub></td>
    <td>0.30</td>
    <td>0.20</td>
    <td>0.25</td>
  </tr>
  <tr>
    <td>Random<sub>diff</sub></td>
    <td>0.15</td>
    <td>0.10</td>
    <td>0.13</td>
  </tr>
  <tr>
    <td>Random<sub>bio</sub></td>
    <td>0.06</td>
    <td>0.11</td>
    <td>0.09</td>
  </tr>
  <tr>
    <td>Oracle</td>
    <td>0.65</td>
    <td>0.63</td>
    <td>0.64</td>
  </tr>
</table>

Table 4: Average relevance score (semantic similarity) between self-generated examples and the query. 'Oracle' stands for the average similarity score between the query and \( k \) most similar training samples (\( k \) is the number of self-generated examples).

<table>
  <tr>
    <th></th>
    <th>Relevant</th>
    <th>N/A</th>
    <th>Random<sub>same</sub></th>
  </tr>
  <tr>
    <td>Accuracy</td>
    <td>62.0</td>
    <td>72.0</td>
    <td>86.0</td>
  </tr>
</table>

Table 5: Accuracy (%) of self-generated examples on the MATH dataset. The examples generated by 'relevant' are less accurate.

curacy. We exclude other methods as it is difficult to define the ‘accuracy’ of the examples they generate. From the results in Table 5, we can observe that while the examples generated by ‘relevant’ are more related to the test query, <i>they are less accurate</i>, raising the question whether the performance of different approaches on mathematical reasoning tasks is strongly correlated with the accuracy of self-generated examples.

Proxy Approaches However, as the accuracy of the examples located at the output cannot be directly controlled, we meticulously design a variant called <i>ICL</i>, which extracts the generated examples from the model output as in-context learning (ICL) demonstrations and combines them with the query as input to LLMs, as a proxy for the original method. We also consider the following two variants: (<i>a</i>) <i>GPT4-Calibration</i> which replaces the answers of demonstrations in <i>ICL</i> with GPT4-generated answers, and (<i>b</i>) <i>Random</i> changes the answers of demonstrations in <i>ICL</i> to random numbers. Our manual verification confirmed that GPT4-generated answers were mostly accurate. We conduct this experiment on GSM8K and MATH with GPT-3.5 as the LLM reasoner.

From the results of different variants reported in Table 7, we can see that increasing the accuracy of generated examples can indeed improve the performance: <i>GPT4-Calibration</i> consistently outperforms <i>ICL</i> by incorporating more accurate answers. In contrast, <i>random</i> always performs the worst among all variants. Therefore, the key factor influencing the performance on mathematical reasoning is <i>the accuracy of self-generated examples</i> rather than their relevance.

It is worthwhile to note that while several papers explore how the correctness of demonstration answers influences in-context learning (Min et al., 2022; Yoo et al., 2022; Wei et al., 2023; Pan et al., 2023; Kossen et al., 2024), our work differs from them in the following aspects: (<i>i</i>) The examples in our work are generated by LLMs rather than real data from NLP benchmarks, <i>i.e.</i>, randomly sampled from the training set. In addition, there are rationales (CoT) in self-generated examples, which are different from the input-label format of in-context learning investigated in these papers; and (<i>ii</i>) These studies mainly evaluate in-context learning on different classification or multi-choice datasets, <i>i.e.</i>, the output space is a finite set. In contrast, we are evaluating mathematical reasoning tasks, where the output space is infinite.

Given the above findings, a natural question is:

Q2-1. Can we ask the LLM to randomly generate a few math or biological problems and manually verify their correctness, then use this fixed set of problems as ICL demonstrations for all test queries?
Query: For how many ordered pairs \((A, B)\) where \(A\) and \(B\) are positive integers is \(AAA_7 + BBB_7 = 666_7\)?

<table>
  <tr>
    <th>Relevant</th>
    <td>In a certain base, the sum of two three-digit numbers is 777. If the digits of one of the numbers are reversed, the sum becomes 888. What is the base of this number system?</td>
  </tr>
  <tr>
    <th>N/A</th>
    <td>What is the value of \(x\) in the equation \(2x + 5 = 10\)?</td>
  </tr>
  <tr>
    <th>Random<sub>same</sub></th>
    <td>In a bag, there are 5 red marbles, 3 blue marbles, and 2 green marbles. If you randomly pick 2 marbles from the bag without replacement, what is the probability that both marbles are red?</td>
  </tr>
  <tr>
    <th>Random<sub>diff</sub></th>
    <td>How do you bake chocolate chip cookies?</td>
  </tr>
  <tr>
    <th>Random<sub>bio</sub></th>
    <td>How does the process of photosynthesis occur in plants?</td>
  </tr>
  <tr>
    <th>Oracle</th>
    <td>Find the number of ordered pairs \((a, b)\) of complex numbers such that \(a^3 b^5 = a^7 b^2 = 1\).</td>
  </tr>
</table>

Table 6: Demonstration examples of different methods on the MATH dataset. The example generated by ‘relevant’ is more related to the query than other examples generated by ‘N/A’ or ‘random’.

<table>
  <tr>
    <th rowspan="2">Variant</th>
    <th colspan="3">GSM8K</th>
    <th colspan="3">MATH</th>
  </tr>
  <tr>
    <th>Relevant</th>
    <th>N/A</th>
    <th>Random<sub>same</sub></th>
    <th>Relevant</th>
    <th>N/A</th>
    <th>Random<sub>same</sub></th>
  </tr>
  <tr>
    <td>ICL</td>
    <td>71.2</td>
    <td>73.8</td>
    <td>72.0</td>
    <td>37.0</td>
    <td>39.8</td>
    <td>39.2</td>
  </tr>
  <tr>
    <td>GPT4-Calibration</td>
    <td><b>75.2</b></td>
    <td><b>75.6</b></td>
    <td><b>75.6</b></td>
    <td><b>44.4</b></td>
    <td><b>41.2</b></td>
    <td><b>40.0</b></td>
  </tr>
  <tr>
    <td>Random</td>
    <td>70.0</td>
    <td>72.0</td>
    <td>68.4</td>
    <td>36.0</td>
    <td>38.0</td>
    <td>37.8</td>
  </tr>
</table>

Table 7: Accuracy (%) of different variants on GSM8K and MATH. When using GPT4-generated answers (mostly accurate), ‘GPT4-Calibration’ consistently outperforms ‘ICL’ for all methods. In contrast, ‘random’ always performs worse than ‘ICL’.

<table>
  <tr>
    <th rowspan="2">Method</th>
    <th colspan="2">Task</th>
    <th rowspan="2">Average</th>
  </tr>
  <tr>
    <th>GSM8K</th>
    <th>MATH</th>
  </tr>
  <tr>
    <td>Relevant</td>
    <td>71.5</td>
    <td>33.3</td>
    <td>52.4</td>
  </tr>
  <tr>
    <td>N/A</td>
    <td>75.5</td>
    <td>36.1</td>
    <td>55.8</td>
  </tr>
  <tr>
    <td>Random<sub>same</sub></td>
    <td>75.1</td>
    <td>36.3</td>
    <td>55.7</td>
  </tr>
  <tr>
    <td>Random<sub>diff</sub></td>
    <td>76.3</td>
    <td>34.1</td>
    <td>55.2</td>
  </tr>
  <tr>
    <td>Random<sub>bio</sub></td>
    <td>75.3</td>
    <td>34.6</td>
    <td>54.9</td>
  </tr>
  <tr>
    <td>ICL<sub>math</sub></td>
    <td>75.7</td>
    <td><b>36.8</b></td>
    <td>56.3</td>
  </tr>
  <tr>
    <td>ICL<sub>bio</sub></td>
    <td><b>77.9</b></td>
    <td>34.9</td>
    <td><b>56.4</b></td>
  </tr>
</table>

Table 8: Comparison of different methods on two mathematical reasoning tasks.

We refer to these two methods as ICL<sub>math</sub> and ICL<sub>bio</sub>, and conduct experiments with them on GSM8K and MATH (see Fig. 4 for example prompts and outputs for generating math problems). Detailed prompts and outputs for different methods are provided in Appendix A.5. Following the original setting, we ask the LLM to randomly generate 5 examples for GSM8K and 3 examples for MATH. As observed from Table 8, ensuring the accuracy of self-generated examples does lead to better performance regardless of the problem type. ICL<sub>math</sub> and ICL<sub>bio</sub> achieve similar average performance, once again demonstrating that relevance does not matter (see Appendix A.6 for more analysis on relevance). Moreover, both ICL variants only need to generate examples once, which significantly reduces the inference cost and further demonstrates their superiority.

4.3 Further Analysis

Difference from Previous Work Apart from the comprehensive analysis, we have designed two novel ICL-based approaches that are completely different from the one in Yasunaga et al. (2024) (Q2-1). The difference lies mainly in the following two aspects: (i) The key claim in Yasunaga et al. (2024) is that we should guide the model to self-generate relevant examples as context. Motivated by the analysis and findings in our work (Q1 and Q2), our methods focus on ensuring the accuracy of self-generated examples rather than their relevance, which leads to better performance regardless of the problem type. (ii) As we have demonstrated that the relevance of self-generated examples does not matter, there is no need to generate relevant examples for each test query (the original method in Yasunaga et al. (2024)). In contrast, our methods use a fixed set of examples for all test queries, which significantly reduces the inference cost.

Generalization to Open-Source LLMs Our experiments and analysis so far used GPT-3.5 as the LLM, which is closed-source and gets updated over time. To verify whether the observations and conclusions are consistent across different models and
Input
Randomly recall five examples of math problems. Note that your problems should be distinct from each other (e.g., involving different numbers and names). For each problem:
- After “Q: ”, describe the problem
- After “A: ”, explain the solution and enclose the ultimate answer in \boxed{}.

Output
Q: If John has 5 apples and he buys 8 more, how many apples does he have in total?
A: To find the total number of apples John has, we add the number of apples he initially has to the number of apples he buys. So, we calculate 5 + 8 = 13. Therefore, John has \boxed{13} apples in total.
...
Q: Find the derivative of the function f(x) = 3x^2 + 2x - 5.
A: To find the derivative of f(x), we apply the power rule to each term. The derivative of 3x^2 is 6x, the derivative of 2x is 2, and the derivative of a constant -5 is 0. Therefore, the derivative of f(x) is f'(x) = 6x + 2. So, the derivative is \boxed{6x + 2}.

Figure 4: Example prompts and outputs for randomly generating math problems. We manually verify the answers to ensure the correctness of the generated examples.

<table>
  <tr>
    <th>Method</th>
    <th>Relevant</th>
    <th>N/A</th>
    <th>Random<sub>same</sub></th>
    <th>Random<sub>diff</sub></th>
    <th>Random<sub>bio</sub></th>
    <th>ICL<sub>math</sub></th>
    <th>ICL<sub>bio</sub></th>
  </tr>
  <tr>
    <td>Llama-2-70b-Chat</td>
    <td>45.1</td>
    <td>51.4</td>
    <td>50.9</td>
    <td>54.3</td>
    <td>47.1</td>
    <td>55.5</td>
    <td><b>56.1</b></td>
  </tr>
  <tr>
    <td>Llama-3-8B-Instruct</td>
    <td>69.5</td>
    <td>72.3</td>
    <td>72.6</td>
    <td>74.1</td>
    <td>73.5</td>
    <td>75.8</td>
    <td><b>76.8</b></td>
  </tr>
  <tr>
    <td>Llama-3.1-8B-Instruct</td>
    <td>74.8</td>
    <td>77.3</td>
    <td>78.4</td>
    <td>78.8</td>
    <td>77.6</td>
    <td>80.2</td>
    <td><b>81.0</b></td>
  </tr>
  <tr>
    <td>Qwen2.5-14B-Instruct</td>
    <td>86.5</td>
    <td>89.1</td>
    <td>88.2</td>
    <td>89.7</td>
    <td>88.4</td>
    <td><b>91.1</b></td>
    <td>90.6</td>
  </tr>
</table>

Table 9: Accuracy (%) of different methods on GSM8K using Llama-2-70b-Chat, Llama-3-8B-Instruct, Llama-3.1-8B-Instruct and Qwen2.5-14B-Instruct models. Self-generated relevant examples always perform worse than irrelevant ones and both ICL variants outperform other approaches.

<table>
  <tr>
    <th rowspan="2">Variant</th>
    <th colspan="3">Method</th>
  </tr>
  <tr>
    <th>Relevant</th>
    <th>N/A</th>
    <th>Random<sub>same</sub></th>
  </tr>
  <tr>
    <td>ICL</td>
    <td>56.2</td>
    <td>58.2</td>
    <td>58.6</td>
  </tr>
  <tr>
    <td>GPT4-Calibration</td>
    <td><b>60.8</b></td>
    <td><b>61.0</b></td>
    <td><b>60.8</b></td>
  </tr>
  <tr>
    <td>Random</td>
    <td>53.2</td>
    <td>54.0</td>
    <td>59.6</td>
  </tr>
</table>

Table 10: Accuracy (%) of different variants on GSM8K using Llama-2-70b-Chat. ‘GPT4-Calibration’ consistently performs better than ‘ICL’ and ‘random’.

additionally for reproducibility, we extend the experiments to Llama-2-Chat (Touvron et al., 2023). Specifically, we use vLLM to serve a Llama-2-70b-Chat model for experiments and report the results of different methods/variants on GSM8K in Table 9 and Table 10. We can draw similar observations: (i) self-generated relevant examples underperform all types of irrelevant ones, (ii) ‘GPT4-Calibration’ consistently outperforms the other two variants, and (iii) ICL<sub>math</sub> and ICL<sub>bio</sub> perform better than other approaches, demonstrating that the conclusions can be generalized to different models.

We further conduct experiments with Llama-3-8B-Instruct, Llama-3.1-8B-Instruct (Dubey et al., 2024) and Qwen2.5-14B-Instruct (Yang et al., 2024). The results reported in Table 9 demonstrate the generalizability of the conclusions across different model types and scales. In addition, since investigating analogical reasoning requires LLMs to self-generate different types of problems, we only experiment with instruction-tuned LLMs to ensure that they can follow the given instructions.

Generalization to Different Tasks To test the generalizability of our findings beyond the math domain, we further conduct experiments on CommonsenseQA (commonsense reasoning) (Talmor et al., 2019), MBPP (code generation) (Austin et al., 2021) and GPQA (question answering of very hard questions) (Rein et al., 2024). The comparison between different methods is shown in Table 11, which demonstrates that our findings can be generalized to different types of tasks.

Comparison Beyond Analogical Reasoning We consider two widely used methods Self-consistency (Wang et al., 2023b) and Auto-CoT (Zhang et al., 2023b), and compare our designed approaches with them on GSM8K using Llama-3.1-8B-Instruct. For Self-consistency, we employ 5 decoding paths for majority voting. The results reported in Table 12 demonstrate that our methods can also outperform other baselines beyond analogical reasoning.

In addition, we show the robustness to prompt format, the effect of the number of demonstrations,
<table>
  <tr>
    <th>Dataset</th>
    <th>Relevant</th>
    <th>N/A</th>
    <th>Random<sub>same</sub></th>
    <th>Random<sub>diff</sub></th>
    <th>Random<sub>bio</sub></th>
    <th>ICL<sub>same</sub></th>
    <th>ICL<sub>bio</sub></th>
  </tr>
  <tr>
    <td>CSQA</td>
    <td>70.8</td>
    <td>73.4</td>
    <td>71.2</td>
    <td>72.9</td>
    <td>72.6</td>
    <td><b>74.6</b></td>
    <td>74.1</td>
  </tr>
  <tr>
    <td>MBPP</td>
    <td>58.2</td>
    <td>59.8</td>
    <td>60.6</td>
    <td>59.6</td>
    <td>60.2</td>
    <td><b>62.0</b></td>
    <td>61.4</td>
  </tr>
  <tr>
    <td>GPQA</td>
    <td>31.6</td>
    <td>34.4</td>
    <td>33.7</td>
    <td>33.1</td>
    <td>32.6</td>
    <td>35.8</td>
    <td><b>36.2</b></td>
  </tr>
</table>

Table 11: Accuracy (%) of different methods on CommonsenseQA, MBPP, and GPQA. ‘same’ in ICL<sub>same</sub> stands for ‘generating correct problems of the same type as the dataset’.

<table>
  <tr>
    <th>Relevant</th>
    <th>Self-consistency</th>
    <th>Auto-CoT</th>
    <th>ICL<sub>math</sub></th>
    <th>ICL<sub>bio</sub></th>
  </tr>
  <tr>
    <td>74.8</td>
    <td>77.6</td>
    <td>75.9</td>
    <td>80.2</td>
    <td><b>81.0</b></td>
  </tr>
</table>

Table 12: Comparison between our designed methods and baselines beyond analogical reasoning.

more analysis on ICL<sub>math</sub> and ICL<sub>bio</sub>, the results of repeating problems and explicitly controlling the semantics of generated examples in Appendix A.7 ~ A.11, respectively.

5 Conclusion

In this work, we have systematically assessed the capability of LLMs to perform analogical reasoning. We have identified key research questions and empirically analyzed a representative set of LLMs on a diverse collection of reasoning tasks. Extensive experimental results and analysis show that LLMs cannot always perform analogical reasoning and the key influencing factor is the accuracy of self-generated examples rather than their relevance. Given these findings, we have designed two ICL-based approaches with better performance and significantly reduced inference costs. In the future, we would like to investigate additional analogical prompting methods to generate more accurate examples.

Limitations

This work has several limitations. First, due to the inference cost of ChatGPT, we conduct experiments on subsets of the test data for mathematical reasoning tasks. Besides, we include 6 datasets requiring different reasoning capabilities in this work. A further improvement could be to explore more diverse types of tasks.

References

Badr Alkhamissi, Siddharth Verma, Ping Yu, Zhijing Jin, Asli Celikyilmaz, and Mona Diab. 2023. OPT-R: Exploring the role of explanations in finetuning and prompting for reasoning skills of large language models. In Proceedings of the 1st Workshop on Natural Language Reasoning and Structured Explanations (NLRSE), pages 128–138, Toronto, Canada. Association for Computational Linguistics.

Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. 2021. Program synthesis with large language models. arXiv preprint arXiv:2108.07732.

Maciej Besta, Nils Blach, Ales Kubicek, Robert Gerstenberger, Michal Podstawski, Lukas Gianinazzi, Joanna Gajda, Tomasz Lehmann, Hubert Niewiadomski, Piotr Nyczyk, et al. 2024. Graph of thoughts: Solving elaborate problems with large language models. In Proceedings of the AAAI Conference on Artificial Intelligence, 16, pages 17682–17690.

Wei-Lin Chen, Cheng-Kuang Wu, Yun-Nung Chen, and Hsin-Hsi Chen. 2023. Self-ICL: Zero-shot in-context learning with self-generated demonstrations. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 15651–15662, Singapore. Association for Computational Linguistics.

Wenhui Chen, Xueguang Ma, Xinyi Wang, and William W Cohen. 2022. Program of thoughts prompting: Disentangling computation from reasoning for numerical reasoning tasks. arXiv preprint arXiv:2211.12588.

Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. 2021. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168.

Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783.

Yao Fu, Hao Peng, Litu Ou, Ashish Sabharwal, and Tushar Khot. 2023. Specializing smaller language models towards multi-step reasoning. arXiv preprint arXiv:2301.12726.

Yao Fu, Hao Peng, Ashish Sabharwal, Peter Clark, and Tushar Khot. 2022. Complexity-based prompting for multi-step reasoning. arXiv preprint arXiv:2210.00720.
D. Gentner and L. Smith. 2012. Analogical reasoning. In V.S. Ramachandran, editor, Encyclopedia of Human Behavior (Second Edition), second edition edition, pages 130–136. Academic Press, San Diego.

Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. 2021. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.03874.

Namgyu Ho, Laura Schmid, and Se-Young Yun. 2022. Large language models are reasoning teachers. arXiv preprint arXiv:2212.10071.

Keith J Holyoak. 2012. Analogy and relational reasoning. The Oxford handbook of thinking and reasoning, pages 234–259.

Tushar Khot, Harsh Trivedi, Matthew Finlayson, Yao Fu, Kyle Richardson, Peter Clark, and Ashish Sabharwal. 2022. Decomposed prompting: A modular approach for solving complex tasks. arXiv preprint arXiv:2210.02406.

Hyuhng Joon Kim, Hyunsoo Cho, Junyeob Kim, Taeuk Kim, Kang Min Yoo, and Sang-goo Lee. 2022. Self-generated in-context learning: Leveraging auto-regressive language models as a demonstration generator. arXiv preprint arXiv:2206.08082.

Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yu-taka Matsuo, and Yusuke Iwasawa. 2022. Large language models are zero-shot reasoners. In Thirty-sixth Conference on Neural Information Processing Systems (NeurIPS 2022).

Jannik Kossen, Yarin Gal, and Tom Rainforth. 2024. In-context learning learns label relationships but is not conventional learning. In The Twelfth International Conference on Learning Representations.

Xiaonan Li and Xipeng Qiu. 2023. Finding support examples for in-context learning. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 6219–6235. Singapore. Association for Computational Linguistics.

Yifei Li, Zeqi Lin, Shizhuo Zhang, Qiang Fu, Bei Chen, Jian-Guang Lou, and Weizhu Chen. 2022. On the advance of making language models better reasoners. arXiv preprint arXiv:2206.02336.

Jiachang Liu, Dinghan Shen, Yizhe Zhang, Bill Dolan, Lawrence Carin, and Weizhu Chen. 2022. What makes good in-context examples for GPT-3? In Proceedings of Deep Learning Inside Out (DeeLIO 2022): The 3rd Workshop on Knowledge Extraction and Integration for Deep Learning Architectures, pages 100–114, Dublin, Ireland and Online. Association for Computational Linguistics.

Kangyang Luo, Zichen Ding, Zhenmin Weng, Lingfeng Qiao, Meng Zhao, Xiang Li, Di Yin, and Jinlong Shu. 2024. Let’s be self-generated via step by step: A curriculum learning approach to automated reasoning with large language models. arXiv preprint arXiv:2410.21728.

Xinxi Lyu, Sewon Min, Iz Beltagy, Luke Zettlemoyer, and Hannaneh Hajishirzi. 2023. Z-ICL: Zero-shot in-context learning with pseudo-demonstrations. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2304–2317, Toronto, Canada. Association for Computational Linguistics.

Lucie Charlotte Magister, Jonathan Mallinson, Jakub Adamek, Eric Malmi, and Aliaksei Severyn. 2022. Teaching small language models to reason. ArXiv preprint, abs/2212.08410.

Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer. 2022. Rethinking the role of demonstrations: What makes in-context learning work? In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 11048–11064, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.

Jane Pan, Tianyu Gao, Howard Chen, and Danqi Chen. 2023. What in-context learning “learns” in-context: Disentangling task recognition and task learning. In Findings of the Association for Computational Linguistics: ACL 2023, pages 8298–8319, Toronto, Canada. Association for Computational Linguistics.

Chengwei Qin, Aston Zhang, Chen Chen, Anirudh Dagar, and Wenming Ye. 2024. In-context learning with iterative demonstration selection. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 7441–7455, Miami, Florida, USA. Association for Computational Linguistics.

Leonardo Ranaldi and Andre Freitas. 2024. Aligning large and small language models via chain-of-thought reasoning. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1812–1827, St. Julian’s, Malta. Association for Computational Linguistics.

Nils Reimers and Iryna Gurevych. 2019. Sentence-BERT: Sentence embeddings using Siamese BERT-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3982–3992, Hong Kong, China. Association for Computational Linguistics.

David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R. Bowman. 2024. GPQA: A graduate-level google-proof q&a benchmark. In First Conference on Language Modeling.

Noah Shinn, Federico Cassano, Beck Labash, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. 2023. Reflexion: Language agents with verbal reinforcement learning.
Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc V Le, Ed H Chi, Denny Zhou, et al. 2022. Challenging big-bench tasks and whether chain-of-thought can solve them. arXiv preprint arXiv:2210.09261.

Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. 2019. CommonsenseQA: A question answering challenge targeting commonsense knowledge. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4149–4158, Minneapolis, Minnesota. Association for Computational Linguistics.

Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Sounya Batra, Prajijwal Bhargava, Shruti Bhosale, et al. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288.

Stella Vosniadou and Andrew Ortony. 1989. Similarity and analogical reasoning. Cambridge University Press.

Boshi Wang, Sewon Min, Xiang Deng, Jiaming Shen, You Wu, Luke Zettlemoyer, and Huan Sun. 2023a. Towards understanding chain-of-thought prompting: An empirical study of what matters. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2717–2739, Toronto, Canada. Association for Computational Linguistics.

Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, and Denny Zhou. 2022a. Rationale-augmented ensembles in language models. arXiv preprint arXiv:2207.00747.

Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, and Denny Zhou. 2022b. Self-consistency improves chain of thought reasoning in language models. arXiv preprint arXiv:2203.11171.

Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2023b. Self-consistency improves chain of thought reasoning in language models. In The Eleventh International Conference on Learning Representations.

Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Chi, Quoc Le, and Denny Zhou. 2022. Chain of thought prompting elicits reasoning in large language models. In Thirty-sixth Conference on Neural Information Processing Systems (NeurIPS 2022).

Jerry Wei, Jason Wei, Yi Tay, Dustin Tran, Albert Webson, Yifeng Lu, Xinyun Chen, Hanxiao Liu, Da Huang, Denny Zhou, et al. 2023. Larger language models do in-context learning differently. arXiv preprint arXiv:2303.03846.

An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyu Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. 2024. Qwen2.5 technical report. arXiv preprint arXiv:2412.15115.

Jinghan Yang, Shuming Ma, and Furu Wei. 2023. Autoicl: In-context learning without human supervision. arXiv preprint arXiv:2311.09263.

Michihiro Yasunaga, Xinyun Chen, Yuja Li, Panupong Pasupat, Jure Leskovec, Percy Liang, Ed H. Chi, and Denny Zhou. 2024. Large language models as analogical reasoners. In The Twelfth International Conference on Learning Representations.

Kang Min Yoo, Junyeob Kim, Hyuhng Joon Kim, Hyunsoo Cho, Hwiyeol Jo, Sang-Woo Lee, Sang-goo Lee, and Taeku Kim. 2022. Ground-truth labels matter: A deeper look into input-label demonstrations. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 2422–2437, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.

Eric Zelikman, Yuhuai Wu, and Noah D Goodman. 2022. Star: Bootstrapping reasoning with reasoning. arXiv preprint arXiv:2203.14465.

Zhuosheng Zhang, Aston Zhang, Mu Li, and Alex Smola. 2023a. Automatic chain of thought prompting in large language models. In The Eleventh International Conference on Learning Representations (ICLR 2023).

Zhuosheng Zhang, Aston Zhang, Mu Li, and Alex Smola. 2023b. Automatic chain of thought prompting in large language models. In The Eleventh International Conference on Learning Representations.

Denny Zhou, Nathanael Schärli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuurmans, Olivier Bousquet, Quoc Le, and Ed Chi. 2022. Least-to-most prompting enables complex reasoning in large language models. arXiv preprint arXiv:2205.10625.

A Appendix

A.1 Prompts for Different Methods
The prompts for different methods on all datasets are shown in Fig. 5 ~ Fig. 7.

A.2 Detailed Results for Different Random Seeds
We report detailed results for different random seeds in Table 13 ~ Table 14.

A.3 Results with GPT-4o-mini
We conduct experiments with GPT-4o-mini on GSM8K and present the results in Table 15, verifying the generalizability of our findings to GPT-4o-mini.
<table>
  <tr>
    <th rowspan="2">Seed</th>
    <th colspan="5">GSM8K</th>
    <th colspan="5">MATH</th>
  </tr>
  <tr>
    <th>Relevant</th>
    <th>N/A</th>
    <th>Random<sub>same</sub></th>
    <th>Random<sub>diff</sub></th>
    <th>Random<sub>bio</sub></th>
    <th>Relevant</th>
    <th>N/A</th>
    <th>Random<sub>same</sub></th>
    <th>Random<sub>diff</sub></th>
    <th>Random<sub>bio</sub></th>
  </tr>
  <tr>
    <td>42</td>
    <td>71.8</td>
    <td><b>76.6</b></td>
    <td>73.2</td>
    <td>74.0</td>
    <td>74.0</td>
    <td>37.4</td>
    <td><b>42.2</b></td>
    <td>41.6</td>
    <td>39.0</td>
    <td>39.2</td>
  </tr>
  <tr>
    <td>100</td>
    <td>71.2</td>
    <td>75.2</td>
    <td>75.2</td>
    <td><b>75.8</b></td>
    <td>74.8</td>
    <td>29.0</td>
    <td>30.6</td>
    <td><b>32.6</b></td>
    <td>29.4</td>
    <td>31.2</td>
  </tr>
  <tr>
    <td>1000</td>
    <td>71.4</td>
    <td>74.8</td>
    <td>77.0</td>
    <td><b>79.2</b></td>
    <td>77.0</td>
    <td>33.6</td>
    <td><b>35.6</b></td>
    <td>34.6</td>
    <td>34.0</td>
    <td>33.4</td>
  </tr>
  <tr>
    <td>Average</td>
    <td>71.5<sub>±0.3</sub></td>
    <td>75.5<sub>±0.8</sub></td>
    <td>75.1<sub>±1.5</sub></td>
    <td><b>76.3<sub>±2.1</sub></b></td>
    <td>75.3<sub>±1.2</sub></td>
    <td>33.3<sub>±3.4</sub></td>
    <td><b>36.1<sub>±4.7</sub></b></td>
    <td><b>36.3<sub>±3.8</sub></b></td>
    <td>34.1<sub>±3.9</sub></td>
    <td>34.6<sub>±3.3</sub></td>
  </tr>
</table>

Table 13: Accuracy (%) of all methods with different random seeds on two mathematical reasoning tasks.

<table>
  <tr>
    <th rowspan="2">Seed</th>
    <th colspan="2">Temporal sequences</th>
    <th colspan="2">Logical deduction five objects</th>
    <th colspan="2">Reasoning about colored objects</th>
    <th colspan="2">Formal fallacies</th>
    <th colspan="2">Word sorting</th>
    <th colspan="1">Average</th>
  </tr>
  <tr>
    <th>Relevant</th>
    <th>N/A</th>
    <th>Relevant</th>
    <th>N/A</th>
    <th>Relevant</th>
    <th>N/A</th>
    <th>Relevant</th>
    <th>N/A</th>
    <th>Relevant</th>
    <th>N/A</th>
    <th>Average</th>
  </tr>
  <tr>
    <td rowspan="4">42</td>
    <td>58.0</td>
    <td>52.8</td>
    <td>76.0</td>
    <td>50.4</td>
    <td>77.2</td>
    <td>62.9</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>56.4</td>
    <td>44.8</td>
    <td><b>77.6</b></td>
    <td><b>54.0</b></td>
    <td>76.8</td>
    <td>61.9</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>52.4</td>
    <td>48.8</td>
    <td>74.8</td>
    <td>51.6</td>
    <td>72.8</td>
    <td>60.1</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>43.2</td>
    <td>46.8</td>
    <td>74.0</td>
    <td>52.4</td>
    <td>67.6</td>
    <td>56.8</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td rowspan="4">100</td>
    <td>58.4</td>
    <td>50.8</td>
    <td>78.4</td>
    <td>51.2</td>
    <td>76.8</td>
    <td>63.1</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>55.2</td>
    <td>46.0</td>
    <td>74.8</td>
    <td>52.8</td>
    <td><b>79.2</b></td>
    <td>61.6</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>50.8</td>
    <td>48.4</td>
    <td>73.6</td>
    <td><b>53.2</b></td>
    <td>75.2</td>
    <td>60.2</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>46.4</td>
    <td>46.8</td>
    <td>72.8</td>
    <td>50.0</td>
    <td>70.4</td>
    <td>57.3</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td rowspan="4">1000</td>
    <td>58.0</td>
    <td>48.4</td>
    <td><b>78.4</b></td>
    <td>51.2</td>
    <td>73.6</td>
    <td>61.9</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>63.6</td>
    <td>50.0</td>
    <td>75.6</td>
    <td>52.0</td>
    <td>76.8</td>
    <td><b>63.6</b></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>60.8</td>
    <td>45.2</td>
    <td>74.0</td>
    <td><b>53.2</b></td>
    <td><b>77.2</b></td>
    <td>62.1</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>56.0</td>
    <td>49.2</td>
    <td>72.0</td>
    <td>52.4</td>
    <td>74.4</td>
    <td>60.8</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td rowspan="4">Average</td>
    <td>60.0<sub>±2.6</sub></td>
    <td><b>51.2<sub>±1.2</sub></b></td>
    <td><b>76.7<sub>±1.2</sub></b></td>
    <td>51.2<sub>±0.7</sub></td>
    <td>76.9<sub>±0.2</sub></td>
    <td><b>63.2<sub>±0.3</sub></b></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>57.5<sub>±2.4</sub></td>
    <td>45.3<sub>±0.5</sub></td>
    <td>75.5<sub>±1.5</sub></td>
    <td><b>53.3<sub>±0.5</sub></b></td>
    <td><b>77.7<sub>±1.0</sub></b></td>
    <td>61.9<sub>±0.2</sub></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>53.1<sub>±2.1</sub></td>
    <td>48.8<sub>±0.3</sub></td>
    <td>73.5<sub>±1.1</sub></td>
    <td>52.4<sub>±0.6</sub></td>
    <td>74.1<sub>±1.0</sub></td>
    <td>60.4<sub>±0.3</sub></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>44.3<sub>±1.5</sub></td>
    <td>44.8<sub>±2.8</sub></td>
    <td>72.4<sub>±1.5</sub></td>
    <td>51.2<sub>±1.0</sub></td>
    <td>69.2<sub>±1.2</sub></td>
    <td>56.4<sub>±1.0</sub></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td rowspan="4">Random<sub>bio</sub></td>
    <td>57.1<sub>±0.7</sub></td>
    <td>49.5<sub>±1.8</sub></td>
    <td>76.1<sub>±1.8</sub></td>
    <td>50.8<sub>±1.2</sub></td>
    <td>74.9<sub>±1.1</sub></td>
    <td>61.7<sub>±0.6</sub></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
</table>

Table 14: Accuracy (%) of all methods with different random seeds on BBH.

A.4 Decomposition of Relevance

The relevance can be further separated into semantic relevance and procedure (reasoning steps) relevance. Our analysis in Q1-1 has demonstrated that semantic relevance does not matter. To investigate the importance of procedure relevance, we perform a similar analysis. Specifically, we compute the average cosine similarity between the rationales of the generated examples and the rationale of the query to quantitatively measure their relevance. The results on GSM8K are reported in Table 16, which highlight that procedure relevance is not the key factor for analogical reasoning performance on mathematical reasoning tasks.

A.5 Prompts and Outputs for Example Generation

We show detailed prompts and outputs for randomly generating math and biological problems in Fig. 8 and Fig. 9, respectively.

A.6 Guided Problem Generation

In addition to random problem generation in §4.2-Q2-1, we further investigate guided problem generation. Specifically, we randomly select 5 training samples to guide LLMs to self-generate relevant math problems. We then manually verify their correctness and use this fixed set of problems as ICL demonstrations for experiments. The performance of this approach (56.1) is slightly lower than that of ICL<sub>math</sub> (56.3), verifying that relevance is not the key influencing factor.

A.7 Robustness to Prompt Format

To verify the robustness of different methods to prompt format, we experiment with two new prompts paraphrased from the original one by GPT-4 and present the results on GSM8K in Table 17.
<table>
  <tr>
    <th>Method</th>
    <th>Relevant</th>
    <th>N/A</th>
    <th>Random_same</th>
    <th>Random_diff</th>
    <th>Random_bio</th>
    <th>ICL_math</th>
    <th>ICL_bio</th>
  </tr>
  <tr>
    <td>GPT-4o-mini</td>
    <td>90.7</td>
    <td>91.9</td>
    <td>92.6</td>
    <td>92.3</td>
    <td>93.2</td>
    <td>94.2</td>
    <td><b>94.5</b></td>
  </tr>
</table>

Table 15: Accuracy (%) of different methods on GSM8K using GPT-4o-mini. Self-generated relevant examples always underperform irrelevant ones and both ICL variants perform better than other approaches.

<table>
  <tr>
    <th></th>
    <th>Relevant</th>
    <th>N/A</th>
    <th>Random_same</th>
    <th>Random_diff</th>
    <th>Random_bio</th>
    <th>Oracle</th>
  </tr>
  <tr>
    <td>GSM8K</td>
    <td><b>0.50</b></td>
    <td>0.16</td>
    <td>0.28</td>
    <td>0.19</td>
    <td>0.08</td>
    <td>0.62</td>
  </tr>
</table>

Table 16: Procedure (reasoning steps) relevance between self-generated examples and the query.

<table>
  <tr>
    <th></th>
    <th>Relevant</th>
    <th>N/A</th>
    <th>Random_same</th>
    <th>Random_diff</th>
    <th>Random_bio</th>
  </tr>
  <tr>
    <td>Prompt_1</td>
    <td>71.2</td>
    <td>74.9</td>
    <td>75.3</td>
    <td><b>75.9</b></td>
    <td>74.3</td>
  </tr>
  <tr>
    <td>Prompt_2</td>
    <td>72.0</td>
    <td>75.2</td>
    <td>74.7</td>
    <td><b>76.2</b></td>
    <td>75.5</td>
  </tr>
</table>

Table 17: Accuracy (%) of different methods with two new prompts.

<table>
  <tr>
    <th>Number</th>
    <th>Relevant</th>
    <th>N/A</th>
    <th>Random_same</th>
    <th>Random_diff</th>
    <th>Random_bio</th>
  </tr>
  <tr>
    <td>3</td>
    <td>73.1</td>
    <td><b>77.3</b></td>
    <td>75.0</td>
    <td>75.3</td>
    <td>75.5</td>
  </tr>
  <tr>
    <td>5</td>
    <td>71.5</td>
    <td>75.5</td>
    <td>75.1</td>
    <td><b>76.3</b></td>
    <td>75.3</td>
  </tr>
</table>

Table 18: Accuracy (%) of all methods with different numbers of demonstrations.

<table>
  <tr>
    <th>ICL_math</th>
    <th>ICL_math<sup>wrong</sup></th>
    <th>ICL_bio</th>
    <th>ICL_bio<sup>wrong</sup></th>
  </tr>
  <tr>
    <td>56.3</td>
    <td>50.9</td>
    <td>56.4</td>
    <td>51.3</td>
  </tr>
</table>

Table 19: Comparison between different ICL variants.

We also observe better performance with irrelevant examples than relevant ones, showing the robustness.

A.8 Different Numbers of Demonstrations

While we mainly follow the setting in Yasunaga et al. (2024) to ask the LLM to generate \( k = 5 \) examples for GSM8K, we further investigate the effect of the number of demonstrations. Specifically, we conduct controlled experiments with \( k = 3 \) and report the results in Table 18. We can observe that irrelevant examples consistently outperform relevant ones across different numbers of demonstrations, emphasizing their effectiveness.

A.9 More Analysis on ICL_math and ICL_bio

Our designed method ICL_math generates correct and relevant examples, and ICL_bio generates correct and irrelevant examples. From the results in Table 8, we can see that ICL_math and ICL_bio achieve similar average performance, demonstrating that relevance does not matter.

We further change the correct answers of the demonstrations in ICL_math and ICL_bio to random answers, obtaining ICL_math<sup>wrong</sup> and ICL_bio<sup>wrong</sup>. Obviously, ICL_math<sup>wrong</sup> generates incorrect and relevant examples, and ICL_bio<sup>wrong</sup> generates incorrect and irrelevant examples. The comparison between these four methods in Table 19 further supports our claim that the key factor influencing the performance on mathematical reasoning is the accuracy of self-generated examples rather than their relevance.

A.10 Repeating Problems

While generating a few accurate problems as ICL demonstrations can achieve better performance, a bolder idea might be to generate one problem and repeat it multiple times as few-shot demonstrations for ICL. To investigate this, we randomly select a generated math problem and repeat it to perform ICL, denoted by ICL_math_repeat. From the results shown in Table 20, we can see that ICL_math_repeat consistently performs worse than ICL_math on both datasets, indicating that the diversity of generated problems also matters.

A.11 Explicit Semantic Control

We explore explicitly controlling the semantics of generated examples (including both problems and reasoning paths) on GSM8K using Llama-3.1-8B-Instruct. Specifically, we investigate the following two approaches: (i) prompting the model to generate semantically similar and correct examples, and (ii) prompting the model to generate semantically different and correct examples. The results reported in Table 21 further verify the correctness of our conclusions.
<table>
  <tr>
    <th rowspan="2">Method</th>
    <th colspan="3">Task</th>
  </tr>
  <tr>
    <th>GSM8K</th>
    <th>MATH</th>
    <th>Average</th>
  </tr>
  <tr>
    <td>ICL-math</td>
    <td><b>75.7</b></td>
    <td><b>36.8</b></td>
    <td><b>56.3</b></td>
  </tr>
  <tr>
    <td>ICL-math_repeat</td>
    <td>73.8</td>
    <td>36.2</td>
    <td>55.0</td>
  </tr>
</table>

Table 20: Comparison of two ICL variants on the GSM8K and MATH datasets.

<table>
  <tr>
    <th>Relevant</th>
    <th>N/A</th>
    <th>Random_same</th>
    <th>Similar and Correct</th>
    <th>Different and Correct</th>
  </tr>
  <tr>
    <td>74.8</td>
    <td>77.3</td>
    <td>78.4</td>
    <td>80.3</td>
    <td><b>80.6</b></td>
  </tr>
</table>

Table 21: Accuracy (%) of different methods on GSM8K using Llama-3.1-8B-Instruct.
Prompt: self-generate relevant examples
Your task is to tackle mathematical problems. When presented with a math problem, recall relevant problems as examples. Afterward, proceed to solve the initial problem.

# Initial Problem:
[The target problem]

# Instructions:
Make sure that your response follows the instructions below.

## Analogous Problems:
Offer five diverse examples of math problems that are relevant or analogous to the initial problem. For each problem, elaborate on the solution and conclude with the ultimate answer (enclosed in 'boxed[]'). For each problem:
- After "O": describe the problem
- After "A": explain the solution and enclose the ultimate answer in 'boxed[]'.

## Solve the Initial Problem:
Q: Copy and paste the initial problem here.
A: Explain the solution and enclose the ultimate answer in 'boxed[]' here.

Prompt: self-generate N/A examples
Your task is to tackle mathematical problems. When presented with a math problem, recall n/a problems as examples. Afterward, proceed to solve the initial problem.

# Initial Problem:
[The target problem]

# Instructions:
Make sure that your response follows the instructions below.

## N/A Problems:
Offer five diverse examples of math problems that are n/a to the initial problem. For each problem, elaborate on the solution and conclude with the ultimate answer (enclosed in 'boxed[]'). For each problem:
- After "O": describe the problem
- After "A": explain the solution and enclose the ultimate answer in 'boxed[]'.

## Solve the Initial Problem:
Q: Copy and paste the initial problem here.
A: Explain the solution and enclose the ultimate answer in 'boxed[]' here.

Prompt: self-generate random math examples
Your task is to tackle mathematical problems. When presented with a math problem, recall random problems as examples. Afterward, proceed to solve the initial problem.

# Initial Problem:
[The target problem]

# Instructions:
Make sure that your response follows the instructions below.

## Random Problems:
Randomly offer five diverse examples of math problems. For each problem, elaborate on the solution and conclude with the ultimate answer (enclosed in 'boxed[]'). For each problem:
- After "O": describe the problem
- After "A": explain the solution and enclose the ultimate answer in 'boxed[]'.

## Solve the Initial Problem:
Q: Copy and paste the initial problem here.
A: Explain the solution and enclose the ultimate answer in 'boxed[]' here.

Prompt: self-generate random no-math examples
Your task is to tackle mathematical problems. When presented with a math problem, recall random problems (remember not to output math problems) as examples. Afterward, proceed to solve the initial problem.

# Initial Problem:
[The target problem]

# Instructions:
Make sure that your response follows the instructions below.

## Random Problems:
Randomly offer five diverse examples of any type, except math problems. For each problem, elaborate on the solution and conclude with the ultimate answer (enclosed in 'boxed[]'). For each problem:
- After "O": describe the problem
- After "A": explain the solution and enclose the ultimate answer in 'boxed[]'.

## Solve the Initial Problem:
Q: Copy and paste the initial problem here.
A: Explain the solution and enclose the ultimate answer in 'boxed[]' here.

Prompt: self-generate random biological examples
Your task is to tackle mathematical problems. When presented with a math problem, recall random biological problems (remember not to output math problems) as examples. Afterward, proceed to solve the initial problem.

# Initial Problem:
[The target problem]

# Instructions:
Make sure that your response follows the instructions below.

## Random Problems:
Randomly offer five diverse examples of biological problems (remember not to output math problems). For each problem, elaborate on the solution and conclude with the ultimate answer (enclosed in 'boxed[]'). For each problem:
- After "O": describe the problem
- After "A": explain the solution and enclose the ultimate answer in 'boxed[]'.

## Solve the Initial Problem:
Q: Copy and paste the initial problem here.
A: Explain the solution and enclose the ultimate answer in 'boxed[]' here.

Figure 5: Prompts for different methods on GSM8K.
Prompt: self-generate relevant examples
Your task is to tackle mathematical problems. When presented with a math problem, recall relevant problems as examples. Afterward, proceed to solve the initial problem.

# Initial Problem:
[The target problem]

# Instructions:
Make sure to include all of the following points:

## Relevant Problems:
Recall three examples of math problems that are relevant to the initial problem. Note that your problems should be distinct from each other and from the initial problem (e.g., involving different numbers and names). For each problem:
- After "Q:", describe the problem
- After "A:", explain the solution and enclose the ultimate answer in [boxed()].

## Solve the Initial Problem:
Say "Let's solve the following math problem." Then formulate your response in the following format:
Q: Copy and paste the initial problem here.
A: Explain the solution and enclose the ultimate answer in [boxed()] here.

Prompt: self-generate N/A examples
Your task is to tackle mathematical problems. When presented with a math problem, recall n/a problems as examples. Afterward, proceed to solve the initial problem.

# Initial Problem:
[The target problem]

# Instructions:
Make sure to include all of the following points:

## N/A Problems:
Recall three examples of math problems that are n/a to the initial problem. Note that your problems should be distinct from each other and from the initial problem (e.g., involving different numbers and names). For each problem:
- After "Q:", describe the problem
- After "A:", explain the solution and enclose the ultimate answer in [boxed()].

## Solve the Initial Problem:
Say "Let's solve the following math problem." Then formulate your response in the following format:
Q: Copy and paste the initial problem here.
A: Explain the solution and enclose the ultimate answer in [boxed()] here.

Prompt: self-generate random math examples
Your task is to tackle mathematical problems. When presented with a math problem, recall random problems as examples. Afterward, proceed to solve the initial problem.

# Initial Problem:
[The target problem]

# Instructions:
Make sure to include all of the following points:

## Random Problems:
Randomly recall three examples of math problems. Note that your problems should be distinct from each other and from the initial problem (e.g., involving different numbers and names). For each problem:
- After "Q:", describe the problem
- After "A:", explain the solution and enclose the ultimate answer in [boxed()].

## Solve the Initial Problem:
Say "Let's solve the following math problem." Then formulate your response in the following format:
Q: Copy and paste the initial problem here.
A: Explain the solution and enclose the ultimate answer in [boxed()] here.

Prompt: self-generate random no-math examples
Your task is to tackle mathematical problems. When presented with a math problem, recall random problems (remember not to output math problems) as examples. Afterward, proceed to solve the initial problem.

# Initial Problem:
[The target problem]

# Instructions:
Make sure to include all of the following points:

## Random Problems:
Randomly recall three examples of any type, except math problems. Note that your problems should be distinct from each other and from the initial problem. For each problem:
- After "Q:", describe the problem
- After "A:", explain the solution and enclose the ultimate answer in [boxed()].

## Solve the Initial Problem:
Say "Let's solve the following math problem." Then formulate your response in the following format:
Q: Copy and paste the initial problem here.
A: Explain the solution and enclose the ultimate answer in [boxed()] here.

Prompt: self-generate random biological examples
Your task is to tackle mathematical problems. When presented with a math problem, recall random biological problems (remember not to output math problems) as examples. Afterward, proceed to solve the initial problem.

# Initial Problem:
[The target problem]

# Instructions:
Make sure to include all of the following points:

## Random Problems:
Randomly recall three examples of biological problems (remember not to output math problems). Note that your problems should be distinct from each other and from the initial problem. For each problem:
- After "Q:", describe the problem
- After "A:", explain the solution and enclose the ultimate answer in [boxed()].

## Solve the Initial Problem:
Say "Let's solve the following math problem." Then formulate your response in the following format:
Q: Copy and paste the initial problem here.
A: Explain the solution and enclose the ultimate answer in [boxed()] here.

Figure 6: Prompts for different methods on MATH.
Prompt: self-generate relevant examples

Your task is to tackle reasoning problems. When presented with a problem, recall relevant problems as examples. Afterward, proceed to solve the initial problem.

# Initial Problem:
[The target problem]

# Instructions:
Make sure to include all of the following points:

## Relevant Problems:
Recall three examples of problems that are relevant to the initial problem. Note that your problems must be distinct from each other and from the initial problem. For each problem:
- After “Q.” - describe the problem
- After “A.” - explain the solution and enclose the ultimate answer in [boxed[]].

## Solve the Initial Problem:
Say “Let’s solve the following reasoning problem.” Then formulate your response in the following format:
Q: Copy and paste the initial problem here.
A: Explain the solution and enclose the ultimate answer in [boxed[]] here.

Prompt: self-generate N/A examples

Your task is to tackle reasoning problems. When presented with a problem, recall n/a problems as examples. Afterward, proceed to solve the initial problem.

# Initial Problem:
[The target problem]

# Instructions:
Make sure to include all of the following points:

## N/A Problems:
Recall three examples of problems that are n/a to the initial problem. Note that your problems must be distinct from each other and from the initial problem. For each problem:
- After “Q.” - describe the problem
- After “A.” - explain the solution and enclose the ultimate answer in [boxed[]].

## Solve the Initial Problem:
Say “Let’s solve the following reasoning problem.” Then formulate your response in the following format:
Q: Copy and paste the initial problem here.
A: Explain the solution and enclose the ultimate answer in [boxed[]] here.

Prompt: self-generate random reasoning examples

Your task is to tackle reasoning problems. When presented with a problem, recall random problems as examples. Afterward, proceed to solve the initial problem.

# Initial Problem:
[The target problem]

# Instructions:
Make sure to include all of the following points:

## Random Problems:
Randomly recall three examples of reasoning problems. Note that your problems must be distinct from each other and from the initial problem. For each problem:
- After “Q.” - describe the problem
- After “A.” - explain the solution and enclose the ultimate answer in [boxed[]].

## Solve the Initial Problem:
Say “Let’s solve the following reasoning problem.” Then formulate your response in the following format:
Q: Copy and paste the initial problem here.
A: Explain the solution and enclose the ultimate answer in [boxed[]] here.

Prompt: self-generate random no-reasoning examples

Your task is to tackle reasoning problems. When presented with a problem, recall random problems (remember not to output reasoning problems) as examples. Afterward, proceed to solve the initial problem.

# Initial Problem:
[The target problem]

# Instructions:
Make sure to include all of the following points:

## Random Problems:
Randomly recall three examples of any type, except reasoning problems. Note that your problems must be distinct from each other and from the initial problem. For each problem:
- After “Q.” - describe the problem
- After “A.” - explain the solution and enclose the ultimate answer in [boxed[]].

## Solve the Initial Problem:
Say “Let’s solve the following reasoning problem.” Then formulate your response in the following format:
Q: Copy and paste the initial problem here.
A: Explain the solution and enclose the ultimate answer in [boxed[]] here.

Prompt: self-generate random biological examples

Your task is to tackle reasoning problems. When presented with a problem, recall random biological problems (remember not to output reasoning problems) as examples. Afterward, proceed to solve the initial problem.

# Initial Problem:
[The target problem]

# Instructions:
Make sure to include all of the following points:

## Random Problems:
Randomly recall three examples of biological problems (remember not to output reasoning problems). Note that your problems must be distinct from each other and from the initial problem. For each problem:
- After “Q.” - describe the problem
- After “A.” - explain the solution and enclose the ultimate answer in [boxed[]].

## Solve the Initial Problem:
Say “Let’s solve the following reasoning problem.” Then formulate your response in the following format:
Q: Copy and paste the initial problem here.
A: Explain the solution and enclose the ultimate answer in [boxed[]] here.

Figure 7: Prompts for different methods on BBH.
Input
Randomly recall five examples of math problems. Note that your problems should be distinct from each other (e.g., involving different numbers and names). For each problem:
- After "Q:", describe the problem
- After "A:", explain the solution and enclose the ultimate answer in 'boxed()'.

Output
Q: If John has 5 apples and he buys 8 more, how many apples does he have in total?
A: To find the total number of apples John has, we add the number of apples he initially has to the number of apples he buys. So, we calculate 5 + 8 = 13. Therefore, John has 'boxed(13)' apples in total.

Q: Solve for x in the equation 2x - 4 = 10.
A: To solve for x, we first add 4 to both sides of the equation to get 2x = 14. Then, we divide both sides by 2 to isolate x, resulting in x = 7. Thus, the solution is 'boxed(x = 7)'.

Q: A rectangle has a length of 10 cm and a width of 4 cm. What is its area?
A: The area of a rectangle is found by multiplying its length by its width. Therefore, the area of this rectangle is 10 * 4 = 40 square centimeters. So, the area is 'boxed(40)'.

Q: A fair six-sided die is rolled. What is the probability of rolling a number greater than 4?
A: A six-sided die has numbers 1 through 6. The numbers greater than 4 are 5 and 6, which are 2 out of the 6 possible outcomes. Therefore, the probability of rolling a number greater than 4 is \( \frac{2}{6} = \frac{1}{3} \). Thus, the probability is 'boxed(\frac{1}{3})'.

Q: Find the derivative of the function f(x) = 3x^2 + 2x - 5.
A: To find the derivative of f(x), we apply the power rule to each term. The derivative of 3x^2 is 6x, the derivative of 2x is 2, and the derivative of a constant -5 is 0. Therefore, the derivative of f(x) is f'(x) = 6x + 2. So, the derivative is 'boxed(6x + 2)'.

Figure 8: Prompts and outputs for generating math problems.

Input
Randomly recall five diverse examples of biological problems (remember not to output math problems). For each problem:
- After "Q:", describe the problem
- After "A:", explain the solution and enclose the ultimate answer in 'boxed()'.

Output
Q: In a certain species of plant, the allele for red flowers (R) is dominant over the allele for white flowers (r). If two heterozygous plants are crossed, what is the probability that all offspring will have red flowers?
A: To solve this problem, we use the Punnett square to determine the genotypes of the offspring. The cross of two heterozygous plants Rr x Rr produces the following genotypes: RR, Rr, Rr, and rr. The probability of an offspring having white flowers rr is 1 out of 4, since only one genotype out of the four possible genotypes results in white flowers. Therefore, the probability is 'boxed(\frac{1}{4})'.

Q: A small pond is introduced with 10 fish, and the fish population doubles every month. How many fish will there be in the pond after 6 months?
A: To find the fish population after 6 months, we use the formula for exponential growth: P = P_0 * 2^n, where P is the final population, P_0 is the initial population, and n is the number of time periods (months, in this case). Plugging in the values, we get P = 10 * 2^6. Calculating this gives P = 10 * 64 = 640. Therefore, there will be 'boxed(640)' fish in the pond after 6 months.

Q: What is the primary mechanism by which the human kidney concentrates urine?
A: The human kidney uses the countercurrent multiplier system through the counter-current multiplier system in the loop of Henle. This system allows for the reabsorption of water and solutes from the filtrate back into the bloodstream, which concentrates the urine. The key to this process is the creation of a high osmolarity in the medullary interstitial fluid, which draws water out of the filtrate in the collecting ducts, concentrating the urine. The ultimate answer is the 'boxed(counter-current multiplier system)'.

Q: What is the principle behind the theory of natural selection as proposed by Charles Darwin?
A: The principle behind the theory of natural selection is that individuals within a species show variation in physical characteristics and behaviors. These variations allow the individuals to adapt to their environment and reproduce. Individuals with traits that are better suited to their environment are more likely to survive and reproduce, passing on those advantageous traits to their offspring. Over time, this can lead to changes in the species as a whole. The ultimate answer is the 'boxed(survival and reproduction of the fittest)'.

Q: How do antibiotics kill bacteria?
A: Antibiotics can kill bacteria through several mechanisms, including inhibiting cell wall synthesis, disrupting protein synthesis, impairing nucleic acid synthesis, and interfering with metabolic pathways. The specific mechanism depends on the class of antibiotic. For example, penicillin disrupts the synthesis of the bacterial cell wall, leading to cell lysis and death. Therefore, the ultimate answer is by 'boxed(inhibiting vital processes)' specific to bacteria.

Figure 9: Prompts and outputs for generating biological problems.

Reviewer paper summaries: This paper investigates the intrinsic analogical reasoning capabilities of LLM in the self-
generated contexts. Through extensive experiments, it answered two important questions:
(1) Can LLM consistently benefit more from self-generated relevant examples? (2) what’s the
most important factor for LLM’s performance in analogical reasoning? Both questions are
impactful and lead us to a deep understanding of LLM analogical reasoning capability.
Summary of strengths: • Interesting findings are revealed in this paper, such as LLM cannot always perform
analogical reasoning, and the accuracy of self-generated examples is pivotal for
LLM’s analogical reasoning performance.
• Experiments are well-designed and fully support the conclusions.
Summary of weaknesses: • For Q2 in sec4.2, the author mentioned that the self-generated relevant examples
tend to be more diﬃcult to solve which corresponds to lower accuracy of the
relevant examples. Combined with the conclusion of this paper that the acc of
generated examples is pivotal, my major concern is the fair comparison between
relevant and random reported in Table 2 which directly draws the answer of Q1. If
‘Relevant’ method performs worse due to the lower accuracy in its answers, the
conclusion of Q1 might not be that convincing.
• The presentation of the ICL method on page 6 is not clear and there are no ICL
prompts and outputs examples provided in the appendix.
Comments, suggestions and typos: • The accuracy of self-generated examples (the answer correctness) should be made
clear or defined in the early pages.
• ICL math and ICL bio mentioned in Table 8 should be explained.
Overall assessment scores: 3