Venue: NAACL 2025

# Self-Harmonized Chain of Thought

Ziqi Jin

Wei Lu

StatNLP Research Group

Singapore University of Technology and Design

ziqi_jin@mymail.sutd.edu.sg, luwei@sutd.edu.sg

###### Abstract

Chain-of-thought (CoT) prompting has demonstrated the capacity of large language models to perform complex reasoning through intermediate steps. While effective, current CoT methods face challenges: Zero-shot-CoT can lead to reasoning errors, and Few-shot-CoT requires labor-intensive manual demonstrations. Auto-CoT attempts to address these issues by automatically generating diverse demonstrations, but this diversity can lead to inconsistent reasoning patterns. We propose ECHO (Self-Harmonized Chain of Thought), a novel method that unifies diverse solution paths into a consistent and effective reasoning pattern. ECHO employs an iterative process to refine and harmonize automatically generated demonstrations, mitigating the limitations of existing approaches. Our comprehensive experiments across arithmetic, commonsense, and symbolic reasoning tasks demonstrate that ECHO outperforms Auto-CoT by an average of 2.8%. These findings suggest that ECHO represents a significant step towards more robust and generalizable automated reasoning in large language models.1

Footnote 1: Our code is available: [https://github.com/Xalp/ECHO](https://github.com/Xalp/ECHO).

## 1 Introduction

Large language models (LLMs) have demonstrated remarkable capabilities in various natural language processing tasks. However, their performance on complex reasoning tasks has been a persistent challenge. A recent technique known as chain-of-thought (CoT) prompting Wei et al. (2022) has significantly enhanced LLMs' ability to tackle such tasks by decomposing complex problems into a series of intermediate steps.

CoT methods typically employ two prompting paradigms: Zero-shot-CoT Kojima et al. (2022) and Few-shot-CoT Wei et al. (2022). Zero-shot-CoT uses a universal prompt like "Let's think step by step" to guide models in forming reasoning chains without specific examples. Few-shot-CoT, on the other hand, involves few-shot prompting with human-crafted demonstrations, pairing each question with a detailed reasoning chain.

While Few-shot-CoT has shown promising results, it requires the creation of human-crafted examples for each specific domain, which can be time-consuming and expensive. To address this limitation, Auto-CoT Zhang et al. (2023) was developed to automate the process of creating demonstrations by employing Zero-shot-CoT. However, Auto-CoT faces its own challenges:

* **Misleading by similarity:** Some demonstrations generated by Zero-shot-CoT may contain incorrect reasoning processes or answers, potentially misleading the model when solv

Figure 1: A comparison between ECHO and other CoT baselines. “Zero-CoT” is short for Zero-shot-CoT and “Few-CoT” is short for Few-shot-CoT. The demonstrations generated by Auto-CoT and ECHO will be applied as few-shot examples during inference.



ing similar problems.
* **Ineffective diversity:** To mitigate the misleading effect, Auto-CoT selects diverse demonstrations. However, this approach can lead to demonstrations that are too dissimilar or irrelevant to the actual question, reducing their effectiveness.
* **Inconsistent solution patterns:** Diverse demonstrations might encompass varied solution patterns, including nuanced ones, making them less representative for the model to learn from.

To address these limitations, we propose ECHO (sElf-Harmonized **C**hain of **Thougth**), a novel method that aims to unify diverse rationale patterns into one general pattern. Our approach is inspired by Cognitive Load Theory Sweller (1988), which posits that learning is most effective when the cognitive load on working memory is minimized. By unifying demonstrations, ECHO creates a more coherent set of examples, potentially reducing the cognitive load, thus facilitating more effective reasoning patterns that can be easily followed.

ECHO consists of three main steps: (1) Cluster a given dataset and select representative questions. (2) Generate initial rationales using Zero-shot-CoT. (3) Employ a dynamic prompting mechanism to iteratively improve demonstrations using each other as in-context examples.

This iterative process not only refines the quality of individual demonstrations but also promotes consistency across the entire set, potentially reducing the cognitive load required to process and apply the demonstrated reasoning patterns.

Our experiments across three different reasoning domains demonstrate that ECHO outperforms other baselines by 2.8% overall. We also conduct comprehensive ablation studies to understand the benefits of unifying diversity in improving performance. These results support the hypothesis that a more unified set of demonstrations, as suggested by Cognitive Load Theory, can indeed lead to improved reasoning capabilities in LLMs.

The main contributions of this work are:

* A novel approach (ECHO) that automatically improves the quality of demonstrations in the CoT process by unifying diversity.
* An iterative unifying prompting framework that is effective across various tasks by reducing demonstration variety.
* Extensive experiments showing competitive results on arithmetic, commonsense, and symbolic reasoning domains through diversity reduction, supporting the efficacy of our cognitively-informed approach.

## 2 Related Work

Chain-of-Thought (CoT) PromptingCoT prompting has emerged as a powerful, gradient-free method for enhancing the reasoning capabilities of Large Language Models (LLMs). Wei et al. (2022) introduced Few-shot-CoT, which utilizes manually crafted demonstrations to guide the model's reasoning process. Kojima et al. (2022) then proposed Zero-shot-CoT, extending the method to zero-shot scenarios.

Recent studies have focused on creating more complex demonstrations or employing ensemble-like strategies. For example, some prompting methods Zhou et al. (2022); Wang et al. (2023); Yao et al. (2023) adopt a problem decomposition approach, breaking down complex problems into simpler subproblems. Self-consistency and other CoT methods Wang et al. (2022); Asai et al. (2023); Sun et al. (submitted) involve reasoning from multiple paths or iteratively. Moreover, some prompting methods can generate executable programs to aid in the computation process within the arithmetic domain Gao et al. (2022); Chen et al. (2022); Pi et al. (2022).

Automated Prompt GenerationAlthough designing increasingly complex prompts improves CoT effectiveness, it also increases the human effort involved. Auto-CoT Zhang et al. (2023) automates the process by clustering a dataset and selecting a representative question from each cluster, followed by using Zero-shot-CoT to generate rationales for these questions.

A key consideration in Auto-CoT Zhang et al. (2023) is the avoidance of using questions in the same cluster as demonstrations. The rationale behind this choice is rooted in the observation that if demonstrations are too similar to the target question, there is a higher risk that the model might replicate its mistakes, since Zero-shot-CoT Kojima et al. (2022) does not guarantee correctness in its responses (i.e., the demonstrations might be wrong). However, the use of diverse demonstrations might include irrelevant demonstrations or non-representative solution patterns 



## 3 Self-Harmonized Chain of Thought

Our method, ECHO, involves three main steps as shown in Figure 2:

1. **Question clustering:** Partition questions of a given dataset into clusters based on their similarity.
2. **Demonstration sampling:** Select a representative question from each cluster and generate its reasoning chain using Zero-shot-CoT.
3. **Demonstration unification:** Iteratively update rationales to build a coherent pattern across all demonstrations.

These steps are designed to address the limitations of previous methods by creating a unified, representative set of demonstrations. The first two steps are similar to Auto-CoT (Zhang et al., 2023) with minor differences, while the third step is unique to our approach.

### Question Clustering

We begin by applying clustering to a set of questions \(Q\). Each question is transformed into a fixed-size vector representation using Sentence-BERT (Reimers and Gurevych, 2019), which provides semantically meaningful embeddings. We then employ a \(k\)-means clustering model to categorize these vector representations into \(k\) distinct clusters. Within each cluster \(i\), the questions are ordered in a list \({\bf q}^{(i)}=[q_{1}^{(i)},q_{2}^{(i)},\ldots]\), based on their distance to the centroid of the cluster.

Unlike Auto-CoT (Zhang et al., 2023), where the number of clusters equals the number of output demonstrations, our approach allows for a greater number of clusters. This expansion enables the inclusion of a wider range of demonstrations in the unification process, facilitating effective learning from more diverse patterns and ensuring that the final pattern can be applied to a wider range of demonstrations.

### Demonstration Sampling

For each cluster \(i\), we sample one demonstration \(d^{(i)}\) by evaluating the questions in \({\bf q}^{(i)}=[q_{1}^{(i)},q_{2}^{(i)},\ldots]\) against predefined selection criteria. For each question \(q_{j}^{(i)}\), we generate its rationale using the prompt 'Let's think step by step' (Kojima et al., 2022), following the methodology used in Auto-CoT (Zhang et al., 2023).

Our selection criteria follow (Zhang et al., 2023), which include two key constraints: (1) The question length should not exceed 60 tokens. (2) The corresponding rationale \(r^{(i)}\) should be limited to no more than 5 steps. These constraints are chosen

Figure 2: Overview of our ECHO method. In the demonstration unification process, ECHO iteratively re-generates the rationale of one demonstration with other demonstrations as in-context examples.



to ensure manageable and focused demonstrations. Steps are demarcated with '\(\backslash\)n', and we count the steps by tallying the number of these separators.

### Demonstration Unification

The core of our method lies in the demonstration unification step, where we regenerate the reasoning chain for the sampled questions to form a convergent pattern.

In each iteration, each demonstration \(d^{(k)}=q^{(k)}\circ r_{0}^{(k)}\) in the set \(\mathcal{D}\) is updated once. The process is as follows:

1. Randomly select a demonstration \(d^{(i)}\).
2. Use the remaining shuffled demonstrations \(\mathcal{D}\setminus d^{(i)}\) as in-context examples.
3. Regenerate the rationale \(r^{(i)}\) for the selected demonstration.
4. Replace the previous rationale with the newly generated one.

This process is repeated for each demonstration in the set. Through successive iterations, this leads to convergence, resulting in a uniform pattern across all rationales. The random selection and shuffling of demonstrations ensure that each rationale learns from a diverse set of examples, promoting robustness in the final pattern.

In our method, the number of clusters \(k\) typically exceeds the number of output demonstrations \(m\). This oversampling approach ensures that the final unified pattern is more robust, generalizable, and suitable for a wider array of samples. It's akin to information compression, where insights from a larger set of demonstrations are distilled into the final set of \(m\) demonstrations, enhancing both the representativeness and adaptability of the output.

Note that although the demonstration unification process introduces more demonstrations, we apply the same number of demonstrations during the inference stage as used in prior approaches to ensure a fair comparison.

By unifying diverse rationale patterns into a coherent set of demonstrations, ECHO addresses the limitations of previous methods, particularly the potential for irrelevant or non-representative demonstrations in diverse sets. This approach aims to create a more effective and generalizable Chain-of-Thought prompting method. We demonstrate detailed implementation of ECHO in Appendix B.

## 4 Why does it Work?

In this section, we offer some insight into why the proposed approach, ECHO, can be effective by presenting a mathematical framework for understanding its iterative refinement process.

**Research Question 1: Why will regenerate rationale using others as demonstrations result in convergence?**

Few-shot-CoT demonstrates that models naturally follow the patterns shown in few-shot demonstrations, providing reasoning steps before producing the final answer. Subsequent research, such as PAL Gao et al. (2022), extends this concept by using code rationales to generate code for solving math problems. These studies collectively show that models not only follow the general question-answering structure but also inherit the patterns from these demonstrations.

**Research Question 2: Why will regenerate rationale using others as demonstrations yield better rationale?**

Consider a set of questions \(\mathcal{Q}=\{q^{(1)},q^{(2)},\ldots,q^{(m)}\}\). Assume that we can use Zero-Shot-CoT to arrive at the rationales (with answers): \(\mathcal{R}=\{r^{(1)},r^{(2)},\ldots,r^{(m)}\}\).

Auto-CoT performs the following: First, it constructs demonstrations \(\mathcal{D}=\{d^{(1)},d^{(2)},\ldots,d^{(m)}\}\), where \(d^{(k)}=q^{(k)}\circ r^{(k)}\). During the inference stage, such demonstrations are then used as few-shot examples for generating "refined" rationales for the \(i\)-th instance, based on \(\mathcal{D}\backslash d^{(i)}\), which we denote as \(r_{0}^{(i)}\). Completing this refinement process leads to \(\mathcal{R}_{0}=\{r_{0}^{(1)},r_{0}^{(2)},\ldots,r_{0}^{(m)}\}\).

Empirically, Auto-CoT was shown to yield better results than Zero-Shot-CoT. Mathematically, this leads to the following hypothesis:

\[p(\mathcal{Q},\mathcal{R}_{0})\geq p(\mathcal{Q},\mathcal{R}) \tag{1}\]

where \(p(\mathcal{Q},\mathcal{R})\) returns the probability for the set of rationales \(\mathcal{R}\) to be assessed as correct for the set of questions \(\mathcal{Q}\). This probability can be interpreted as a measure of the overall quality and accuracy of the rationales.

Equation 1 shows that the set of refined rationales is likely to be more accurate than the original set of rationales, which were generated without any demonstrations. This improvement can be attributed to the beneficial effect of using in-context examples during the refinement process.

In practice, we can keep refining the set of rationales by reconstructing them. Specifically, we can first construct \(\mathcal{D}_{0}=\{d_{0}^{(1)},d_{0}^{(2)},\ldots,d_{0}^{(m)}\}\), where \(d_{0}^{(k)}=q^{(k)}\circ r_{0}^{(k)}\). Repeating the above process, we arrive at \(\mathcal{R}_{1}=\{r_{1}^{(1)},r_{1}^{(2)},\ldots,r_{1}^{(m)}\}\).

 

If the above hypothesis for Auto-CoT is true, it is not unreasonable to state the following hypothesis as its extension:

\[p({\cal Q},{\cal R}_{1})\geq p({\cal Q},{\cal R}_{0}) \tag{2}\]

We can repeat the above process \(T\) times to arrive at a chain of inequalities:

\[p({\cal Q},{\cal R}_{T})\geq\cdots\geq p({\cal Q},{\cal R}_{1})\geq p({\cal Q}, {\cal R}_{t}) \tag{3}\]

This chain of inequalities argues why our proposed approach works empirically. Each iteration of refinement is likely to produce a set of rationales that are at least as good as, if not better than, the previous set.

It's important to note that while the above process updates the rationales in a batch mode, in our ECHO method, we adopted an online approach. In this approach, we utilize the recently updated rationales for the next instance within the same iteration. This online approach allows for more immediate incorporation of improvements, potentially leading to faster convergence to high-quality rationales. However, this behaviour may also negatively impact model performance, as some patterns are specific to particular questions and cannot be generalized effectively to others (we demonstrate performance on combined dataset in Section 6.7).

## 5 Experimental Setup

### Tasks and Datasets

Following prior work on CoT, we evaluate our method on 10 reasoning datasets, including: (1) 6 arithmetic datasets: SingleEq (Koncel-Kedziorski et al., 2015), AddSub (Hosseini et al., 2014), MultiArith (Roy and Roth, 2015), GSM8K (Cobbe et al., 2021), AQUA-RAT (Ling et al., 2017), and SVAMP (Patel et al., 2021); (2) 2 commonsense reasoning datasets: CommonsenseQA (Talmor et al., 2019) and StrategyQA (Geva et al., 2021); (3) 2 symbolic reasoning datasets: Last Letter and Coin Flip (Wei et al., 2022); Table 8 contains the statistics for all benchmarks.

### Models

Following Wei et al. (2022), we used the OpenAI API for our experiments. We chose GPT-3.5-Turbo-0301 in our main experiments because it is easy to access and more affordable. We also tested on Mixtrtal-8x7B-Instruct in the ablation study to validate the generalizability of our method. To ensure the reproducibility of our experiments, we fixed the temperature parameter at 0.

### Configuration

For our primary experiment, we set the following parameters:

* Iteration count: \(T=1\)
* Number of demonstrations for unification process: \(k=m\) (equal to the number of output demonstrations)2 Footnote 2: We followed Wei et al. (2022) for the number of demonstrations: 4 for AQUA, Coin Flip, and Last Letters; 6 for StrategyQA; 7 for CSQA; 8 for all other datasets

## 6 Results

### Main Experiment Analysis

The data presented in Table 1 offers a detailed comparison of various methods across three categories: Arithmetic, Commonsense, and Symbolic. In these evaluations, ECHO consistently outperforms Auto-CoT (Zhang et al., 2023) in each domain, matching Few-Shot-CoT's overall performance. Specifically, ECHO achieves an overall performance improvement of 1.7% over Auto-CoT. The overall performance of ECHO and Few-Shot-CoT stays the same, demonstrating ECHO's ability to match human-crafted demonstrations without manual effort.

Notably, Auto-CoT falls short in performance compared to Few-Shot-CoT across all domains, with a particularly significant gap of 1.7% in Commonsense reasoning. This suggests that Auto-CoT may not fully replace human effort in Few-Shot-CoT, highlighting the value of ECHO's approach.

\begin{table}
\begin{tabular}{l c c c c c c c c c c c c c c} \hline \hline Method & \multicolumn{6}{c}{_Arithmetic_} & \multicolumn{6}{c}{_Commonsense_} & \multicolumn{6}{c}{_Symbolic_} \\ \cline{2-13}  & MultiArith & GSM8K & SingleEq & AddSub & AQua & SVAMP & avg. & CSOA & Strategy & avg. & Letter & coin & avg. \\ \hline Zero-Shot & 740 & 20.8 & 87.2 & 86.8 & 26.8 & 72.3 & 87.3 & 71.4 & 55.6 & 63.5 & 1.2 & 48.4 & 24.8 & 54.5 \\ Zero-Shot-CoT & 84.2 & 74.5 & 88.0 & 84.3 & 54.3 & 78.5 & 77.3 & 69.6 & 53.1 & 61.4 & 69.6 & 81.6 & 63.1 & 71.3 \\ \hline Few-Shot & 80.0 & 20.8 & 87.0 & 85.6 & 30.7 & 76.0 & 63.4 & **78.4** & 49.5 & 64.0 & 6.2 & 57.2 & 31.7 & 57.1 \\ Few-Shot-CoT & **98.3** & 77.9 & 92.5 & 85.6 & 56.7 & 81.5 & 82.1 & 76.1 & 63.2 & 69.7 & 81.6 & 95.4 & 88.5 & 80.9 \\ \hline Auto-CoT & 96.0 & 76.2 & 92.1 & 85.8 & 52.4 & 82.6 & 80.8 & 74.9 & 56.4 & 65.7 & 76.2 & 99.4 & 87.8 & 79.2 \\ ECHO (\(k=m,T=1\)) & 98.0 & 78.4 & 91.3 & **87.3** & 52.8 & 81.1 & 81.5 & 77.2 & 59.9 & 68.6 & **83.6** & **94.1** & **91.5** & 80.9 \\ \hline ECHO (\(k=max,T=1\)) & 97.7 & **78.5** & 89.8 & 87.1 & 55.5 & 84.2 & 82.1 & 73.8 & 58.0 & 65.9 & 81.3 & **99.8** & 90.6 & 80.8 \\ ECHO (\(k=max,T=4\)) & 97.2 & 76.9 & **93.1** & 86.8 & **89.1** & **85.4** & **83.1** & 77.5 & **63.4** & **70.5** & 81.0 & 99.6 & 90.3 & **82.0** \\ \hline \hline \end{tabular}
\end{table}
Table 1: Accuracy on ten datasets from three categories of reasoning tasks.

 

ECHO, on the other hand, aligns closely with Few-Shot-CoT's overall performance, indicating its potential as a significant advancement towards a powerful reasoning framework without manual efforts.

Interestingly, ECHO outperformed Few-Shot-CoT's average score in symbolic tasks by 3.0%, but slightly lags in arithmetic and common sense tasks by 0.6% and 1.1% respectively. We observe that symbolic reasoning tasks tend to have a more uniform pattern, making a single iteration sufficient for effective reasoning, whereas arithmetic and common sense reasoning tasks are more diverse in question contexts and answer patterns. We hypothesize that this lag in performance is due to the insufficient number of rationales and iterations for the self-harmonization process in these more diverse domains. This suggests the potential for further improving ECHO's performance, particularly in tasks with greater pattern variety.

### A comparison between ECHO-generated and Manual Prompts

To understand the difference between ECHO and manually written prompts, we applied both to our method. Our experiments employ the average score from six arithmetic reasoning benchmarks.

The rationales of ECHO are initialized with Zero-Shot-CoT (Kojima et al., 2022), while manual prompts are crafted by humans. We initialized our method with both automatically generated and human-written prompts to check if ECHO can be directly applied to existing manual prompts. The outcomes, depicted in Figure 3, reveal three key findings:

1. Initially, manual prompts surpassed those generated via Zero-Shot-CoT.
2. After one iteration of ECHO, the performance difference diminishes significantly.
3. Following three iterations, automatically generated prompts exceed manual ones, achieving peak performance.

Table 2 presents three examples from both manually written and ECHO-generated prompts. Manually written prompts start coherent and maintain uniformity across iterations. In contrast, generated rationales begin highly varied but unify towards a singular pattern through ECHO iterations. This suggests that generated rationales provide a more

\begin{table}
\begin{tabular}{l l} \hline \hline
**General Rationale (3 iterations)** & \\ \hline First, we need to add the number of nickels Sam had originally to the number of nickels his dad gave him. 24 (original nickels) + 39 (new nickelés) = 63 So Sam now has 83 nickels. \\ \hline First, we need to add the weight of peanuts to the weight of raasins. 0.1 \\ (peanuts) + 0.4 (raisins) = 0.5 So Kelly bought a total of 0.5 pounds of snacks. \\ \hline First, we need to add the number of pernelsi Joan placed in the daware to the original number of pencils in the dawner. 33 (original pencils) + 27 (new pencils) = 60 So there are now 60 pencils in the drawer in total. \\ \hline \hline \end{tabular}
\end{table}
Table 2: A comparison of three demonstrations between manual prompts and generated prompts. Manual prompts exhibit a consistent pattern starting from iteration 0, whereas the generated rationales lack consistency initially but achieve consistency after three iterations of ECHO.

Figure 3: Performance for ECHO initialized by manual prompts and Zero-Shot-CoT generated prompts with 0, 1 and 3 iterations. We conclude the generated prompts perform better after the unification process, while the manual ones perform better when applied directly.

\begin{table}
\begin{tabular}{l l l l} \hline \hline \multirow{2}{*}{Type} & \multicolumn{4}{c}{Divergence} \\ \cline{2-4}  & RoBERTa-Large & T5-XL & T5-XXL \\ \hline Manual & 0.758 & 0.245 & 0.276 \\ Generated & 0.660 & 0.201 & 0.244 \\ \hline \(\Delta\) & 0.098 \(\downarrow\) & 0.044 \(\downarrow\) & 0.032 \(\downarrow\) \\ \(\Delta\) \% & 12.9\% \(\downarrow\) & 18.0\% \(\downarrow\) & 11.6\% \(\downarrow\) \\ \hline \hline \end{tabular}
\end{table}
Table 3: The averaged divergence for best-performing manual and generated rationales, using three different encoders. We observe a lower divergence for generated rationales, which may justify its superior performance.

 

diverse initial set of patterns, aiding in identifying the most effective pattern.

To quantify the convergence, we compared the average divergence between manual rationales (0 iterations) and generated rationales (3 iterations). Using "_roberta-large-nil-stsb-mean-tokens_", "_sentence-15-large_", and "_sentence-15-xxl_" as encoders, we computed pairwise cosine similarities and defined average divergence as 1 - average similarity. Table 3 shows that the average divergence is significantly reduced in the generated demonstrations, confirming ECHO's ability to unify diverse initial patterns into a coherent, effective set of prompts.

### Effect of Hyperparameters

Building on our initial insights, we explored how the initial diversity impacts our method's performance. We tested ECHO in a configuration where the number of demonstrations considered, \(k\), exceeds the original count, \(m\), in the self-harmonized process. This adjustment does not alter the number of output demonstrations used during inference.

The rationale behind this enhancement is to increase diversity by starting with a more varied set of patterns, potentially increasing ECHO's adaptability across different reasoning tasks. To maximize diversity, we chose the largest \(k\) within the model's token limit.

Results of this adjustment, shown in Table 1, reveal some improvements in the arithmetic domain but a noticeable decline in overall performance. Based on observations from Figure 3, we hypothesize that a single iteration may be insufficient when dealing with a larger number of demonstrations, suggesting the need for more iterations.

To determine the optimal number of iterations, we tested ECHO across various iteration counts (Figure 4). In all cases, our method's average overall performance remained superior to Auto-CoT [23]. However, we observed a trend of overfitting with excessive iterations, evident from the model's performance peaking at an optimal iteration count before gradually declining.

A case study (Appendix A) provides further insights:

* After a single iteration, the model adopts a consistent rationale structure, with phrases like "Sure, let's break this down" or "First, we need to find" becoming standard openings.
* After 32 iterations, the model attempts to condense multi-step reasoning into a single step, resulting in complex equations with meticulous unit notation, suggesting an overemphasis on uniformity at the cost of conciseness and clarity.

Analysis of Figure 4 indicates that an iteration count of \(T=4\) offers an optimal balance, with peak overall average performance and generalizability across various domains.

### Does ECHO Suffer from "Misleading by Similarity"?

The Auto-CoT approach [23] acknowledges a vulnerability to "misleading by similarity." This occurs when incorrect demonstrations from Zero-shot-CoT [69] closely resemble the target problem, causing the LLM to replicate mistakes.

To investigate this limitation, we conducted experiments on mathematical reasoning tasks under three conditions:

1. **Diverse:** Demonstrations chosen from the centroid of each cluster, ensuring diverse representation.
2. **Random:** Randomly selected demonstrations from the dataset.

Figure 4: Performance of ECHO in different domains with iteration increases exponentially. We found that \(T=4\) results in the best overall performance.

Figure 5: Performance of ECHO and Auto-CoT under 3 different settings of demonstrations: one from each cluster, randomly sampled and all from same cluster. ECHO with more iterations gain more performance from diverse demonstrations.

 

3. **Uniform:** Demonstrations strictly from the same cluster as the target question, promoting similarity.

We set the iteration count \(T=1,2\) and the number of demonstrations \(m=8\) for all tests.

Results, illustrated in Figure 5, reveal key findings:

* Unlike Auto-CoT, ECHO improved as demonstrations became more uniform, suggesting that closer relationships between demonstrations and questions enhance utility, outweighing the risk of misleading information.
* ECHO with one iteration performed best in the random setting, indicating effective learning from all available demonstrations, regardless of direct relevance.
* ECHO with two iterations illustrated that: (1) Overly diverse demonstrations may require more than one iteration to establish a uniform solution pattern. (2) Excessively similar demonstrations risk overfitting after just one iteration.

These findings suggest that diverse demonstrations, even if not directly related, can help build a robust, generalizable problem-solving framework. We conclude that a strategic mix of demonstrations, balancing relatedness and diversity, could offer generalization without sacrificing effectiveness.

Future work could explore a hybrid selection strategy that dynamically adjusts the demonstration set based on dataset properties, potentially optimizing the balance between diversity and similarity.

### Results with Mixtral-8x7B

In addition to GPT-3.5-Turbo-0301, we tested our method on 10 benchmarks aligned with the main experiments using Mixtral-8x7B. The results, reported in Table 4, show that our approach outperforms Auto-CoT by an average margin of +2.3% with Mixtral-8x7B. This consistency with our existing findings demonstrates ECHO's generalizability to other models.

However, we note that the overall performance metrics are lower than those achieved with GPT-3.5-Turbo-0301, indicating the significant impact of the underlying model used. Notably, our method failed to outperform Few-shot-CoT on average when applied to Mixtral-8x7B. We believe there are two main reasons for this outcome.

First, the quality of the generated rationales may differ depending on the model's capability, while the quality of manual prompts remains constant. In this case, the rationales generated by Miktral-8x7B may not be as sophisticated as those produced by GPT-3.5-Turbo-0301.

Second, we observed a significant performance drop in the "coin Flip" dataset. Consider an example question: "A coin is heads up. Irving flips the coin. Hans flips the coin. Moses does not flip the coin. Nicole does not flip the coin. Is the coin still heads up?" We found that the rationales generated by Auto-CoT or ECHO explicitly track the coin's state when compared with GPT-3.5-Turbo-0301. In contrast, the human-written prompts applied by Few-shot-CoT employ a shortcut solution by counting overall flips to determine the outcome. As the coin always starts heads up, if the number of total flips is even, the coin will remain heads up. We observed that Mixtral-8x7B is less adept at tracking coin states, which may explain its inferior performance on this particular task.

These observations highlight the potential limitations of automated rationale generation across different language models.

\begin{table}
\begin{tabular}{l c r r r r r r r r r r r r r} \hline \hline Method & \multicolumn{6}{c}{_Arithmetic_} & \multicolumn{6}{c}{_Communonsense_} & \multicolumn{3}{c}{_Symbolic_} & Overall \\ \cline{2-19}  & Multi/Arith & GSMSK & Singlefa & AddSub & AQua & SVAMP & avg. & CSQA & Strategy & avg. & Letter & Coin & avg. \\ \hline Few-Shot-CoT & 94.0 & 65.3 & 87.4 & 83.5 & 47.2 & 78.2 & 79.9 & 74.1 & **61.5** & **67.8** & **59.4** & **97.0** & **78.2** & **74.8** \\ Auto-CoT & 92.1 & 71.6 & **88.8** & 84.1 & 45.7 & 82.2 & 77.4 & 71.8 & 55.9 & 63.9 & 52.2 & 71.4 & 62.3 & 71.7 \\ ECHO (\(h=\max T=4\)) & **96.8** & **72.4** & 88.0 & **85.3** & **55.5** & **82.8** & **80.2** & **74.2** & 54.6 & 64.4 & 56.8 & 73.8 & 65.3 & 74.0 \\ \hline \hline \end{tabular}
\end{table}
Table 4: Accuracy on ten datasets for Mixtral-8x7B.

\begin{table}
\begin{tabular}{l l l l l l l l l l l l l l l l} \hline \hline Method & \multicolumn{6}{c}{_Arithmetic_} & \multicolumn{6}{c}{_Communonsense_} & \multicolumn{3}{c}{_Symbolic_} & Overall \\ \cline{2-19}  & Multi/Arith & GSMSK & Singlefa & AddSub & AQua & SWAMP & avg. & CSQA & Strategy & avg. & Letter & Coin & avg. \\ \hline Few-Shot-CoT & 98.3 & 77.9 & 92.5 & 85.6 & 56.7 & 81.5 & 82.1 & 76.1 & 63.2 & 69.7 & 81.6 & 95.4 & 88.5 & 80.9 \\ -half & 97.5 & 75.1 & 91.5 & 85.6 & 57.9 & 78.3 & 81.0 & 70.8 & 58.8 & 64.8 & 81.8 & 98.6 & 90.2 & 79.6 \\ \hline Auto-CoT & 96.0 & 76.2 & 92.1 & 85.8 & 52.4 & 82.6 & 80.8 & 74.9 & 56.4 & 65.7 & 76.2 & 99.4 & 87.8 & 79.2 \\ -half & 97.0 & 77.2 & 92.3 & 85.6 & 54.7 & 80.0 & 81.1 & 74.2 & 54.8 & 64.5 & 77.8 & 99.6 & 88.7 & 79.3 \\ \hline ECHO (\(k=\max T=4\)) & 97.2 & 76.9 & 93.1 & 86.8 & 59.1 & 85.4 & 83.1 & 77.5 & 63.4 & 70.5 & 81.0 & 99.6 & 90.3 & 82.0 \\ -half & 98.3 & 78.1 & 92.3 & 87.3 & 85.7 & 84.9 & 83.3 & 72.5 & 59.8 & 66.15 & 82.4 & 97.6 & 90.0 & 81.2 \\ \hline \hline \end{tabular}
\end{table}
Table 5: Performance with half number of demonstrations.

 



### Results with 50% Demonstrations

To assess the robustness of our method, we evaluated the impact of reducing the number of demonstrations by half. The results reveal interesting patterns across different approaches.

When we decrease the number of demonstrations by half, the overall performance of ECHO decreased by only 0.8%, compared to a 1.3% decline for Few-Shot-CoT. This smaller performance drop suggests that ECHO's unified rationales are more robust to demonstration reduction. We infer that the consistency achieved through ECHO's unification process allows each demonstration to effectively capture patterns from the others, minimizing the impact of reducing their number. In essence, the remaining demonstrations retain information from those that are dropped, contributing to this enhanced robustness.

Interestingly, Auto-CoT showed a slight performance improvement of 0.1% with the reduced number of demonstrations. This counterintuitive result suggests that diversity in demonstrations can sometimes hinder performance. We hypothesize that this improvement stems from the reduced diversity of demonstrations, which may lead to more consistent reasoning patterns.

These findings highlight an important insight: increasing the number of demonstrations (or "shots") does not necessarily lead to better performance. The quality and consistency of demonstrations, rather than just their quantity, play a crucial role in the effectiveness of reasoning methods.

### A Study on Combined Dataset

Similar to Zhang et al. (2023), our work requires access to the entire dataset to construct demonstrations. We assume that each dataset exhibits a certain level of internal similarity. For example, a math dataset should consist exclusively of math questions, while a yes-or-no dataset should contain only yes-or-no questions. However, some datasets are more diverse in nature. In this section, we aim to evaluate our method in a more complex scenario: we randomly select 500 samples each from both GSM8K and StrategyQA. GSM8K focuses on arithmetic tasks, whereas StrategyQA involves yes-or-no commonsense reasoning tasks. We apply our method to this combined dataset, and the results are presented in Table 6.

Among the 8 demonstrations that are automatically selected after clustering, 3 are taken from GSM8K and 5 from StrategyQA. We observed a performance drop on both datasets when using our method. This outcome highlights the limitations of our approach, as it attempts to identify a uniform solution pattern. Clearly, no single pattern can be effectively applied across two fundamentally different domains.

Here's a polished version of your section, including the transposed and one-liner table as requested:

### Results with GPT-4o

We experimented with our method using the up-to-date GPT-4o model. Due to cost constraints, we conducted experiments only on GSM8K. The results are summarized in Table 7.

These results demonstrate that ECHO outperforms both Few-shot-CoT and Auto-CoT under this setup. However, we also observe that Zero-shot-CoT achieves comparable performance to ECHO in this scenario. This finding suggests that a superior model like GPT-4o can achieve strong results without demonstrations. It already has the capability to answer arithmetic questions in a well-structured format, making additional demonstrations potentially redundant.

## 7 Conclusion

In this work, we propose a novel method called ECHO for improving chain-of-thought prompting in large language models. We have shown that our method results in consistientimprovements in three domains, confirming the feasibility and significance of adopting the self-harmonization mechanism in the CoT prompting process. We also carried out extensive experiments and case studies to understand the behavior of the proposed approach. We hope that these insights can help to improve future automatic reasoning frameworks.

\begin{table}
\begin{tabular}{l c c c} \hline \multirow{2}{*}{Method} & \multicolumn{3}{c}{Accuracy} \\ \cline{2-4}  & GSM8K & StrategyQA & Average \\ \hline ECHO (original) & 78.1 & 59.7 & 68.9 \\ ECHO (combined) & 74.2 & 58.0 & 66.1 \\ \hline \end{tabular}
\end{table}
Table 6: A Comparison of the performances between ECHO using demonstrations from original and combined datasets.

\begin{table}
\begin{tabular}{l c c c c} \hline \hline  & Few-shot-CoT & Auto-CoT & ECHO (ours-Ti) & Zero-shot-CoT \\ \hline Performance & 90.9 & 91.9 & 92.4 & 92.3 \\ \hline \hline \end{tabular}
\end{table}
Table 7: Performance comparison on GSM8K using GPT-4o, with methods as column headers.





## Limitations

Our study has identified several limitations within the ECHO method:

**(1)** The method incurs a higher inference cost due to the necessity of an additional unification process for demonstrations. This process requires extra computational resources as it involves generating multiple inferences to consolidate the demonstrations into a coherent pattern. For a benchmark of \(n\) samples, while the other method requires inference for \(n\) times, our method requires \(n+T\cdot k\) times (\(T\) is the number of iterations and \(k\) is the number of samples used). For example, for GSM8K, the ECHO with 4 iterations requires 5.8% more number of inferences.

**(2)** The method is prone to overfitting, which can lead to a decrease in generalizability. Although we have introduced an equation to estimate the optimal number of iterations, this equation may not hold universally across different domains or datasets.

**(3)** The method assumes a certain level of similarity within the data from which it learns, which might not always be the case. In scenarios where the data are highly unrelated or where the relationships between data points are complex, the unification process may struggle to recognize a representative pattern. Future work may explore adaptive mechanisms that can recognize and adapt to the diversity of the data, ensuring that the unification process remains effective in various types of problem.

**(4)** Similar to Zhang et al. (2023), our method can be applied to a dataset containing multiple questions rather than a single question.

## Acknowledgement

We would like to thank the anonymous reviewers, our meta-reviewer, and senior area chairs for their constructive comments and support of our work. This research/project is supported by the National Research Foundation Singapore under the AI Singapore Programme (AISG Award No: AISG2-TC-2023-013), and the Ministry of Education, Singapore, under its Academic Research Fund (AcRF) Tier 2 Programme (MOE AcRF Tier 2 Award No: MOET2EP20122- 0011) and its Tier 3 Programme (The Award No.: MOET32020-0004). Any opinions, findings and conclusions or recommendations expressed in this material are those of the authors and do not reflect the views of the National Research Foundation or Ministry of Education, Singapore.

## References

* Asai et al. (2023) Akari Asai, Zeqiu Wu, Yizhong Wang, Avirup Sil, and Hannaneh Hajishirzi. 2023. Self-rag: Learning to retrieve, generate, and critique through self-reflection.
* Chen et al. (2022) Wenhu Chen, Xueguang Ma, Xinyi Wang, and William W Cohen. 2022. Program of thoughts prompting: Disentangling computation from reasoning for numerical reasoning tasks. _arXiv preprint arXiv:2211.12588_.
* Cobbe et al. (2021) Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reitichiro Nakano, Christopher Hesse, and John Schulman. 2021. Training verifiers to solve math word problems. _arXiv preprint arXiv:2110.14168_.
* Gao et al. (2022) Luyu Gao, Aman Madaan, Shuyan Zhou, Uri Alon, Pengfei Liu, Yiming Yang, Jamie Callan, and Graham Neubig. 2022. Pal: Program-aided language models. In _Proceedings of ICLR_.
* Geva et al. (2021) Mor Geva, Daniel Khashabi, Elad Segal, Tushar Khot, Dan Roth, and Jonathan Berant. 2021. Did aristotle use a laptop? a question answering benchmark with implicit reasoning strategies. _Transactions of the Association for Computational Linguistics_, 9:346-361.
* Hosseini et al. (2014) Mohammad Javad Hosseini, Hannaneh Hajishirzi, Oren Etzioni, and Nate Kushman. 2014. Learning to solve arithmetic word problems with verb categorization. In _Proceedings of EMNLP_.
* Kojima et al. (2022) Takeshi Kojima, Shixiang Shane Gu, Michel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022. Large language models are zero-shot reasoners. In _Proceedings of NeuIPS_.
* Koncel-Kedziorski et al. (2015) Rik Koncel-Kedziorski, Hannaneh Hajishirzi, Ashish Sabharwal, Oren Etzioni, and Siena Dumas Ang. 2015. Parsing algebraic word problems into equations. _Transactions of the Association for Computational Linguistics_, 3:585-597.
* Ling et al. (2017) Wang Ling, Dani Yogatama, Chris Dyer, and Phil Blunsom. 2017. Program induction by rationale generation: Learning to solve and explain algebraic word problems. In _Proceedings of ACL_.
* Patel et al. (2021) Arkil Patel, Satwik Bhattacharya, and Navin Goyal. 2021. Are NLP models really able to solve simple math word problems? In _Proceedings of NAACL_.
* Pi et al. (2022) Xinyu Pi, Qian Liu, Bei Chen, Morteza Ziyadi, Zeqi Lin, Qiang Fu, Yan Gao, Jian-Guang Lou, and Weizhu Chen. 2022. Reasoning like program executors. In _Proceedings of the EMNLP_

Nils Reimers and Iryna Gurevych. 2019. Sentence-BERT: Sentence embeddings using Siamese BERT-networks. In _Proceedings of the 2019 EMNLP-IJCNLP_, pages 3982-3992.
* Roy and Roth (2015) Subhro Roy and Dan Roth. 2015. Solving general arithmetic word problems. In _Proceedings of EMNLP_.
* Sun et al. (2017) Jiashuo Sun, Yi Luo, Yeyun Gong, Chen Lin, Yelong Shen, Jian Guo, and Nan Duan. Enhancing chain-of-thoughts prompting with iterative bootstrapping in large language models. In _Findings of NAACL 2024_.
* Sweller (1988) John Sweller. 1988. Cognitive load during problem solving: Effects on learning. _Cognitive Science_, 12(2):257-285.
* Talmor et al. (2019) Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. 2019. CommonsenseQA: A question answering challenge targeting commonsense knowledge. In _Proceedings of NAACL_.
* Wang et al. (2023) Lei Wang, Wanyu Xu, Yihuai Lan, Zhiqiang Hu, Yunshi Lan, Roy Ka-Wei Lee, and Ee-Peng Lim. 2023. Plan-and-solve prompting: Improving zero-shot chain-of-thought reasoning by large language models. _arXiv preprint arXiv:2305.04091_.
* Wang et al. (2022) Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Akakanksha Chowdhery, and Denny Zhou. 2022. Self-consistency improves chain of thought reasoning in language models. In _Proceedings of ICLR_.
* Wei et al. (2022) Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. 2022. Chain of thought prompting elicits reasoning in large language models. In _Proceedings of NeuIPS_.
* Yao et al. (2023) Shunya Yao, Qian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L. Griffiths, Yuan Cao, and Karthik Narasimhan. 2023. Tree of thoughts: Deliberate problem solving with large language models.
* Zhang et al. (2023) Zhuosheng Zhang, Aston Zhang, Mu Li, and Alex Smola. 2023. Automatic chain of thought prompting in large language models. In _Proceedings of ICLR_.
* Zhou et al. (2022) Denny Zhou, Nathanael Scharli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuurmans, Claire Cui, Olivier Bousquet, Quoc Le, and Ed Chi. 2022. Least-to-most prompting enables complex reasoning in large language models. In _Proceedings of ICLR_.

## Appendix B Examples for Case Studies

We demonstrate how the rationales evolve with \(0\), \(1\), \(5\) and \(32\) iterations of our method in Table 10.

```
0:A set of questions \({\cal Q}\), the number of demonstrations for unification \(k\), number of iteration \(T\) and number of demonstration for the output m
0:Demonstration list \({\cal D}=[d^{(1)},\dots,d^{(m)}]\)
1:Encode each \(q\in{\cal Q}\) by Sentence-BEKT \(\triangleright\) Question Clustering
2:Cluster all the encoded sequence representations into \(k\) clusters
3:for each cluster \(i=1,...,k\)do
4: Sort questions \({\bf q}^{(i)}=[q^{(i)}_{j},q^{(i)}_{j},\ldots]\) in the ascending order of the distance to the cluster center
5:endfor
6:for each cluster \(i=1,...,k\)do\(\triangleright\) Demonstration Sampling
7:for each equation \(q^{(i)}_{j}\) in \({\bf q}^{(i)}\) do
8: Generate rationale \(r^{(i)}_{j}\) for \(q^{(i)}_{j}\) using Zero-Shot-CoT
9:if\(q^{(i)}_{j},r^{(i)}_{j}\) satisfy selection criteria then
10: Add \(d^{(i)}=q^{(k)}\circ r^{(k)}_{0}\) to \({\cal D}\)
11:break
12:endif
13:endfor
14:endfor
15:for each iteration \(t=1,\dots,T\)do\(\triangleright\) Demonstration Unification
16:for each demonstration \(d^{(i)}\) in \({\cal D}\)do
17: Create prompt \(P\) with demonstrations \({\cal D}\setminus d^{(i)}\) in random order
18: Regenerate rationale \(r^{(i)}_{wav}\) for question \(q^{(i)}\) using Few-Shot-CoT
19: Update \(d^{(i)}=q^{(i)}\circ r^{(i)}_{new}\) to \({\cal D}\)
20:endfor
21:endfor
22:Keep the first run elements and drop the remaining elements from \({\cal D}\).
23:return\({\cal D}\)
```

**Algorithm 1** Self-Harmonized CoT

## Appendix C Data Statistics

Our research indicates that demonstrations with incorrect answers do not necessarily impair overall performance. This observation is particularly evident in the context of the AGUA dataset, where a notable instance occurred: among the four demonstrations generated by ECHO, two contained wrong answers, while in the manually written prompts, all demonstrations were correct. Despite this, ECHO's performance significantly surpassed that of the human-written prompts.

This outcome suggests that in the ECHO method, the collective contribution of the demonstrations to

\begin{table}
\begin{tabular}{l l c c} \hline \hline Reasoning Type & Dataset & Size & Answer Type \\ \hline \multirow{4}{*}{Arithmetic} & SingleEq & 508 & Nameral \\  & AdISub & 395 & Nameral \\  & MultiAttrib & 600 & Nameral \\  & GSMX & 1,319 & Nameral \\  & AUQIA & 254 & Multiple Choice \\  & SVAMP & 1,000 & Nameral \\ \hline Symbolic & Con Flip & 1,000 & Yes or No \\  & Last Letter & 254 & String \\ \hline \multirow{2}{*}{Common Seme} & StrategyQA & 2,290 & Yes or No \\  & CommonsenseQA & 1,221 & Multiple Choice \\ \hline \hline \end{tabular}
\end{table}
Table 8: Data Statistics 

the reasoning pattern is more critical than the individual precision of each demonstration. It appears that the model can effectively extract and leverage useful patterns from the demonstrations, even if some contain errors. This ability to distill valuable reasoning patterns from imperfect data underscores the robustness of the ECHO approach. It indicates that the model's effectiveness relies more on the breadth and diversity of demonstrations rather than their individual correctness. This finding opens up possibilities for utilizing a wider range of demonstrations, including those with inaccuracies, without necessarily compromising the model's overall performance.

## Appendix E More Demonstrations Helps Better Commonsense Reasoning

The overall performance of ECHO (\(k=\max,T=1\)) is not as strong as that of ECHO (\(k=m,T=1\)). We hypothesize that the reason for this is a single iteration is not enough when handling a larger number of demonstrations. To further investigate the impact of increasing the number of demonstrations in the unification process, we conducted experiments with both 8 and the maximum possible number of demonstrations in 4 iterations. 3 The results are illustrated in Table 9.

Footnote 3: maximum refers to the highest number of demonstrations that can be accommodated within the token limit, approximately 20.

Our results reveal that in the arithmetic domain, the performance of ECHO under both conditions is comparable, with each achieving an average score of 83.1%. The notable difference emerges in the realm of commonsense reasoning. In this area, ECHO (\(k=\max,T=1\)) with the maximum number of demonstrations surpasses its counterpart by a margin of 3.0% across both benchmarks. This outcome suggests that the sensitivity to the number of demonstrations varies across different domains, with commonsense reasoning being particularly influenced by the count of demonstrations.

## Appendix F ECHO constructed demonstrations

We append a full list of ECHO-constructed demonstrations.

\begin{table}
\begin{tabular}{l l c c} \hline \hline Type & \multicolumn{3}{c}{Accuracy} \\ \cline{2-5}  & CommonsenseQA & StrategicQA & Average \\ \hline max & 77.5 & 63.4 & 70.5 \\ s & 74.9 & 59.9 & 67.4 \\ \hline \hline \end{tabular}
\end{table}
Table 9: The performances in commonsense reasoning.

 



\begin{table}
\begin{tabular}{c p{14.2pt} l c} \hline Context & Q & Question/Solution & Remark \\ \hline
**Quotions** & **1:** & **Q” Mary is having a cake: The recipe calls for 9 cups of flour and 5 cups of sugar. She already put in 3 cups of flour and 62 cups of sugar. How many more cups of flour does the need to add?** & GT: **G’s** \\  & **2:** & **Q-Dancy collects bottle caps and wrappers. He found 22 needle caps and 30 wrappers at the park. Now he has 17 bottle caps and 57 wrappers in his collection. How many progress did dummy have at first?** & GT: **27** \\
**3:** & **Q- Jerry had s action figures and 94 books on an edit hit in this room. Later he added 10 more bodies to the shelf. How many books were on his shelf in all?** & GT: **19** \\
**4:** & **Q" Every fly Ryan spends 8 hours on learning English and 7 hours on learning chinese. How many more boats des spend on learning Chinese than he does on learning english?** & GT: **1** \\ \hline
* 6 hours (English) = 1 hour Therefore, Ryan spends 1 more born on learning Chinese than he does on learning English. & Correct \\ \hline
* 22 * 3 bottles caps. This doesn’t make sense, so we can assume that Danny had 0 bottle caps before he found the ones at the park. Therefore, Danny had 27 wrappers at first.** & Correct \\  & Surre; let’s break it down. First, we need to find out how many books Jerry hal on the shelf before he added intoe. We know he had 9 books, so the total number of blocks on the shelf before he added more was 9. Next, we need to add the 10 books he added more than 9 books on carrying labels to each entry of their own hands. Therefor, Mary needs to add 6 more cups of flour.** & Correct \\  & Surre; let’s break it down. First, we need to find out how many works Danny had on the shelf before he added more. We know he had 9 books, so the total number of blocks on the shelf before he added more was 9. Next, we need to add the 10 books he added more than 9 books on carrying labels to each entry of their own hands. Herefore, Mary needs to add 6 more cups of flour. & Correct \\  & Surre; let’s break it down. First, we need to find out how many works Danny had there be found the 30 wrappers at the park. We can do this by subtracting the number of hours he spends on learning Chinese: 7 hours (Chinese)
- 6 hours (English) = 1 hour. Therefore, Ryan spends 1 more born on learning Chinese than he does on learning English. & Correct \\  & Surre; let’s break it down. First, we need to find out how many works Danny had there be found the 30 wrappers at the park. We can do this by subtracting the number of hours he spends on learning English from the number of hours he spends on learning Chinese: 7 hours (Chinese)
- 6 hours (English) = 1 hour. Therefore, Ryan spends 1 more born on learning Chinese than he does on learning English. & Correct \\  & Surre; let’s break it down. First, we need to find out how many works Danny had there be found the 30 wrappers at the park. We can do this by subtracting the number of hours he spends on learning English from the number of hours he spends on learning Chinese: 7 hours (Chinese)
- 6 hours (English) = 1 hour. Therefore, Ryan spends 1 more born on learning Chinese than he does on learning English. & Correct \\  & Surre; let’s break it down. First, we need to find out how many works Danny had there be found the 30 wrappers at the park. We can do this by subtracting the number of hours he spends on learning English from the number of hours he days. After words with 6 more cups of flour. & Correct \\  & Surre; let’s break it down. First, we need to find out how many works Danny had there be around the other two books they had there be around the other two books they had there be around the other two books they had there are around the other two books they had there are around the other two books they had there are around the other two books they had there are around the other two books they had there are around the other two books they had there are around the other two books they had there are around the other two books they had there are around the other two books they had there are around the other two books they had there are around the other two books they had there are around the other two books they have around the other two books they have around the other two books they have around the other two books they have around the other two books they have around the other two books they have around the other two books they have around the other two books they have around the other two books they have around the other two books they have around the other two books they have around the other two books they have around the other two books they have around the other two books they have around the other two books they have around the other two books they have around the other two books they have around the other two books they have around the other two books they have around the other two books they have around the other two books they have around the other two books they have around the other two books they have around the other two books they have around the other two books they have around the other two books they have around the other two books they have around the other two books they have around the other two books they have around the other two books they have around the other two books they have around the other two books they have around the other two books they have around the other two books they have around the other two books they have around the other two books they have around the other two books they have around the other two books they have around the other two books they have around the other two books they have around the other two books they have around the other two books they have around the other two books they have around the other two books they have around the other two books they have around the other two books they have around the other two books they have around the other two books they have around the other two books they have around the other two books they have around the other two books they have around the other two books they have around the other two books they have around the other two books they have around the other two books they have around the other two books they have around the other two books they have around the other two books they have around the other two books they have around the other two books they have around the other two books they have around the other two books they have around the other two books they have around the other two books they have around the other two books they have around the other two books they have around the other two books they have around the other two books they have around the other two books they have around the three books they have around the other two books they have around the other two books they have around the other two books they have around the other two books they have around the other two books they have around the other two books they have around the other two books they have around the other two books they have around the other two books they have around the other two books they have around the other two books they have around the other two books they have around the other two books they have around the other two books they have around the other two books they have around the other two books they have around the other two books they have around the other two books they have around the other two books they have around the other two books they have around the other two books they have around the other two books they have around the other two books they have around the other two books they have around the other two books they have around the other two books they have around the other two books they have around the other two books they have around the other two books they have around the other two books they have around the other two books they have around the same two books they have around the other two books they have around the


* [105] M. C. Bender, J. S. Kim, and D. K. Lee, "A new approach to the design of a new model for the 



**Q: Alyssa's high school played 31 hockey games this year. She attended 13 games. How many hockey games did Alyssa miss?**

A: Let's think step by step. Sure, let's break it down. First, we need to find out how many hockey games Alyssa missed. To do this, we can subtract the number of games she attended (13) from the total number of games played (31): 31 - 13 = 18 Therefore, Alyssa missed 18 hockey games.

**Q: Sandy worked 45 hours in the last 5 days. Assuming that she worked the same amount of hours each day, how long did she work each day?**

A: Let's think step by step. Sure, let's break it down. First, we need to find out how many hours Sandy worked each day. To do this, we can divide the total number of hours worked (45) by the number of days worked (5): \(45+5=9\) Therefore, Sandy worked 9 hours each day.

**Q: Sam had 9 dimes in his bank. His dad gave him 7 more dimes. How many dimes does Sam have now?**

A: Let's think step by step. Sure, let's solve this problem step by step. First, we need to add the number of dimes Sam had before to the number of dimes his dad gave him. So, the total number of dimes Sam has now is: 9 + 7 = 16 Therefore, Sam now has 16 dimes.

**Q: On Monday, 375 students went on a trip to the zoo. All 7 buses were filled and 4 students had to travel in cars. How many students were in each bus?**

A: Let's think step by step. Sure, let's break it down. First, we need to find out how many students were on each bus. To do this, we can divide the total number of students (375) by the number of buses (7): 375 + 7 = 53.57 (rounded to the nearest whole number, this is 54) Therefore, there were 54 students on each bus.

**Q: Bryan took a look at his hooks as well. If Bryan has 56 books in each of his 9 bookshelves, how many books does he have in total?**

A: Let's think step by step. Sure, let's break it down. First, we need to find out how many books Bryan has in total. To do this, we can multiply the number of books in each bookshelf (56) by the number of bookshelves (9): 56 x 9 = 504 Therefore, Bryan has 504 books in total.

**Q: There are 107 walnut trees currently in the park. Park workers will plant 104 more walnut trees today. How many walnut trees will the park have when the workers are finished?**

A: Let's think step by step. Sure, let's break it down. First, we need to add the number of walnut trees currently in the park to the number of walnut trees the workers will plant today. 107 (current walnut trees) + 104 (walnut trees to be planted) = 211 Therefore, the park will have 211 walnut trees when the workers are finished.

**Q: Kelly bought 0.1 pounds of peanuts and 0.4 pounds of raisins. How many pounds of snacks did she buy in all?**

A: Let's think step by step. Sure, let's break it down. First, we need to add the weight of peanuts to the weight of raisins. 0.1 pounds (peanuts) + 0.4 pounds (raisins) = 0.5 pounds Therefore, Kelly bought a total of 0.5 pounds of snacks.

**Q: Mike joined his school's band. He bought a trumpet for $145.16, and a song book which was $5.84. How much did Mike spend at the music store?**

A: Let's think step by step. Sure, let's break it down. First, we need to add the cost of the trumpet to the cost of the song book. $145.16 (trumpet) + $5.84 (song book) = $151 Therefore, Mike spent $151 at the music store.



* [105] A. Let's think step by step. Sure, let's break down the problem.
* [106] F. Nester, W. J. H. Lee, and M. R. T. Newman, _The number of people in a social network_, vol. 27, no. 3, pp. 49-58, 2000.

[MISSING_PAGE_POST]

 

* [105] M. C. Bender, J. S. H. Lee, and D. R. K. Smith, "The role of the social network in a social system: A survey on social systems," _Social Networks_**26**, pp. 347-389.
* [106] G. L. Miller, "A study of social networks with social interactions," _Social Networks_**26**, pp. 1-10.
* [107] P. N. F. Newman, "The world of social networks," _Science_**306**, pp. 115-122.
* [108] E. W. Newman, "The social network," _Social Networks_**26**, pp. 1-12.

[MISSING_PAGE_POST]

 

* [105] A. K. He, S. J. Park, and M. H. T. Newman. The number of nodes in the network: a new approach to the problem of finding a node's degree. In _Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining_, pages 3-46, 2007.
* [106] R. K. He, S. J. Park, and M. H. T. Newman. The number of nodes in the network: a new approach to the problem of finding a node's degree. In _Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining_, pages 89-90, 2007.
* [107] C. L. Bennett, P. W. Shorton, and D. E. Berman. Finding a node's degree. _Journal of Computational Physics_, 1(1):1-10, 1994.
* [108] G. F. Hinton, I. Goodfellow, and Y. Bengio. Generative adversarial nets. In _Advances in neural information processing systems_, volume 1, pages 1-10, 2006.
* [109] V. Mnih, N. Bahdan, A. Graves, A. Rabinovici, and O. Vinyals. Deep residual learning for image recognition. In _NIPS Conference on Neural Information Processing Systems_, pages 1-10, 2015.
* [110] A. Mnih, N. Srivastava, A. Sutskever, and I. Sutskever. Very deep networks for large scale machine translation. In _Advances in neural information processing systems_, volume 1, pages 1-10, 2015.
* [111] A. Mnih, N. Srivastava, A. Shazeer, and A. Zisserman. Very deep convolutional neural networks for large scale machine translation. In _Advances in neural information processing systems_, volume 1, pages 1-10, 2015.
* [112] A. Mnih, N. Srivastava, A. Shazeer, and A. Zisserman. Very deep convolutional neural networks for large scale machine translation. In _Advances in neural information processing systems_, volume 1, pages 1-10, 2015.
* [113] A. Mnih, N. Srivastava, A. Shazeer, and A. Zisserman. Very deep convolutional neural networks for large scale machine translation. In _Advances in neural information processing systems_, volume 1, pages 1-10, 2015.
* [114] A. Mnih, N. Srivastava, A. Shazeer, and A. Zisserman. Very deep convolutional neural networks for large scale machine translation. In _Advances in neural information processing systems_, volume 1, pages 1-10, 2015.
* [115] A. Mnih, N. Srivastava, A. Shazeer, and A. Zisserman. Very deep convolutional neural networks for large scale machine translation. In _Advances in neural information processing systems_, volume 1, pages 1-10, 2015.
* [116] A. Mnih, N. Srivastava, A. Shazeer, and A. Zisserman. Very deep convolutional neural networks for large scale machine translation. In _Advances in neural information processing systems_, volume 1, pages 1-10, 2015.
* [117] A. Mnih, N. Srivastava, A. Shazeer, and A. Zisserman. Very deep convolutional neural networks for large scale machine translation. In _Advances in neural information processing systems_, volume 1, pages 1-10, 2015.
* [118] A. Mnih, N. Srivastava, A. Shazeer, and A. Zisserman. Very deep convolutional neural networks for large scale machine translation. In _Advances in neural information processing systems_, volume 1, pages 1-10, 2015.
* [119] A. Mnih, N. Srivastava, A. Shazeer, and A. Zisserman. Very deep convolutional neural networks for large scale machine translation. In _Advances in neural information processing systems_, volume 1, pages 1-10, 2015.
* [120] A. Mnih, N. Srivastava, A. Shazeer, and A. Zisserman. Very deep convolutional neural networks for large scale machine translation. In _Advances in neural information processing systems_, volume 1, pages 1-10, 2015.
* [121] A. Mnih, N. Srivastava, A. Shazeer, and A. Zisserman. Very deep convolutional neural networks for large scale machine translation. In _Advances in neural information processing systems_, volume 1, pages 1-10, 2015.
* [122] A. Mnih, N. Srivastava, A. Shazeer, and A. Zisserman. Very deep convolutional neural networks for large scale machine translation. In _Advances in neural information processing systems_, volume 1, pages 1-10, 2015.
* [123] A. Mnih, N. Srivastava, A. Shazeer, and A. Zisserman. Very deep convolutional neural networks for large scale machine translation. In _Advances in neural information processing systems_, volume 1, pages 1-10, 2015.
* [124] A. Mnih, N. Srivastava, A. Shazeer, and A. Zisserman. Very deep convolutional neural networks for large scale machine translation. In _Advances in neural information processing systems_, volume 1, pages 1-10, 2015.
* [125] A. Mnih, N. Srivastava, A. Shazeer, and A. Zisserman. Very deep convolutional neural networks for large scale machine translation. In _Advances in neural information processing systems_, volume 1, pages 1-10, 2015.
* [126] A. Mnih, N. Srivastava, A. Shazeer, and A. Zisserman. Very deep convolutional neural networks for large scale machine translation. In _Advances in neural information processing systems_, volume 1, pages 1-10, 2015.
* [127] A. Mnih, N. Srivastava, A. Shazeer, and A. Zisserman. Very deep convolutional neural networks for large scale machine translation. In _Advances in neural information processing systems_, volume 1, pages 1-10, 2015.
* [128] A. Mnih, N. Srivastava, A. Shazeer, and A. Zisserman. Very deep convolutional neural networks for large scale machine translation. In _Advances in neural information processing systems_, volume 1, pages 1-10, 2015.
* [129] A. Mnih, N. Srivastava, A. Shazeer, and A. Zisserman. Very deep convolutional neural networks for large scale machine translation. In _Advances in neural information processing systems_, volume 1, pages 1-10, 2015.
* [130] A. Mnih, N. Srivastava, A. Shazeer, and A. Zisserman. Very deep convolutional neural networks for large scale machine translation. In _Advances in neural information processing systems_, volume 1, pages 1-10, 2015.
* [131] A. Mnih, N. Srivastava, A. Shazeer, and A. Zisserman. Very deep convolutional neural networks for large scale machine translation. In _Advances in neural information processing systems_, volume 1, pages 1-10, 2015.
* [132] A. Mnih, N. Srivastava, A. Shazeer, and A. Zisserman. Very deep convolutional neural networks for large scale machine translation. In _Advances in neural information processing systems_, volume 1, pages 1-10, 2015.
* [133] A. Mnih, N. Srivastava, A. Shazeer, and A. Zisserman. Very deep convolutional neural networks for large scale machine translation. In _Advances in neural information processing systems_, volume 1, pages 1-10, 2015.
* [134] A. Mnih, N. Srivastava, A. Shazeer, and A. Zisserman. Very deep convolutional neural networks for large scale machine translation. In _Advances in neural information processing systems_, volume 1, pages 1-10, 2015.
* [135] A. Mnih, N. Srivastava, A. Shazeer, and A. Zisserman. Very deep convolutional neural networks for large scale machine translation. In _Advances in neural information processing systems_, volume 1, pages 1-10, 2015.
* [136] A. Mnih, N. Srivastava, A. Shazeer, and A. Zisserman. Very deep convolutional neural networks for large scale machine translation. In _Advances in neural information processing systems_, volume 1, pages 1-10, 2015.
* [137] A. Mnih, N. Srivastava, A. Shazeer, and A. Zisserman. Very deep convolutional neural networks for large scale machine translation. In _Advances in neural information processing systems_, volume 1, pages 1-10, 2015.
* [138] A. Mnih, N. Srivastava, A. Shazeer, and A. Zisserman. Very deep convolutional neural networks for large scale machine translation. In _Advances in neural information processing systems_, volume 1, pages 1-10, 2015.
* [139] A. Mnih, N. Srivastava, A. Shazeer, and A. Zisserman. Very deep convolutional neural networks for large scale machine translation. In _Advances in neural information processing systems_, volume 1, pages 1-10, 2015.
* [140] A. Mnih, N. Srivastava, A. Shazeer, and A. Zisserman. Very deep convolutional neural networks for large scale machine translation. In _Advances in neural information processing systems_, volume 1, pages 1-10, 2015.
* [141] A. Mnih, N. Srivastava, A. Shazeer, and A. Zisserman. Very deep convolutional neural networks for large scale machine translation. In _Advances in neural information processing systems_, volume 1, pages 1-10, 2015.
* [142] A. Mnih, N. Srivastava, A. Shazeer, and A. Zisserman. Very deep convolutional neural networks for large scale machine translation. In _Advances in neural information processing systems_, volume 1, pages 1-10, 2015.
* [143] A. Mnih, N. Srivastava, A. Shazeer, and A. Zisserman. Very deep convolutional neural networks for large scale machine translation. In _Advances in neural information processing systems_, volume 1, pages 1-10, 2015.
* [144] A. Mnih, N. Srivastava, A. Shazeer, and A. Zisserman. Very deep convolutional neural networks for large scale machine translation. In _Advances in neural information processing systems_, volume 1, pages 1-10, 2015.
* [145] A. Mnih, N. Srivastava, A. Shazeer, and A. Zisserman. Very deep convolutional neural networks for large scale machine translation. In _Advances in neural information processing systems_, volume 1, pages 1-10, 2015.
* [146] A. Mnih, N. Srivastava, A. Shazeer, and A. Zisserman. Very deep convolutional neural networks for large scale machine translation. In _Advances in neural information processing systems_, volume 1, pages 1-10, 2015.
* [147] A. Mnih, N. Srivastava, A. Shazeer, and A. Zisserman. Very deep convolutional neural networks for large scale machine translation. In _Advances in neural information processing systems_, volume 1, pages 1-10, 2015.
* [148] A. Mnih, N. Srivastava, A. Shazeer, and A. Zisserman. Very deep convolutional neural networks for large scale machine translation. In _Advances in neural information processing systems_, volume 1, pages 1-10, 2015.
* [149] A. Mnih, N. Srivastava, A. Shazeer, and A. Zisserman. Very deep convolutional neural networks for large scale machine translation. In _Advances in neural information processing systems_, volume 1, pages 1-10, 2015.
* [150] A. Mnih, N. Srivastava, A. Shazeer, and A. Zisserman. Very deep convolutional neural networks for large scale machine translation. In _Advances in neural information processing systems_, volume 1, pages 1-10, 2015.
* [151] A. Mnih, N. Srivastava, A. Shazeer, and A. Zisserman. Very deep convolutional neural networks for large scale machine translation. In _Advances in neural information processing systems_, volume 1, pages 1-10, 2015.
* [152] A. Mnih, N. Srivastava, A. Shazeer, and A. Zisserman. Very deep convolutional neural networks for large scale machine translation. In _Advances in neural information processing systems_, volume 1, pages 1-10, 2015.
* [153] A. Mnih, N. Srivastava, A. Shazeer, and A. Zisserman. Very deep convolutional neural networks for large scale machine translation. In _Advances in neural information processing systems_, volume 1, pages 1-10, 2015.
* [154] A. Mnih, N. Srivastava, A. Shazeer, and A. Zisserman. Very deep convolutional neural networks for large scale machine translation. In _Advances in neural information processing systems_, volume 

* [105] M. C. Bader, A. J. Burger, and R. G. Levitt, _Theoretical Computer Science: An Introduction to the Theory of Computing_. Springer-Verlag, Berlin, Heidelberg, 2003.
* [106] D. E. Bertsekas, _A new method for computing 

* [164] A. Let's think step by step. If you were looking for a ticket booth and a concert? Answer Choices: (A) derk (B) indoors (C) movie theater (D) venue (E) auditorium

A: Let's think step by step. If you were looking for a ticket booth and a concert, you would likely find them at a (D) venue or (E) auditorium. These are locations where concerts and other live events are typically held, and where tickets are sold at a ticket booth. Option (A) clerk and option (B) indoors are not necessarily related to the context of the question or the experience of attending a concert. Option (C) movie theater is not appropriate, as it is not typically associated with live music events. Therefore, the most appropriate answer is (D) venue or (E) auditorium.

**Q: What must someone do before they shop? Answer Choices: (A) get money (B) have money (C) bring cash (D) go to market (E) bring cash**

A: Let's think step by step. Before someone can shop, they must have a means of payment. Therefore, the most appropriate answer is (A) get money or (B) have money. Option (C) bring cash is a specific means of payment, but it is not necessarily the only option. Option (D) go to market is not necessarily related to the context of the question or the act of shopping. Therefore, the most appropriate answer is (A) get money or (B) have money.

**Q: John went to a party that lasted all night. Because of this, he didn't have time for what? Answer Choices: (A) meeting (B) blowing off steam (C) stay home (D) partying hard (E) studying**

A: Let's think step by step. If John went to a party that lasted all night, he likely did not have time for (E) studying or other academic or work-related responsibilities. Option (A) meeting and option (B) blowing off steam are not necessarily related to the context of the question or the experience of attending a party. Option (C) stay home is the opposite of attending a party and is not a valid answer. Option (D) partying hard is redundant and not appropriate. Therefore, the most appropriate answer is (E) studying.

**Q: Where might someone store a reusable shopping bag? Answer Choices: (A) supermarket (B) home (C) mart (D) obesity (E) closet**

A: Let's think step by step. Someone might store a reusable shopping bag at (B) home or in a (E) closet. These are common locations for storing reusable bags when they are not in use. Option (A) supermarket and option (C) mart are not appropriate choices, as they are locations where shopping bags are typically used rather than stored. Option (D) obesity is not related to the context of the question or the act of storing a reusable shopping bag. Therefore, the most appropriate answers are (B) home and (E) closet.

**Q: What is someone usually doing if someone else is talking to him or her? Answer Choices: (A) listening (B) language (C) looking at eyes (D) planning the perfect murder (E) voice**

A: Let's think step by step. If someone else is talking to you, you are usually (A) listening to them. Listening is the act of paying attention to what someone else is saying and processing their words and meaning. Option (B) language, option (C) looking at eyes, option (D) planning the perfect murder, and option (E) voice are not necessarily related to the context of the question or the act of listening to someone else. Therefore, the most appropriate answer is (A) listening.

**Q: What is the process of going somewhere? Answer Choices: (A) fire (B) energy (C) car (D) transporting (E) staying in place**

A: Let's think step by step. The process of going somewhere typically involves (D) transporting oneself from one location to another. Option (A) fire and option (B) energy are not necessarily related to the context of the question or the process of going somewhere. Option (C) car is a possible means of transportation, but it is not necessarily the only way to go somewhere. Option (E) staying in place is the opposite of going somewhere and is not a valid answer. Therefore, the most appropriate answer is (D) transporting.

**Q: The freeway had no traffic and few buildings, where is it! Answer Choices: (A) california (B) countryide (C) big town (D) florida (E) america**

A: Let's think step by step. The description of the freeway having no traffic and few buildings suggests a more rural or suburban environment. Therefore, the most appropriate answer is (B) countryside. Option (A) California, option (C) big town, option (D) Florida, and option (E) America are not necessarily related to the context of the question or the description of the freeway. Therefore, the most appropriate answer is (B) counryside.

\begin{table} \end{table}
Table 17: Few-shot samples for CommonsenseQA.

 

* [105] M. C. Bender, A. K. Sridharan, and R. J. Schultz, "A new approach to the design of a single-layer model for a large number of applications," in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_, pages 237-246, 2009.
* [106] D. E. Gower, P. W. H. van der Heuvel, and T. L. Van den Drielke, "The design of a single layer model for a large number of applications," _IEEE Transactions on Information Theory and Systems_, vol. 8, no. 1, pp. 1-10, Jan. 1996.
* [107] D. E. Gower, P. W. H. van der Heuvel, and T. L. Van den Drielke, "Designing the design of a single layer model for a large number of applications," _IEEE Transactions on Information Theory and Systems_, vol. 8, no. 1, pp. 1-10, Jan. 1996.
* [108] D. E. Gower, P. W. H. van der Heuvel, and T. L. Van den Drielke, "Designing the design of a single layer model for a large number of applications," _IEEE Transactions on Information Theory and Systems_, vol. 8, no. 1, pp. 1-10, Jan. 1996.
* [109] D. E. Gower, P. W. H. van der Heuvel, and T. L. Van den Drielke, "Designing the design of a single layer model for a large number of applications," _IEEE Transactions on Information Theory and Systems_, vol. 8, no. 1, pp. 1-10, Jan. 1996.
* [110] D. E. Gower, P. W. H. van der Heuvel, and T. L. Van den Drielke, "Designing the design of a single layer model for a large number of applications," _IEEE Transactions on Information Theory and Systems_, vol. 8, no. 1, pp. 1-10, Jan. 1996.
* [111] D. E. Gower, P. W. H. van der Heuvel, and T. L. Van den Drielke, "Designing the design of a single layer model for a large number of applications," _IEEE Transactions on Information Theory and Systems_, vol. 8, no. 1, pp. 1-10, Jan. 1996.
* [112] D. E. Gower, P. W. H. van der Heuvel, and T. L. Van den Drielke, "Designing the design of a single layer model for a large number of applications," _IEEE Transactions on Information Theory and Systems_, vol. 8, no. 1, pp. 1-10, Jan. 1996.
* [113] D. E. Gower, P. W. H. van der Heuvel, and T. L. Van den Drielke, "Designing the design of a single layer model for a large number of applications," _IEEE Transactions on Information Theory and Systems_, vol. 8, no. 1, pp. 1-10, Jan. 1996.
* [114] D. E. Gower, P. W. H. van der Heuvel, and T. L. Van den Drielke, "Designing the design of a single layer model for a large number of applications," _IEEE Transactions on Information Theory and Systems_, vol. 8, no. 1, pp. 1-10, Jan. 1996.
* [115] D. E. Gower, P. W. H. van der Heuvel, and T. L. Van den Drielke, "Designing the design of a single layer model for a large number of applications," _IEEE Transactions on Information Theory and Systems_, vol. 8, no. 1, pp. 1-10, Jan. 1996.
* [116] D. E. Gower, P. W. H. van der Heuvel, and T. L. Van den Drielke, "Designing the design of a single layer model for a large number of applications," _IEEE Transactions on Information Theory and Systems_, vol. 8, no. 1, pp. 1-10, Jan. 1996.
* [117] D. E. Gower, P. W. H. van der Heuvel, and T. L. Van den Drielke, "Designing the design of a single layer model for a large number of applications," _IEEE Transactions on Information Theory and Systems_, vol. 8, no. 1, pp. 1-10, Jan. 1996.
* [118] D. E. Gower, P. W. H. van der Heuvel, and T. L. Van den Drielke, "Designing the design of a single layer model for a large number of applications," _IEEE Transactions on Information Theory and Systems_, vol. 8, no. 1, pp. 1-10, Jan. 1996.
* [119] D. E. Gower, P. W. H. van der Heuvel, and T. L. Van den Drielke, "Designing the design of a single layer model for a large number of applications," _IEEE Transactions on Information Theory and Systems_, vol. 8, no. 1, pp. 1-10, Jan. 1996.
* [120] D. E. Gower, P. W. H. van der Heuvel, and T. L. Van den Drielke, "Designing the design of a single layer model for a large number of applications," _IEEE Transactions on Information Theory and Systems_, vol. 8, no. 1, pp. 1-10, Jan. 1996.
* [121] D. E. Gower, P. W. H. van der Heuvel, and T. L. Van den Drielke, "Designing the design of a single layer model for a large number of applications," _IEEE Transactions on Information Theory and Systems_, vol. 8, no. 1, pp. 1-10, Jan. 1996.
* [122] D. E. Gower, P. W. H. van der Heuvel, and T. L. Van den Drielke, "Designing the design of a single layer model for a large number of applications," _IEEE Transactions on Information Theory and Systems_, vol. 8, no. 1, pp. 1-10, Jan. 1996.
* [123] D. E. Gower, P. W. H. van der Heuvel, and T. L. Van den Drielke, "Designing the design of a single layer model for a large number of applications," _IEEE Transactions on Information Theory and Systems_, vol. 8, no. 1, pp. 1-10, Jan. 1996.
* [124] D. E. Gower, P. W. H. van der Heuvel, and T. L. Van den Drielke, "Designing the design of a single layer model for a large number of applications," _IEEE Transactions on Information Theory and Systems_, vol. 8, no. 1, pp. 1-10, Jan. 1996.
* [125] D. E. Gower, P. W. H. van der Heuvel, and T. L. Van den Drielke, "Designing the design of a single layer model for a large number of applications," _IEEE Transactions on Information Theory and Systems_, vol. 8, no. 1, pp. 1-10, Jan. 1996.
* [126] D. E. Gower, P. W. H. van der Heuvel, and T. L. Van den Drielke, "Designing the design of a single layer model for a large number of applications," _IEEE Transactions on Information Theory and Systems_, vol. 8, no. 1, pp. 1-10, Jan. 1996.
* [127] D. E. Gower, P. W. H. van der Heuvel, and T. L. Van den Drielke, "Designing the design of a single layer model for a large number of applications," _IEEE Transactions on Information Theory and Systems_, vol. 8, no. 1, pp. 1-10, Jan. 1996.
* [128] D. E. Gower, P. W. H. van der Heuvel, and T. L. Van den Drielke, "Designing the design of a single layer model for a large number of applications," _IEEE Transactions on Information Theory and Systems_, vol. 8, no. 1, pp. 1-10, Jan. 1996.
* [129] D. E. Gower, P. W. H. van der Heuvel, and T. L. Van den Drielke, "Designing the design of a single layer model for a large number of applications," _IEEE Transactions on Information Theory and Systems_, vol. 8, no. 1, pp. 1-10, Jan. 1996.
* [130] D. E. Gower, P. W. H. van der Heuvel, and T. L. Van den Drielke, "Designing the design of a single layer model for a large number of applications," _IEEE Transactions on Information Theory and Systems_, vol. 8, no. 1, pp. 1-10, Jan. 1996.
* [131] D. E. Gower, P. W. H. van der Heuvel, and T. L. Van den Drielke, "Designing the design of a single layer model for a large number of applications," _IEEE Transactions on Information Theory and Systems_, vol. 8, no. 1, pp. 1-10, Jan. 1996.
* [132] D. E. Gower, P. W. H. van der Heuvel, and T. L. Van den Drielke, "Designing the design of a single layer model for a large number of applications," _IEEE Transactions on Information Theory and Systems_, vol. 8, no. 1, pp. 1-10, Jan. 1996.
* [133] D. E. Gower, P. W. H. van der Heuvel, and T. L. Van den Drielke, "Designing the design of a single layer model for a large number of applications," _IEEE Transactions on Information Theory and Systems_, vol. 8, no. 1, pp. 1-10, Jan. 1996.
* [134] D. E. Gower, P. W. H. van der Heuvel, and T. L. Van den Drielke, "Designing the design of a single layer model for a large number of applications," _IEEE Transactions on Information Theory and Systems_, vol. 8, no. 1, pp. 1-10, Jan. 1996.
* [135] D. E. Gower, P. W. H. van der Heuvel, and T. L. Van den Drielke, "Designing the design of a single layer model for a large number of applications," _IEEE Transactions on Information Theory and Systems_, vol. 8, no. 1, pp. 1-10, Jan. 1996.
* [136] D. E. Gower, P. W. H. van der Heuvel, and T. L. Van den Drielke, "Designing the design of a single layer model for a large number of applications," _IEEE Transactions on Information Theory and Systems_, vol. 8, no. 1, pp. 1-10, Jan. 1996.
* [137] D. E. Gower, P. W. H. van der Heuvel, and T. L. Van den Drielke, "Designing the design of a single layer model for a large number of applications," _IEEE Transactions on Information Theory and Systems_, vol. 8, no. 1, pp. 1-10, Jan. 1996.
* [138] D. E. Gower, P. W. H. van der Heuvel, and T. L. Van den Drielke, "Designing the design of a single layer model for a large number of applications," _IEEE Transactions on Information Theory and Systems_, vol. 8, no. 1, pp. 1-10, Jan. 1996.
* [139] D. E. Gower, P. W. H. van der Heuvel, and T. L. Van den Drielke, "Designing the design of a single layer model for a large number of applications," _IEEE Transactions on Information Theory and Systems_, vol. 8, no. 1, pp. 1-10, Jan. 1996.
* [140] D. E. Gower, P. W. H. van der Heuvel, and T. L. Van den Drielke, "Designing the design of a single layer model for a large number of applications," _IEEE Transactions on Information Theory and Systems_, vol. 8, no. 1, pp. 1-10, Jan. 1996.
* [141] D. E. Gower, P. W. H. van der Heuvel, and T. L. Van den Drielke, "Designing the design of a single layer model for a large number of applications," _IEEE Transactions on Information Theory and Systems_, vol. 8, no. 1, pp. 1- 

* [105] M. C. Bader, J. D. Murray, and R. A. Olson. The role of the coin in a new era: how to do it? _Nature_**463**, 279-384 (2000).

[MISSING_PAGE_POST]