Semantic Self-Consistency: Enhancing Language Model Reasoning via Semantic Weighting

Authors: tim  knappe
ryan  li
ayush  chauhan
kaylee  chhua
kevin  zhu
sean  o'brien
Venue: NeurIPS_cc_2024_Workshop_MATH-AI
Published: 2024-09-26 14:00:21

Abstract: While large language models (LLMs) have rapidly improved their performance on a broad number of tasks, they still often fall short on reasoning tasks. As LLMs become more integrated in diverse real-world tasks, advancing their reasoning capabilities is crucial to their effectiveness in nuanced, complex problems. \citet{wang2023selfconsistency}’s \textit{self-consistency} framework reveals that sampling multiple rationales before taking a majority vote reliably improves model performance across various closed-answer reasoning tasks. Standard methods based on this framework aggregate the final decisions of these rationales but fail to utilize the semantic information detailed in the step-by-step reasoning paths. Our work introduces \textit{semantic self-consistency}, enhancing this approach by incorporating and analyzing both the reasoning paths of these rationales in addition to their final decisions before taking a majority vote. These methods not only improve the reliability of reasoning paths but also cause more robust performance on complex reasoning tasks.

Main paper: 
 introduction: 
 In recent years, the development of large language models has witnessed remarkable strides, with significant advancements in their accuracy and expressive capabilities [3, 28, 24, 4] . Despite these achievements, models still perform suboptimally in domains such as mathematics, commonsense, and complex algorithmic reasoning [10] . Various methods such as chain-of-thought prompting have been developed to further increase reasoning capabilities and was further enhanced by the introduction of self-consistency, which demonstrate that baselines can be pushed forward by sampling and ensembling multiple model responses with chain-of-thought to improve prediction quality [34, 23] .We build on the framework of self-consistency, proposing two techniques that add a separate semantic weighting step to rerank results based on their reasoning paths. To achieve this, we use semantic vector embeddings in combination with self-consistency to group consistent model outputs, aiding in the identification of similar responses to estimate the most likely output. Additionally, we introduce a semantic filtering mechanism that discards degenerate or hallucinated outputs, which can be utilized for analyzing smaller sample sizes. Overall, we demonstrate that self-consistency with semantic marginalization not only improves accuracy across a range of benchmarks but also serves as a filtering mechanism. By introducing these methods, we aim to provide a framework for improving performance and analyzing the semantic usage of model outputs in reasoning. 3) choose results based on the most occurring final output, our proposed method, shown above, decides based on the semantic consistency of the employed reasoning path. Our assumption is that language models often apply the correct reasoning but lack the ability to conclude to the correct result.To enhance separation and clustering in t-SNE for data exploration and pattern recognition tasks, we use a perplexity parameter of 2. This choice is based on the fact that local distributions in out scenario provide a more informative representation than global distributions due to the increased density of points in close proximity, which improves the detail captured in the mapping.Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?Answer: [Yes]Justification: Results and impact are discussed in the abstract, discussion and inside the limitations.Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 
datasets: 
 We evaluate the models on arithmetic and commonsense reasoning using three datasets: AQuA-RAT, SVAMP, and StrategyQA. AQuA-RAT assesses models' ability to solve arithmetic problems involving basic calculations, numerical relationships, and multi-step reasoning [17] . SVAMP challenges models with math problems focused on algebraic manipulations and symbolic reasoning [25] . Strate-gyQA tests models on answering complex, open-domain questions that require strategic thinking rather than simple factual knowledge [9] . For specific information please refer to Appendix L.Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes]Justification: We provide an in depth acessment of our Limitations in our Limitations Section, backed up by additional prove in the appendix and Ethical Considerations.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
language models: 
 Our models are categorized into two types: generators, which produce sequences such as text, code, or reasoning steps, and featurizers, which transform these outputs into numerical representations (vector embeddings) that summarize their meaning for analysis.Detailed information on the configurations used for our models can be found in Appendix I. 3 . Additionally important hyperparameters for different methods are discussed in Appendix I We use chain-of-thought prompting for all of our experiments. The prompts can be found in Appendix K.For our evaluation, we use several models with varying architectures and sizes. First, we utilize GPT-3.5, a closed-source large-scale transformer model developed by OpenAI [3] . Additionally, we evaluate both Llama 2 (7B parameters) [32] and Llama 3 (8B parameters) [8] , which are open-weight models known for their strong performance on numerous benchmarks. We also include Mistral 7B (version 0.1), recognized for its robust capabilities across a variety of language processing tasks [13] . Lastly, we assess GPT-4o mini, a lower parameter variant of the GPT-4o architecture that balances computational efficiency with high performance across diverse language tasks.All of our featurizers are based on the BERT (Bidirectional Encoder Representations from Transformers) model architecture [7] , with various fine-tuned versions used to generate embedding vectors tailored to specific datasets. RoBERTa is employed for the StrategyQA dataset, which requires reading comprehension and contextual reasoning, benefiting from RoBERTa's robustness in general language processing tasks [19] . Additionally, we use SciBERT for the AQuA-RAT and SVAMP datasets, which focus on mathematical reasoning, as its specialization in scientific texts makes it well-suited to handle the language present in these datasets [2] .We apply various algorithms to weight and aggregate the responses based on their featurized embedding vectors, enhancing decisionmaking by emphasizing semantically consistent reasoning paths or removing outliers. . Finally, the total weight for each unique output is computed as sum_weights[u] = i∈I(u) weights[i], where outputs with the highest total weights are considered the most likely to be correct.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?Answer: [No] Justification: All datasets can easily be accessed, specific command environments are not provided. With the information given about used formulars and featurizers, the code can reproduced and tweaked for the specific usecase as intended.Guidelines:• The answer NA means that paper does not include experiments requiring code. • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
methodology: 
 We analyze three main mechanisms for weighting and categorization (CPW, sequence comparison, and filtering of anomalous points) that follow a similar operational pattern outlined below:1. Generate candidate responses: Given a query of few-shot examples, we generate n samples based on chain-of-thought prompting [34] .2. Embed reasoning paths: We represent each generated rationale as a vector embedding using fine-tuned BERT models (e.g., SciBERT for mathematical reasoning tasks). Instead of focusing on individual sentences or tokens, we obtain a single vector representation for each entire reasoning path, capturing its overall semantic content.To compare the weighting of embedding positions, we introduce another method and weigh responses relative to their respective sequences with cosine similarity, a measurement of how similar two vectors. We take n 1 , n 2 , n 3 , . . . , n i as distinct elements in our set N , where each n corresponds to a featurized embedding vector. The cosine similarity between vectors n a and n b is given by cosine_similarity(n a , n b ) = na•n b ∥na∥2∥n b ∥2, and for each n e , we compute the cosine similarity with every n i in N and aggregate the scores: S ne = ni∈N cosine_similarity(n e , n i ). This process is repeated for each n j in N , resulting in aggregated scores S n1 , S n2 , S n3 , . . . , S ni , and the scores are summed based on their answer decision, leading to the selection of the highest consensual response.To eliminate outliers, we filter responses based on proximity [18, 21, 5] , isolating data points that significantly deviate and identifying flawed reasoning, degenerated outputs, or model hallucinations. We examine the following common methods: (1) K-nearest neighbor, using n i=1 (x i -y i ) 2 ;(2) Isolation forest, where s(x, n) = 2 -E(h(x)) c(n) ; and (3) Support vector machines, defined by1 2 ω T ω + C n i=1 ζ i . 
semantic consistency results: 
 We compared Centroid Proximity Weighting (CPW) and Semantic Consensus Weighting (SCW) with the self-consistency baseline across datasets. As shown in CPW improved self-consistency by 3.14% on AQuA-RAT and 0.97% on SVAMP but decreased performance by -1.63% on StrategyQA, likely due to its limited reasoning paths. This effect was seen across self-consistency, where improvements were smaller compared to other datasets. A detailed discussion of these suboptimal cases is in Appendix D.SCW showed that weighting sequences based on consistency reduces errors and improves accuracy, outperforming baseline self-consistency.The results from our analysis of various outlier detection methods isolation forest, k-nearest neighbor, one-class support vector machines (SVM) demonstrate their effectiveness in refining the quality of model output. The observed increases in accuracy across these methods remain consistent towards reduced sample sizes as well, suggesting that the effectiveness of anomaly detection techniques are not solely dependent on sample size. Obtained results exhibited slight deviations between the different configurations. A review across different sets of configurations and parameters can be found under Appendix I.2.1 to I.2.3.The found results highlight variability across datasets, with isolation forest and one-class SVM performing better on certain datasets. 
discussion: 
 It is worth noting that our system uses embedding vectors to filter responses based on general reasoning accuracy, prioritizing broad similarity over subtle variations, as the benefit of choosing the numerical majority vote from self-consistency to yield correct answers still applies, especially in the limited rationale space. An additional analysis can be found in Appendix B.Diverse responses are not necessarily undesirable and can lead to elevated results as shown in Appendix G.1. Against the natural feel, employed methods do not discriminate against diverse reasoning. Lowering the temperature will make multiple responses more diverse and, therefore, broaden the distribution. This will not affect performance when using CPW or outlier detection, since outputs farther from the mean are not outliers but sensible parts of a wider distribution. Consequently, the weighting process will remain consistent, as all values will proportionally receive lower weights.Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes] Justification: Our appendix aswell as our Dataset and Model section provide detail to the configuration of models and dataset splits and usage.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
conclusion: 
 Our investigation into weighting and anomaly detection methods shows that cosine similarity outperforms CPW in improving model accuracy, particularly for models like Llama 2 7B and GPT 3.5 on datasets such as StrategyQA. CPW was effective for AQuA-RAT and SVAMP, leading to accuracy increases, but less so for StrategyQA. Our system prioritizes general reasoning accuracy using embedding vectors, with numerical majority voting from self-consistency remaining a key factor in achieving correct answers, especially within limited rationale spaces. Please note that the recommended methods should be employed with carefully tested hyperparameters, as their effectiveness may vary with subtle implementation nuances.Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [Yes] Justification: We use different methods to evaluate significance and discuss those in the results and discussion.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.• The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. Justification: We show our GPU usage in our Reproducibility statement and in certain parts of the Appendix.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). 
related work: 
 Reasoning is an ubiquitous issue across many domains. [6] . One significant advancement in the area has been the development of the chain-of-thought prompting [34, 27] and self-consistency [33] , which we extend for our Method. Self improvement of Language Models after generation is a well-known method for improving accuracy [12] . This concept has often been adapted by other weighting methods during pre-training to improve overall accuracy [31, 20] , using different methods to shift the distribution [14] . 
limitations: 
 Our study proposes the application of semantic vector representations to group and weigh model outputs, which is designed to facilitate the identification of consensus responses [33] . Semantic vectors must capture variations in meaning and context, which is particularly hard in abstract reasoning tasks without a sufficient amount of context making prompting techniques to enhance the models output structure and size an important factor as visualized in Table 3 . The process of clustering based on semantic vectors can be challenging due to the nuanced and abstract nature of reasoning processes. This limitation underscores the need for advanced featurization models and explicit choice of a fitting fine-tuned model [22] . Like showcased in Table 6 , multiple models should be considered for semantic analysis, to ensure that the model outputs are grouped in a way that truly reflects their underlying meaning and relevance. Without these fitting featurizers, on fields with more subtle variations or on short sequences, the employed method might not be able to distinguish different sequences well enough to uphold a notable positive effect.Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines? 
reproducibility statement: 
 Our experiments include a variety of models with different sizes. GPT 3.5 as well as GPT-4o mini have API endpoints that are open for public use https://openai.com/blog/openai-api.Mistral 7B is available for unrestricted use under the Apache 2.0 license, while its model architecture and setup are open source: https://github.com/Mistralai/Mistral-src.Llama 2 7B and Llama 3 8B are models with restricted access, made available by Meta. One can gain access to them by requesting permission through the provided Meta license. https: //ai.meta.com/llama/.All of our BERT models are built upon the BERT-base model developed by google-research, which is accessible under the Apache 2.0 license, including MathBERT and SciBERT. RoBERTa can be used under the MIT license.Our datasets as well as the configurations used for our language models are accessible throughout this paper and in the Appendix to aid the reproducibility of our experiments. Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes] Justification: We show impact of our work in the Ethics statement and discuss usage of Language Models.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
ethical considerations & risks: 
 Language models may produce factually incorrect or biased outputs based on user prompts. The BERT-based featurizers, trained on English corpora, may yield inconsistent results in other languages. Mistral 7B, Llama 2 7B, and Llama 3 8B lack built-in content moderation, needing external safeguards against harmful content. While GPT-4o and GPT-4o mini have stronger moderation, biases may still emerge.Further risks include that embedding and clustering methods may introduce subtle biases by emphasizing specific response types over others. Additionally variations in model temperature and sampling can add unintended randomness. Controlled sampling and inverse temperature weighting help but require careful tuning.We recommend using monitoring tools and responsible model deployment, particularly in high-stakes applications.Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: Our methods cant be exploited for misuse in such a scenario.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes]Justification: We credit all owners, by citing their work and provide a detailed overview over licenses for models and datasets, in the reproducibility statement.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Answer: [NA] Justification: We introduce no new assets. Details of Models, Datasets and Code are provided.Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 
crowdsourcing and research with human subjects: 
 Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?Answer: [NA]Justification: The paper does not involve crowdsourcing nor research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector. 
institutional review board (irb) approvals or equivalent for research with human subjects: 
 Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?Answer: [NA]Justification: The paper does not involve crowdsourcing nor research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.• Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: Across different findings, we see a variation in performance with a general upward trend. As shown in Section 3 and discussed in Appendix B, sequence length seems to affect model performance positively. Smaller sequences tend to contain to be less similar in terms of informational density compared to all other sequences.Moreover, GPT 3.5's and GPT-4o mini's instruction fine-tuning positively affects sequence length and output content, leading to longer and more contextual sentences. Additionally, there's a trend towards larger models, suggesting that increased parameter size may improve performance across tasks and the way information is packed across the exemplars.Subtle variations in reasoning or content, particularly in fields like mathematics, can lead to significant divergences in outcomes, suggesting a preference for symbolic logic to distinguish these differences precisely. This approach presupposes that correct reasoning across various contexts tends to follow similar operational patterns. By leveraging embedding vectors, the system isolates responses that deviate significantly in reasoning quality or factuality, rather than getting entangled in the minutiae of every possible variation. Thus, while embedding vectors may overlook some subtle differences, their use is justified by their effectiveness in broadly categorizing and filtering responses according to general reasoning accuracy.Additionally, the inherently delivered effect of self-consistency implies that multiple exemplars, when exhibiting correct or similar reasoning, will eventually result in the majority of correct numerical answers, which will prove especially effective when the space of rationales is limited to these that are sufficiently supported by its reasoning path.We observe a slight correlation between the average sequence length generated by our models and improvements in accuracy, emphasizing the role of exemplar selection in the reasoning process. Longer chains of thought can provide more context, but they are also more prone to outliers and inaccuracies. Similarly, shorter sequences often lack sufficient context to differentiate responses effectively.Although sequence length scales with accuracy, we observe no correlation between accuracy and the averaged BLEU score. This suggests that improvements in text generation quality, as measured by BLEU, do not necessarily translate to better reasoning accuracy, underscoring the trade-off between context depth and noise in model predictions. It is important to distinguish that the employed system focuses on identifying consensual responses and broader similarity in the representational space of embeddings, rather then subtle nuances. A clear analysis of our embeddings in connection to symbolic logic and subtle details can be found in Appendix B.To test our embeddings and ensure that embeddings do not solely discriminate on numerical output, we randomly removed numerical outputs before generating embedding vectors. As visible in the results, performance remained stable and proves that even correctly reasoned but arithmetically incorrect responses can still be used in different methods to enhance overall output quality and mechanisms that make use of semantic evaluation.Further analysis of both the embedding distribution as well as our dimensionality reduction can be found in Appendix O.Although we observe an upward trend in performance, there are certain scenarios where the applied methods fail to deliver the desired results.• Overly similar generations: Generations that provide overly similar reasoning will likely be categorized in a similar position in the embedding space, which will lead to our semantic methods, not being able to discern between elements. • Small subtleties in generations: As described in Section B & Section 6 small subtleties aren't captured directly by our model, making it less capable in tasks like Symbolic Reasoning or state tracking.While meta-reasoning has proven effective on tasks that have qualitative evident information, its ability to stay consistent between arithmetic operations and its subsequent reasoning path witnesses the same limitations as baseline self-consistency and chain-of-thought [35] .Unlike previously established self-improvement and Importance weighting methods as proposed by Jiang et al. [14] . Our methods weighs results directly after generation in a separate weighting/filtering step. While results showed some frailty if not tuned with fitting parameters we spare computational efforts by not requiring an addition pre-training step. Pre-trained self-improvement Models could be used together with our introduced weighting method, to test performance and facilitate accuracy even furtherIn the evaluation of AQuA-RAT, some results exhibited noise. Particularly smaller models failed to consistently follow the few-shot chain-of-thought examples occasionally. This led to instances where outputs could not be parsed for final analysis. To ensure reproducibility, we employed the parsing extraction approach from baseline self-consistency. Furthermore, some models showed degeneration after generating the initial response, highlighting the need of development for a custom extraction function to ensure accurate semantic interpretation, particularly when utilizing functions that include embeddings.Other than self-consistency our methods require additional computation, other than the initial generation to compute its results.• Embeddings: The computational cost is moderate, as the BERT model utilized is of a manageable size, keeping resource usage at a reasonable level.• Centroid Proximity Weighting: This method is computationally inexpensive, as it relies solely on mathematical operations without requiring extensive resources.• Semantic Consensus Weighting: Similarly, this technique is computationally efficient due to its reliance on lightweight mathematical computations.• Outlier Detection: All three outlier detection methods employed are computationally low-cost, ensuring minimal impact on overall performance.Compared to baseline self-consistency, the performance loss is minimal on modern GPUs, with the most computational effort still lying on the initial generation. Additionally, unlike other selfimprovement methods, our techniques don't require an extra pre-training phase and can be applied directly. As advancements in computational resources continue and smaller models grow increasingly capable, we expect this concern to become even less significant.Baseline self-consistency samples of static temperature models often result in deterministic or overly random outputs. We sampled from five different temperatures per generation finding that it provides a wider range of outputs with a more diverse spectrum of answers and performs above average compared to baseline self-consistency. It is to note that higher temperature showed a degree of randomness that can lead to higher degeneration. However this limiting factor can be mitigated when applied with inverse temperature weighting and improve performance of up to 2.5%. The effect of different temperature sets can be found in Appendix N.The process of converting rationales into semantic embedding vectors was applied to multiple featurizer-models at different forms of fine-tuning to measure the ability of models to effectively convert sequences into fitting embedding vectors. The results revealed elevated results for SciBERT and MathBERT [29] when compared to RoBERTa. This is likely due to RoBERTa's general robust training where in contrast, both MathBERT and SciBERT exhibit stronger performance 3 . We conjecture that this is due to their training data being more representative of the reasoning tasks that we evaluate on here [30] . This observation suggests that improper or "unfitting" fine-tuning reduces overall data point density, resulting in a loss of information within the produced vectors, and consequently hindering subsequent marginalization techniques [22] .The implementation of k-means clustering 4 showed that regardless of the fact that reasoning can be improved by detailed mappings, clustering didn't attribute to enhance the quality of the semantic evaluation. Additionally we reason this to be attributed to two limiting factors: We experimented with a spectrum of values for the parameter k, with a significant emphasis on k=2 to ensure that the clusters would still provide a sufficient amount of associated rationales with each cluster to utilize the effect of self-consistency. This method implies that the predictions associated with the majority cluster are the ones for which the model exhibits the greatest overall confidence. A detailed assessment of the found results can be accessed in Appendix M.1.H.1 Rouge-N as a performance measure Contrary to GPT 3.5's performance in terms of accuracy, it under performs in comparison when taking ROUGE metrics into account. As expected it excels in generating accurate, contextually relevant responses but expressed responses more detailed in a more comprehensive fashion, leading to lower ROUGE scores due to the strictly accurate less extensive rationale annotated in the dataset. [16] The other Models like Llama 2 7B and Mistral 7B produce higher scores. This might be related to factors like style of writing and higher text length which although it leads to more comprehensive embeddings lowers it's score when compared with a metric like Rouge-N as visible in Table 3 LLAMA GPT 3.5 GPT-4o miniMistral LLAMA2-7B The low accuracy and poor results, coupled with a degree of randomness in the result distribution, indicate challenges in effectively correlating text using N-Grams. We experimented with different values of 'n' for N-Grams, aiming to improve performance, but encountered limitations. As depicted in the table, the effectiveness of N-Grams varied, suggesting that the pure similar wording in rationales cant be utilized in an effective way to improve or even stably perform similar to the baseline. Higher values of "n" consecutively worsened results.Our analysis indicates that maintaining a minimum of 7-10 responses is crucial to achieving consistent performance comparable to the baseline. When k is set to lower values, the performance gains diminish, sometimes leading to completely random results with low accuracy. We expect that increasing the number of responses could enhance the effectiveness of our methods, leveraging the additional context and range of responses available to each weighting mechanism for improved accuracy.In the k-nearest neighbor (KNN) algorithm, parameters such as the number of neighbors (n_neighbors), the distance metric (metric), and the algorithm used for computing nearest neighbors were varied. The best-performing configuration in terms of accuracy was found with n_neighbors set to 5, using the euclidean metric using the ball_tree algorithm and a threshold of 90% that concluded to an averaged accuracy of 56.18% with all Models and Datasets.For the Isolation Forest, the grid search varied parameters including the number of estimators (n_estimators), the contamination factor, and the max samples size. The configuration yielding the highest accuracy utilized n_estimators=200, contamination=auto, and max_samples=auto with an performance of 58.56% averaged across all Models and Datasets.In the case of Support Vector Machines (SVM), the kernel type (kernel), the regularization parameter (nu), and the gamma value were among the parameters adjusted. The most accurate results were achieved with a linear kernel, nu set to 0.01, and gamma set to scale. Configurations may deviate slightly on GPT 3.5 & GPT4o-mini due to usage via the public API.We used a default of 250 max-new-tokens across all models on SVAMP, due to the complexity and length of sequences on AQuA-RAT we chose 400 max-new-tokens. Humaneval is known to cause degeneration after given stopwords, to limit potential faulty generation of new tokens to we set max new tokens to 400. To ensure long enough reasoning chains we limited the generation on StrategyQA to 450 tokens.We tested our theory of abstraction on a variety of temperature sets and found that set 1 exhibits the best balance between diversity and correctness in our examples. Therefore, it outperforms the other proposed sets.All other experiments have been conducted on a static temperature of 0.8 to aid reproducibility and comparability between results and effects of the employed mechanisms.We propose several methods for weighing sequences from different temperatures. Additionally, we employ a weighing system based on the inverse of the applied temperature. Furthermore, we conducted tests using weighted squared inverse weighting on a small subset. However, these tests did not yield substantially elevated results and performed on a similar margin. 10 : Each Temperature is tested on 1/5 of the samples per generation, to ensure an even distribution. Previous work in self-consistency indicated that chain-of-thought yielded the most favorable outcomes both in terms of accuracy and employed reasoning path. This strategy aligned well with the specific requirements and objectives of a long and sensical reasoning path. Therefore, we chose not to employ other prompting strategies as they did not match the required qualities as observed with chain-of-thought prompting.The used 8-Shot prompt for mathematical reasoning follows the example provided in pg. 43 and use the on pg. 36 referenced set for AQuA on the AQuA-RAT dataset with of the original chain-of-thought paper.Q: There are 15 trees in the grove. Grove workers will plant trees in the grove today. After they are done, there will be 21 trees. How many trees did the grove workers plant today? A: We start with 15 trees. Later we have 21 trees. The difference must be the number of trees they planted. So, they must have planted 21 -15 = 6 trees. The answer is 6.Q: If there are 3 cars in the parking lot and 2 more cars arrive, how many cars are in the parking lot? A: There are 3 cars in the parking lot already. Q: Shawn has five toys. For Christmas, he got two toys each from his mom and dad. How many toys does he have now? A: He has 5 toys. He got 2 from mom, so after that he has 5 + 2 = 7 toys. Then he got 2 more from dad, so in total he has 7 + 2 = 9 toys. The answer is 9.Q: There were nine computers in the server room. Five more computers were installed each day, from Monday to Thursday. How many computers are now in the server room?A: There are 4 days from Monday to Thursday. 5 computers were added each day. That means in total 4 * 5 = 20 computers were added. There were 9 computers in the beginning, so now there are 9 + 20 = 29 computers. The answer is 29.Q: Michael had 58 golf balls. On Tuesday, he lost 23 golf balls. On Wednesday, he lost 2 more. How many golf balls did he have at the end of Wednesday? A: Michael initially had 58 balls. He lost 23 on Tuesday, so after that he has 58 -23 = 35 balls. On Wednesday, he lost 2 more, so now he has 35 -2 = 33 balls. The answer is 33.Q: Olivia has $23. She bought five bagels for $3 each. How much money does she have left? A: She bought 5 bagels for $3 each. This means she spent 5 * $3 = $15 on the bagels. She had $23 in the beginning, so now she has $23 -$15 = $8. The answer is 8. Our generation on Humaneval was conducted 0-shot using just the raw prompt given by the dataset.We selected the datasets that are commonly used in similar methods such as baseline self-consistency [33] and related work to simplify reproduction and comparison to ensure consistency in our results.We use the recommended configuration splits for testing as suggested by default for each dataset. For AQuA-RAT, our test set includes the full set of 254 examples. In the case of StrategyQA, we employ the complete test split, which consists of 687 samples. Specifically, for SVAMP, we utilize the train and test split comprising 1,000 samples to achieve a less noisy evaluation.Across our study we employed kmeans to cluster datapoints mapped by our featurizer model.Clustering has shown diminishing returns in terms of accuracy, however the herein provided evidence shows that clustering with k-means provides a notable advantages which even tho the accuracy was low can be used as a diagnostic tool and similarity measureWe used the silhouette score to evaluate clustering effectiveness. This score measures how similar an object is to its own cluster compared to other clusters, ranging from -1 to 1.Our obtained averaged silhouette score equals 0.41, suggesting a moderate level of distinction between clusters. This range indicates that, on average, objects within a cluster are closer to each other than to objects in other clusters, but the separation is not highly distinct.This finding suggests that clusters are indicating a clear structure in sentence and wording of results and due to Kmeans nature perform better on higher sample sizes.Despite the fragility shown during evaluation on benchmarks, the k-means accurately categorizes the majority of correct answer within the preponderant cluster, not only based on cluster size. This implies that the method, even with limited data, captures essential patterns effectively.High-performing models are more likely to adhere closely to the chosen method. This is because when most answers are correct, there's a lower chance of incorrect responses outweighing the correct ones, which could lead to inaccuracies.The shown results indicate a trend demonstrating that the selected cluster is more likely to feature the majority of correct responses, with an average of 60.5%.We witness the same strides towards higher sample sizes with the usage of k-means as already conveyed in the original self-consistency paper, here larger sample sizes might be able to capture the amount of correct answers in a more favorable manner due to their enabled potential for a higher number of clusters, capturing more nuanced and subtle variations rather than the broad range of responses.The primary cluster and the ostensibly weaker, later-disregarded cluster exhibit comparable performance in terms of the average distance of the data points to its subsequent cluster centroid. Higher temperature in generative models introduces a degree of randomness that can negatively impact performance by increasing degeneration in model outputs. However, this limiting factor can be partially mitigated through techniques such as inverse temperature weighting. When applied perplexity = 2 perplexity = 7 appropriately alongside temperature variation. The benefits of higher temperature are not monotonicbeyond an optimal level, continuing to increase temperature will again degrade performance. There exists a sweet spot where judiciously elevated temperature and re-weighting allows models to produce greater diversity without excessive degradation which we found to lay between t = 0.5 and t = 0.9.Dimensionality reduction did improve performance in edge cases, but it should not be relied upon for consistent results and was generally unstable. We recommend that our methods be used without any additional reduction to ensure more reliable and consistent outcomes. [26, 11, 15] O.In our scenario, while PCA might be better under very specific random circumstances, such as when linear relationships dominate the data, t-SNE is generally superior for visualization. t-SNE excels in revealing complex structures and patterns by capturing local relationships, making it more suitable for understanding the data visually.NeurIPS Paper ChecklistJustification: Our paper is conform to the given guidelines. Nonetheless we discuss ethical considerations and risks in the main text.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).Tested on arithmetic samples only, due to their greater variability and problem-solving scope compared to the more logic-bound and less varied nature of coding tasks and QA tasks.Averaged over 10 random states to ensure an representative example. 


Decision: The work presents an interesting technique to improve the reasoning abilities of LLMs and will be of interest to attendees of the workshop.

Reviews: review: title: title: Official Review
main_review: main_review: ### Summary
This work proposes consistency and filtering mechanisms in the intermediates steps of reasoning steps of LLMs.


### Strengths

1. Strong indication of performance improvement.
2. The paper presents the motivation behind considering different weighting metrics and anomaly detection algorithms.

### Weaknesses

1. The paper misses some related developments in the weighting and filtering of LLM generation literature (Jiang, Chunyang, et al.).
2. The analysis only focuses on performance, and so the trade-offs (time or memory or API cost) are unclear when compared to baselines.


Jiang, Chunyang, et al. "Importance Weighting Can Help Large Language Models Self-Improve." arXiv preprint arXiv:2408.09849 (2024).

Minor:
Broken latex in Appendix D.
title: title: Promising Weighting Mechanism for Self-Consistency Framework but Need More Clarification
main_review: main_review: This paper introduces a novel approach to enhance reasoning in large language models through semantic similarity weighting and outlier detection. The contribution is promising, as it aims to improve reasoning coherence across multiple generated chains, but there are notable areas for improvement.

Advantages
- Innovative Weighting Method: The idea of weighting chains based on their semantic similarity is novel and potentially impactful for improving complex reasoning.
- Evaluation Across Multiple Datasets: The empirical results provide evidence of improved reasoning on datasets like AQuA-RAT and StrategyQA.
- Clear Methodology: The paper provides a step-by-step description of the proposed approach, making it easier for readers to understand the experimental setup, semantic weighting techniques, and evaluation metrics.

Areas for Improvement
- Inconsistent Performance Gains: While the proposed methods show improvement in some cases, results are inconsistent (e.g., CPW shows a drop on StrategyQA for GPT-4o mini). It would be better if the paper could address this by ablation studies or more explanation of these observations.
- Diversity vs. Coherence Trade-Off: The emphasis on semantic similarity may penalize diverse reasoning paths that reach the same conclusion from different angles. The paper would be more solid it it can explore this trade-off and its impact on final predictions. 
- Lack of Interpretability: Since semantic embeddings may not perfectly capture reasoning, adding visualizations or case studies of how different reasoning paths are weighted would enhance transparency and help readers understand the method’s decision-making process.
title: title: Official review for Enhancing Language Model Reasoning via Weighted Reasoning in Self-Consistency
main_review: main_review: Summary:
The paper introduces a new approach extending the self-consistency work of Wang et al by looking at the semantic consistency using multiple approaches between different rationales generated on the same input. The author(s) suggest utilizing outlier removal algorithms before taking a final majority vote on the model's response. The approach, while producing useful results in the specific limited context that the author(s) have tested in, lacks generalizability and could do with deeper investigations around the methodology itself as well as the candidate LLMs.

Strengths:
1. The author(s) propose a novel approach of analyzing the semantic relationships and similarities between responses generated by an LLM for the same prompts, combining Chain-of-thought reasoning with word embeddings.
2. The proposed approach of Semantic Consensus Weighting shows promising results for a subset of mathematical tasks and the results have been presented in an intuitive way.
3. The author(s) acknowledge the limitations of the proposed approaches.

Major concerns:
1. Details : Additional details on the size of the responses generated for each input would be helpful. It can also help with creating more diverse responses.
2. Robustness : Number of responses generated per input seems like an obvious hyper-parameter to try experimenting with to see if the performance of techniques improve.
3. Conclusion : Conclusion lacks a clear recommendation, even if the generalizability of the results is limited. 
4. Clarity : No rationale/ explanation is provided on why some techniques work on some models and do not on others. This would be an important addition to the conclusion section and would help solidify use cases where authors' findings can be valuable.

Minor concerns:
1. Length of the paper exceeds 4 pages

Recommendation:
Author(s) must at least address major concerns before the paper can be accepted.
title: title: It is a strong paper that makes a valuable contribution to enhancing language model reasoning capabilities. The novel techniques are well-justified, the experimental evaluation is rigorous, and the analyses provide useful insights
main_review: main_review: Quality:

This paper presents a solid extension to the self-consistency framework for improving language model reasoning. The authors propose two main techniques - Centroid Proximity Weighting (CPW) and Semantic Consensus Weighting (SCW) - to incorporate semantic analysis of reasoning paths in addition to final decisions. The methodology is well-explained and the experimental setup covers multiple datasets and models.

The quality of the work is good, with a clear motivation, well-designed experiments, and thorough analysis of results. The authors provide extensive ablation studies and additional analyses in the appendix, demonstrating rigorous investigation. 

Clarity:

The paper is clearly written overall and follows a logical structure. The methodology section explains the proposed techniques in sufficient detail. The results are presented clearly in tables and analyzed thoughtfully. 

Some minor points could be clarified further:

More explanation of why certain datasets were chosen would be helpful.

The distinction between CPW and SCW could be explained more explicitly early on.

Some of the mathematical notation in Section 4 could benefit from more explanation

However, these are minor issues and overall the paper communicates the key ideas and findings effectively.

Originality:

The paper presents an original extension to the self-consistency framework by incorporating semantic analysis of reasoning paths. While building on existing work, the proposed techniques of CPW and SCW are novel contributions. The idea of using embedding vectors to analyze and weight reasoning paths is innovative in this context. The exploration of outlier detection methods for filtering responses is also an original angle.

Significance:

This work makes a meaningful contribution to improving language model reasoning capabilities. The proposed techniques show consistent improvements over baseline self-consistency across multiple datasets and models. The gains are particularly notable on more complex reasoning tasks like StrategyQA.
The paper provides useful insights into the effectiveness of semantic analysis for enhancing reasoning, which could inspire further work in this direction. The extensive ablation studies and analyses offer valuable findings for researchers working on language model reasoning.

Pros:

Novel techniques (CPW and SCW) for incorporating semantic analysis of reasoning paths
Consistent improvements over baseline self-consistency across multiple datasets/models
Thorough experimental evaluation and ablation studies
Insightful analysis of results and failure cases
Extensive additional studies and analyses provided in appendix

Cons:

Could provide more motivation/context for choice of datasets
Some explanations of techniques could be clearer/more detailed
Limited exploration of potential negative impacts or ethical considerations
Lacks comparison to some other recent techniques for improving LM reasoning

Overall, this is a strong paper that makes a valuable contribution to enhancing language model reasoning capabilities. The proposed techniques are novel and well-justified, the experimental evaluation is thorough, and the analyses provide useful insights. While there are some minor areas for improvement in clarity and broader impact discussion, the core technical contribution is solid and significant.