Chorus of Thoughts: Exploring Collaborative Reasoning Abilities of Large Language Models
Authors: Musarrat Hussain, Sungyoung Lee, Seong Tae Kim, Sung-Ho Bae, Jung Uk Kim, Lars Ailo Bongo, Brita Elvevåg


Reviewer paper summaries: The paper titled "Chorus of Thoughts: Exploring Collaborative Reasoning Abilities of Large Language Models" introduces a novel approach aimed at enhancing the reasoning capabilities of large language models (LLMs) through collaborative methods. By leveraging multiple LLMs, the study investigates how a collective reasoning strategy, dubbed "Chorus of Thoughts," can improve problem-solving and task comprehension across diverse datasets. The methodology involves multiple stages, including independent reasoning strategy development by each LLM, followed by a collaborative phase where these strategies are shared, refined, and unified into a comprehensive approach. The paper evaluates this methodology against twelve tasks from the Big Bench-Hard test suite, finding that while the collaborative approach shows promise, it often does not surpass the performance of individual models. This counterintuitive result highlights the complexity of effectively integrating multiple LLMs and offers insights into the potential limits and challenges of such collaborative efforts in artificial intelligence research.
Summary of strengths: The introduction of the "Chorus of Thoughts" methodology, which harnesses the collaborative efforts of multiple large language models (LLMs), represents a certain innovation. It attempts to combine the strengths of various models to enhance task understanding and reasoning capabilities.
Summary of weaknesses: The study primarily focuses on a limited set of mostly closed-source models. Including a broader spectrum of LLMs, especially strong open-source models like Llama 3 and Mistral, could provide a more comprehensive understanding of the methodology's effectiveness across different types of LLM architectures. Additionally, experimenting with various combinations of stronger and weaker models could yield insightful data on how mixed-capability ensembles perform on Big Bench-Hard (BBH) tasks. This approach would help in drawing more generalized conclusions about the efficacy of the "Chorus of Thoughts" strategy.
The paper lacks detailed explanation on how the reasoning steps generated by each LLM are coordinated and integrated. A more thorough description of the processes involved in ordering these generative steps would aid in understanding the operational dynamics of the collaborative framework.
While the paper introduces an innovative approach to LLM collaboration, it does not adequately discuss existing literature on similar topics, such as the use of multi-agent systems for improving factuality and reasoning in LLMs.
Comments, suggestions and typos: 1. Add more models in the experiment and try different mix-and-match to draw more general conclusion on "chrous of thoughts".
2. Add detailed explanation on how the reasoning steps generated by each LLM are combined and ordered. 
3. Incorporating a discussion on related works, including studies on multi-LLM collaboration like "Improving factuality and reasoning in language models through multiagent debate," would provide a richer context for the paper's contributions. This discussion should highlight how the proposed methodology compares with, contradicts, or enhances the findings of previous research.
Overall assessment scores: 2.5