Unlocking Structured Thinking in Language Models with Cognitive Prompting

## Abstract

Abstract We propose cognitive prompting as a novel approach to guide problem-solving in large language models (LLMs) through structured, human-like cognitive operations such as goal clarification, decomposition, filtering, abstraction, and pattern recognition. By employing systematic, step-by-step reasoning, cognitive prompting enables LLMs to efficiently tackle complex, multi-step tasks. We evaluate the effectiveness of cognitive prompting on Meta’s LLaMA models, comparing performance on arithmetic reasoning tasks using the GSM8K dataset and on commonsense reasoning benchmarks. Our analysis includes comparisons between models without cognitive prompting, models with a static sequence of cognitive operations, and models using reflective cognitive prompting, where the LLM dynamically self-selects the sequence of cognitive operations. The results show that cognitive prompting, particularly when dynamically adapted, significantly improves the performance of larger models, such as LLaMA3.1 70B, and enhances their ability to handle multi-step reasoning tasks. This approach also improves interpretability and flexibility, highlighting cognitive prompting as a promising strategy for general-purpose AI reasoning.

## 1 Introduction

Recent advancements in artificial intelligence (AI), especially with large language models (LLMs), have made great progress in emulating human reasoning to solve tasks like text summarization (Stiennon et al., ), code generation (Guo et al., ), and question answering (Lu et al., ). While LLMs excel at generating coherent text and handling vast data, their ability to perform multi-step reasoning still falls short of human cognitive processes. Human cognition, marked by its structured nature, provides a compelling blueprint for guiding AI through complex tasks that require layered thinking and adaptability.

Figure: Figure 1: Left: General cognitive prompting, Right: Cognitive prompting adapted to arithmetical reasoning.
Refer to caption: x1.png

This paper introduces a novel approach called cognitive prompting, designed to enhance problem-solving in LLMs by systematically emulating human cognitive operations (COPs). Cognitive prompting organizes problem-solving into distinct cognitive steps—such as goal clarification, task decomposition, and pattern recognition—allowing LLMs to tackle complex tasks in a more structured and interpretable manner, see Figure 1. Inspired by cognitive psychology and cognitive architectures like ACT-R (Anderson & Lebiere, ), this method bridges the gap between human-like reasoning and AI’s computational power, enabling models to handle tasks in fields such as mathematics, logic, decision-making, and creativity with greater precision.
Our experiments, conducted with Meta’s LLaMA models (Touvron et al., ) on the GSM8K (Cobbe et al., ) and a commonsense benchmark (Hendrycks et al., ), demonstrate significant improvements in task performance when cognitive prompting is applied. In particular, the reflective variant of cognitive prompting leads to enhanced reasoning capabilities.

The structure of the paper is as follows: Section 2 introduces the concept of cognitive prompting, detailing its core operations and their application in problem-solving. Section 3 presents experimental results on the impact of cognitive prompting on arithmetic reasoning tasks, while Section 4 explores its effectiveness in commonsense reasoning.
Section 5 reviews related work on prompting engineering strategies. Finally, Section 6 concludes the paper.
The appendix contains exemplary reasoning processes and examples for problem-specific COPs.

## 2 Cognitive Prompting

Cognitive prompting organizes problem-solving through a structured sequence of human-like COPs, enabling LLMs to tackle complex tasks across domains such as mathematics, logic, creativity, and decision-making. This method, inspired by principles in cognitive psychology, breaks problems into stages like goal clarification, decomposition, filtering, and integration—mimicking the way humans refine their understanding of tasks. By leveraging this structured approach, cognitive prompting enhances clarity, interpretability, and adaptability in LLM reasoning.

Unlike methods like Chain of Thought (CoT) (Wei et al., ), cognitive prompting offers more general multi-dimensional operational depth, allowing LLMs to approach a wider variety of problems with reasoning progression. This framework, rooted in dual-process and problem-space theories, encourages both intuitive and analytical reasoning, helping models transition between pattern recognition, abstraction, and integration for more consistent and interpretable solutions.
Cognitive prompting can be formalized as an optimization problem. Let $C=\{c_{1},c_{2},\dots,c_{n}\}$ represent a set of COPs and $S=\{s_{1},s_{2},\dots,s_{k}\}$ denote a sequence of $k$ operations from $C$. The objective is to find the sequence $S^{*}$ that maximizes task performance $S^{*}=\arg\max_{S\subseteq C}f(S)$ subject to constraints such as $|S|=k$, $s_{1}=\text{goal clarification}$, and $s_{k}=\text{integration}$. Here, $f(S)$ represents task performance, e.g., accuracy, efficiency, coherence.

Cognitive prompting follows a structured process that mirrors human problem-solving. Key COPs include:

### Goal Clarification:

Clearly define the objective of the problem to maintain focus on solving it effectively. In the context of COP, goal clarification ensures that the model aligns its reasoning with the desired outcome, minimizing distractions. Let G represent the goal, and all subsequent operations should be oriented toward achieving G , helping the model concentrate on the correct direction of reasoning.

### Decomposition:

Break down the problem $P$ into smaller, manageable components $\{P_{1},P_{2},\dots,P_{n}\}$, where $P=\bigcup_{i=1}^{n}P_{i}$. This step is crucial in COP as it allows the model to tackle complex, multi-step problems incrementally. Decomposition is particularly useful in mathematical problem-solving and logic tasks, where breaking a problem into sub-problems allows the model to apply specific operations or strategies to each part. Moreover, decomposition helps to identify the core structure of the problem, isolating the critical steps required for a comprehensive solution.

### Filtering:

Select the most relevant information from $I=\{i_{1},i_{2},\dots,i_{m}\}$ using a filtering function $F(I)=I_{\text{rel}}\subseteq I$. Filtering is essential in COP to prevent the model from being overwhelmed by irrelevant details. In complex tasks, the problem statement may include redundant or distracting information, so filtering ensures that the model focuses on the essential data points that directly impact problem-solving. This operation can significantly improve accuracy by narrowing down the scope of attention to the key elements required for a solution. Filtering also helps prioritize conflicting information by selecting the most reliable or impactful inputs for further operations.

### Reorganization:

Rearrange data, variables, or equations $D$ to reveal patterns or simplify the structure, such that $\textit{Reorder}(D)\to D^{\prime}$. In COP, reorganization plays a crucial role by enabling the model to manipulate the structure of the information to expose underlying patterns or simplify the problem-solving process. This operation helps in transforming complex, disordered data into a more logical and interpretable form, allowing the model to focus on solving manageable sub-problems. Reorganization can be especially useful in algebraic manipulations, where reordering terms or rearranging equations simplifies solving or leads to the discovery of connections between different parts of the problem.

### Pattern Recognition:

Identify recurring relationships or patterns $\mathcal{P}$ in the data, which facilitates the application of known solutions. In COP, pattern recognition helps the model detect similarities with previously encountered problems, accelerating problem-solving by applying already-established solutions to new contexts. Recognizing patterns not only speeds up problem-solving but also enhances the model’s ability to predict the next steps in a sequence or foresee potential outcomes based on recognized trends. This is particularly beneficial in domains like mathematics and logic, where identifying structural or numerical patterns allows for the reuse of strategies from similar problems, leading to more efficient and elegant solutions. Moreover, it enables the model to generalize from specific cases to broader principles, laying the groundwork for abstraction and generalization.

### Abstraction:

Extract broader principles $A$ from the identified patterns $\mathcal{P}$, and generalize them to apply across different problems or contexts. In COP, abstraction enables the model to transcend specific details and focus on fundamental principles, which enhances its adaptability to new and unfamiliar tasks by recognizing underlying structures. Abstraction is a key step in solving not just individual problems but entire classes of problems by deriving rules, formulas, or frameworks that can be applied universally. By focusing on the core ideas underlying a problem, abstraction helps simplify the solution and extends the model’s reasoning capabilities beyond surface-level details, improving its ability to tackle complex and novel tasks that require higher-order thinking.

### Generalization:

Apply abstracted principles $A$ to the broader problem or similar contexts, such that $f_{\text{gen}}(A)=\{P_{1},P_{2},\dots,P_{k}\}$. Generalization in COP ensures that solutions are not isolated to the specific instance but are scalable across various related problems. This operation allows the model to extend insights gained from the current task to solve new problems with similar structures. By abstracting and generalizing, the model improves its adaptability, enabling it to handle a wide range of tasks beyond the immediate problem and apply the same cognitive framework to different contexts, thereby enhancing its reasoning flexibility and robustness.

### Integration:

Synthesize the individual solutions $Q_{i}$ into a cohesive final solution $Q$, ensuring all components of the problem are addressed and fit together logically. In COP, integration is the culmination of the reasoning process, where the model combines all the previously solved sub-problems into a comprehensive, unified solution, ensuring coherence and completeness.

### Static and Reflective Cognitive Prompting:

This flexible process allows LLMs to dynamically apply the most relevant operations based on the task’s context, enhancing problem-solving performance across various domains. In static cognitive prompting, a fixed order $S=[s_{1},s_{2},\dots,s_{k}]$ of COPs is followed throughout the problem-solving process, ensuring a structured yet rigid approach. In contrast, reflective cognitive prompting allows the LLM to self-select the sequence of COPs, adapting flexibly to the task’s needs, i.e., choosing the next COP $s_{i}\in C$ in each step. This adaptability not only improves the model’s ability to solve complex problems but also offers structured, interpretable explanations of the reasoning processes.

### Domain Specific COPs

The effectiveness of cognitive prompting is significantly enhanced when the general COPs are adapted to specific problem domains. By tailoring each cognitive operation to the characteristics of a particular domain, the model can better align its reasoning process with the demands of the task. For example, the decomposition operation in scientific inquiry might involve breaking down a complex hypothesis into smaller, testable components, while in ethical decision-making, decomposition could involve identifying and separating conflicting moral principles and stakeholder interests. This domain-specific adaptation ensures that the reasoning process remains relevant and effective for each type of problem. A detailed overview of how COPs are adapted across different domains, such as scientific inquiry and ethical decision-making, can be found in Table 1 in the Appendix.

## 3 Arithmetic Reasoning

### Benchmark

We evaluate the performance of cognitive prompting with Meta’s LLaMA models (8B and 70B) on the GSM8K dataset (Cobbe et al., ), a widely used benchmark for math problem-solving. GSM8K consists of about 7k training and 1.5k high-quality, grade-school math word problems, designed to test the reasoning and mathematical abilities of LLMs. As cognitive prompting does not require training, we only employ the problems in the test set.

### COPs

The general COPs are adapted to arithmetic reasoning as follows, see Figure 1, right. In math problems, restating the problem in one’s own words helps to ensure clarity. Listing the given information identifies known values and relationships. Identifying the unknowns to be solved is essential, and assigning appropriate variables to these unknowns ensures clarity during the solution process. Defining each variable clearly avoids confusion. Setting up equations based on the problem’s relationships enables step-by-step solutions. Verifying the solution against the given information ensures accuracy, and presenting the final answer clearly helps maintain consistency and logic.

### Results

The 8B model achieves scores of 0.7 across all prompting techniques. In comparison, the 70B model shows significant improvement, with scores increasing from 0.87 (no prompting) to 0.89 (static cognitive prompting) and 0.91 (reflective cognitive prompting), see Figure 2 (left).

Figure: Figure 2: Left: Accuracies of cognitive prompting (CP) strategies and models (3 repetitions) on artichmetic reasoning problems, Right: Occurrence of top nine cognitive prompting sequences in 70B model with goal clarification (GC), decomposition (DC), pattern recognition (PR), generalization (GN), and reorganization (RE).
Refer to caption: x2.png

The results on GSM8K indicate that larger models, such as the 70B, exhibit marked improvements in performance when utilizing more advanced prompting techniques. While the 8B model’s scores remain consistent at around 0.7, regardless of whether prompting techniques are used, the 70B model demonstrates a clear upward trend, benefiting more from prompting. Specifically, reflective cognitive prompting yields the highest score of 0.91, followed by static at 0.89, and no prompting at 0.87. This suggests that larger models are better able to take advantage of prompting techniques, especially Reflective cognitive prompting, which seems to facilitate deeper reasoning or reflection in the model. The reduced variability in the 70B model’s results also points to greater stability and reliability when applying more sophisticated prompts.

Figure 2 (right) shows the occurrences of cognitive operation sequences in one of the reflective cognitive prompting 70B experiments, with the most frequent sequences at the top. Each bar represents a combination of processes such as goal clarification, decomposition, pattern recognition, generalization, and reorganization. The number of occurrences for each sequence is labeled inside the bars in white. The plot presents the data in descending order, from the most common to the least frequent cognitive operation sequences.
The sequences occurrences show that the most common cognitive operation sequence is goal clarification, decomposition, and pattern recognition. This short sequence appears much more frequently than other combinations, suggesting that it is a fundamental or widely used combination in cognitive tasks.
Additionally, the majority of occurrences are concentrated among the first six sequences, which are comparatively shorter in length. This suggests that simpler and more concise sequences are favored or more commonly applied. Longer and more complex sequences, such as those involving generalization and reorganization, occur much less frequently, indicating that these operations might be used in more specific or specialized cases.

## 4 Commonsense Reasoning

To further assess the versatility of cognitive prompting, we tested its effectiveness on a set of commonsense reasoning problems. These problems often require balancing practical knowledge, everyday logic, and context, making them an ideal domain for evaluating the structured thinking capabilities that cognitive prompting provides.

### Benchmark

We evaluate the cognitive prompting approach using both sizes of LLaMA3.1 on a randomly selected subset of questions from the commonsense dataset, a component of the ethics problem benchmark (Hendrycks et al., ). Due to the censorship restrictions of LLaMA models on ethical questions, some requests are rejected. Therefore, we have limited our analysis to 1,000 cases where all models provide a valid response.

Figure: Figure 3: Commonsense reasoning prompts used for cognitive prompting.
Refer to caption: x3.png

### COPs

The cognitive operations are adapted to handle commonsense reasoning tasks by guiding models through structured problem-solving steps, see Figure 3. Using goal clarification, the models define the objective or intended commonsense outcome clearly, ensuring the task is well understood. Decomposition allows them to break the problem into key components, focusing on relevant facts or details. Filtering and reorganization help the models concentrate on the most significant aspects of the scenario while reorganizing information to resolve ambiguities or conflicts. Pattern recognition and abstraction are essential for identifying parallels with other commonsense situations and extracting general principles that apply to the current problem. Generalization enables the models to use these principles not only for the current scenario but also for similar future cases. Integration brings together all perspectives and principles to form a cohesive and logical commonsense conclusion. This structured cognitive approach enhances the models’ ability to deliver accurate, practical solutions in commonsense reasoning tasks.

### Results

Figure 4 (left) illustrates that static cognitive prompting outperforms the absence of cognitive prompting, while reflective cognitive prompting further improves performance over static in the 8B model. The 70B model consistently outperforms the 8B model.
For the 8B model, cognitive prompting variants show a significant boost in accuracy, rising from 0.605 without prompting to over 0.74 with cognitive prompting. Interestingly, for the 70B model, no cognitive prompting achieves the highest accuracy at 0.84, slightly outperforming reflective cognitive prompting at 0.81. Upon further analysis of the models’ outputs, we found that the larger model tends to over-process multiple reasoning steps, leading to errors when too many steps are chosen—an effect resembling overfitting. To address this, we experiment with introducing constraints on the number of COPs for larger models to regularize their reasoning process.

Figure: Figure 4: Left: Accuracies of cognitive prompting (CP) strategies and models (2 repetitions for 8B, 1 for 70B) on commonsense reasoning problems, Right: Occurrence of top nine and other cognitive prompting sequences in 70B model with abbreviations like in Figure 2 (right) and filtering (FI), abstraction (AB), reasoning (RS), and integration (IN).
Refer to caption: x4.png

Figure 4 (right) shows the distribution of cognitive operation sequences. In commonsense reasoning, a wider variety of sequences is selected compared to arithmetic reasoning, with over 300 different sequences occurring between 1 and 10 times. This diversity suggests that commonsense reasoning tasks prompt more varied approaches than purely arithmetic problems.

## 5 Related Work

Prompting is a key technique for leveraging pre-trained LLMs to perform tasks by guiding their outputs through well-crafted instructions. In zero-shot prompting, models generate responses without task-specific examples, while few-shot prompting (Brown et al., ) improves performance by including a few task examples.
CoT prompting (Wei et al., ) breaks down complex reasoning into intermediate steps, enabling systematic problem-solving, while Tree of Thoughts (ToT) (Yao et al., ) extends CoT by enabling LLMs to explore multiple reasoning paths and make deliberate decisions. Building on CoT, ReAct (Yao et al., ) combines reasoning with real-time decision-making, enhancing models’ abilities to handle dynamic tasks. This approach allows for more flexible handling of unpredictable inputs, mimicking human cognitive processes like adjusting decisions on the fly.

Prompt Breeder (Fernando et al., ) optimizes prompts using evolutionary computation to iteratively refine and improve performance. Similarly, self-consistency (Wang et al., ) enhances reliability by generating multiple responses and selecting the most consistent one, reducing variability in complex tasks. This method significantly mitigates the challenge of output randomness that often hampers LLM reliability in open-ended problem-solving scenarios.

Automated Prompt Engineering (APE) (Zhou et al., ) automates prompt optimization through model self-instruction and feedback loops, pushing the boundaries of human-computer collaboration. Optimization by PROmpting (OPRO) (Yang et al., ) uses LLMs to iteratively generate and refine solutions, significantly outperforming human-designed prompts in optimization tasks. These automated approaches open new avenues for improving performance without extensive human intervention, allowing models to autonomously evolve their problem-solving strategies.

Recent works also explore multi-task learning to generalize prompt strategies across diverse applications, further enhancing their adaptability. Techniques like retrieval-augmented generation (RAG) (Lewis et al., ) combine prompting with external knowledge sources, offering richer context and better-informed outputs, demonstrating how prompts can evolve to integrate more human-like reasoning.
Recent advancements in parameter-efficient fine-tuning methods, such as decomposed prompt tuning (DePT) (Shi & Lipani, ), have demonstrated how efficient prompt-based strategies can reduce memory and computational costs in large language models, which can complement the flexibility provided by cognitive prompting in adapting models to complex problem-solving tasks.
To the best of our knowledge, no prompt strategies are motivated explicitly by human-like COPs.

## 6 Conclusions

Cognitive prompting models human reasoning as a sequence of COPs delivered through prompts. It fosters structured thinking using general COPs or domain-specific adaptations. Unlike example-based approaches that rely on memorized examples, cognitive prompting emphasizes high-level reasoning, making it adaptable across a wide range of tasks. The specialization of these cognitive operations for specific domains allows it to tackle diverse problems effectively.
Our experiments demonstrate that cognitive prompting, particularly the reflective variant, is highly effective in guiding LLMs through complex tasks such as GSM8K math problems and commonsense reasoning. Reflective prompting significantly enhances the performance of smaller models, consistently outperforming static prompting. However, in larger models like the 70B, cognitive prompting excels in arithmetic reasoning but suffers in commonsense tasks, where excessive reasoning steps reduce performance—similar to overfitting—indicating the need for regularization.
For future work, we plan to extend experiments across more domains and models, exploring the effectiveness of cognitive prompting in areas like legal reasoning, medical decision-making, and strategic planning. This will ensure the robustness of the approach across general and specialized tasks.

## Reproducibility Statement

Our experiments use Meta’s LLaMA models, which are open-source and accessible. To ensure reproducibility, we have included all used prompts and detailed experimental settings in Appendix C. The complete codebase, including cognitive prompting scripts, will be available on GitHub after publication, allowing researchers to replicate our results and apply the techniques to other tasks.

## Ethics Statement

Cognitive prompting promotes structured, human-like reasoning, enhancing transparency and consistency. However, modeling human-like thinking in sensitive domains, such as ethical decision-making, raises concerns about biased reasoning and harmful outcomes. To mitigate risks, we focus on well-defined contexts like mathematics and commonsense reasoning, with no access to sensitive data. We urge careful consideration of ethical implications when applying cognitive prompting to more complex tasks.

## Appendix A Arithmetic Reasoning Example

Here is an example for arithmetic reasoning using static cognitive prompting and a LLaMA3.1 70B answer:

## Appendix B Commonsense Reasoning Example

Here is an example for commonsense reasoning using reflective cognitive prompting and a LLaMA3.1 8B answer:

## Appendix C List of Problem-Specific Cognitive Operations

**Table 1: Exemplary specialization of COPs for various problem domains.**
| General Cognitive Operation | Creative Problem Solving |
| --- | --- |
| Goal Clarification | Clarify the Creative Challenge |
| Decomposition | Break the Challenge into Parts |
| Filtering | Focus on Key Constraints |
| Reorganization | Explore New Perspectives |
| Pattern Recognition | Identify Creative Patterns |
| Abstraction | Develop Broad Concepts |
| Generalization | Test and Refine Ideas |
| Integration | Synthesize Novel Solutions |
| Decision-Making | Scientific Inquiry |
| Define the Decision Objective | Formulate the Research Question |
| Break Decision into Factors | Break Research into Sub-Questions |
| Focus on Critical Information | Identify Key Variables |
| Arrange Alternatives | Plan the Experiment |
| Identify Patterns in Choices | Look for Patterns in Data |
| Extract General Principles | Develop Theoretical Insights |
| Test Against Criteria | Apply Findings Broadly |
| Make a Final Decision | Form Conclusions |
| Strategic Planning | Ethical Problem-Solving |
| Define the Strategic Objective | Clarify the Ethical Dilemma |
| Break Strategy into Steps | Break Dilemma into Components |
| Prioritize Focus Areas | Focus on Pressing Issues |
| Arrange Steps Logically | Consider Different Perspectives |
| Identify Strategic Trends | Identify Similar Cases |
| Formulate High-Level Plans | Develop Ethical Principles |
| Test Strategies Against Scenarios | Evaluate Solutions Against Principles |
| Develop a Cohesive Plan | Make a Final Ethical Judgment |
| Math Problem-Solving | Logical Problem-Solving |
| Restate the Problem in Your Own Words | Restate the Logical Problem Clearly |
| List the Given Information | Break Problem into Key Logical Clues |
| Identify What You Need to Find | Focus on the Most Critical Clues |
| Assign Variables to the Unknowns | Organize Information Logically |
| Define Each Variable Clearly | Identify Logical Deductions |
| Set Up Equations Based on the Problem | Generalize Rules or Inferences |
| Solve the Equations Step by Step | Test Inferences Against Remaining Clues |
| Verify Your Solution with the Given Information | Synthesize a Complete Solution |
| Provide a Clear and Direct Answer | Provide the Final Answer |

## Peer Reviews

confidence: 4
contribution: 1
initial_rating: 3
presentation: 3
questions: none.
soundness: 3
strengths: Potentially interesting idea.
summary: In the paper, the authors suggest prompting LLMs based on human cognitive operations for guiding problem-solving.
weaknesses: The paper outlines an interesting idea, however, it
1) does not sufficiently qualify major differences in the considered/realised key COPs, compared to related work, including the various contributions of in-context learning, instruction/demonstration finetuning, and CoT prompting. As an arbitrary example: How is filtering (FI) differently realised and investigated as in previous approaches?

Furthermore, the current version insufficiently studies the idea. The results seem too coarse for understanding and valuing significant differences in related work. In particular, the following points should be considered:
2) The quantitative results both in sec 3 and 4 need comparisons with baselines, e.g. finetuning and CoT prompting.
3) For the universality of the results, other models, both open and proprietary, need to be investigated as well, such as Mistral/Bloom and GPT/Gemni/Claude.
4) Qualitative results are missing, thus particular representative cases for good and bad accuracies of the key COPs. 

Therefore, I do not see a significant and novel contribution to ICLR.

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 2
questions: Are these cognitive operations handcrafted? How does it transfer to other questions?
soundness: 2
strengths: 1. This paper is well-written. The explanation of each component in cognitive prompting is clear.  
2. This paper makes LLMs equipped with cognitive operations, which is insightful for exploring the thinking modes of LLMs and future prompting design.  
3. This paper exhibits various cognitive processes of different questions. In experiments, it counts the frequency of each combination of operations, which can reflect the internal thought process of LLMs.
summary: This paper proposes cognitive prompting to enhance the structured thinking capabilities of LLMs. The prompting method offers cognitive operations such as goal clarification, decomposition, abstraction, among others. Utilizing cognitive prompting, LLMs are capable of human-like thinking to simplify complex questions. This paper also evaluates cognitive prompting using benchmarks in arithmetic reasoning and commonsense reasoning. The results show that cognitive prompting can improve the performance of LLMs.
weaknesses: 1. The experiments are not comprehensive. Only two datasets are used in the paper. It would be better to have more results on datasets of varying difficulties (like MATH [1]).  
2. The gain brought by cognitive prompting is minor and unstable. In arithmetic reasoning, the improvements are relatively limited. In commonsense reasoning, the use of cognitive prompting severely drops performance for the 70B model. 
3. There is a lack of baselines. This paper only compares CP with the vanilla models, but there are also some prompting methods for enhancing reasoning capabilities (Chain of Thought[2], Tree of Thought[3], Reflexion[4]). 
4. This method's effectiveness depends on the design of cognitive operations, which leads to poor generalization.

[1] Hendrycks D, Burns C, Kadavath S, et al. Measuring mathematical problem solving with the math dataset[J]. arXiv preprint arXiv:2103.03874, 2021.
[2]Wei J, Wang X, Schuurmans D, et al. Chain-of-thought prompting elicits reasoning in large language models[J]. Advances in neural information processing systems, 2022, 35: 24824-24837.
[3]Yao S, Yu D, Zhao J, et al. Tree of thoughts: Deliberate problem solving with large language models[J]. Advances in Neural Information Processing Systems, 2024, 36.
[4]Shinn N, Cassano F, Gopinath A, et al. Reflexion: Language agents with verbal reinforcement learning[J]. Advances in Neural Information Processing Systems, 2024, 36.

---

confidence: 2
contribution: 2
initial_rating: 5
presentation: 3
questions: Can the authors provide more detailed descriptions of the implementation of each cognitive operation and the dynamic selection process for reflective cognitive prompting?

How do the authors plan to address the issue of overprocessing/overthinking in larger models?

Are there any plans to validate the effectiveness of cognitive prompting across more diverse domains, e.g., RL decision-making?
For instance see: Momennejad, Ida, et al. "Evaluating cognitive maps and planning in large language models with CogEval." Advances in Neural Information Processing Systems 36 (2024).
soundness: 3
strengths: - The experiments are conducted rigorously, with thorough comparisons and analyses demonstrating the effectiveness of the proposed approach.
- The paper is well-structured and clearly articulates the concept of cognitive prompting, its implementation, and its benefits.
summary: The paper introduces an approach called cognitive prompting to enhance problem-solving abilities in large language models (LLMs) by mimicking human cognitive operations (COPs). These operations include goal clarification, decomposition, filtering, reorganization, pattern recognition, abstraction, generalization, and integration. The authors evaluate the effectiveness of this approach using LlamA models on arithmetic reasoning tasks from the GSM8K dataset and commonsense reasoning benchmarks. They compare models without cognitive prompting, models with a static sequence of cognitive operations, and models using reflective cognitive prompting where the LLM dynamically selects the sequence of operations. Results indicate performance improvements, especially with reflective cognitive prompting in larger models, demonstrating enhanced interpretability and flexibility in problem-solving.
weaknesses: Please see the questions

---

confidence: 5
contribution: 1
initial_rating: 3
presentation: 2
questions: As mentioned before, the idea of organizing problem-solving procedures through a structured sequence is intuitive and well-motivated. However, the method's generalizability needs careful examination. I would encourage the authors to further explore the potential benefits of this approach across diverse datasets and domains.
soundness: 2
strengths: - The approach of organizing problem-solving procedures through a structured sequence is intuitive and well-motivated. However, the method's generalizability needs careful examination.
- The experimental results indicate that designing a list of basic operations and allowing LLMs to select one at each reasoning step effectively enhances performance on certain tasks, such as the GSM8K dataset in mathematics. I would encourage the authors to further explore the potential benefits of this approach across diverse datasets and domains.
summary: This paper introduces a novel prompting method called cognitive prompting, in which a large language model is instructed to select an operation from a predefined operation list at each reasoning step. Experimental results on the GSM8K dataset and a commonsense reasoning benchmark demonstrate the effectiveness of this proposed prompting method.
weaknesses: - The proposed method depends heavily on a human-defined operation list, which might limit the generalizability of the method. 
- The evaluation lacks comprehensiveness, as it uses only one dataset for the arithmetic reasoning task and one benchmark for the commonsense reasoning task. The current experimental results are insufficient to substantiate the claim that "Unlike example-based approaches that rely on memorized examples, cognitive prompting emphasizes high-level reasoning, making it adaptable across a wide range of tasks." To provide a fair comparison with example-based approaches, such as few-shot CoT [1], additional experiments on other datasets should be considered, for example, those used in [1], [2], [3], and [4]. 
- Some experimental details are missing, e.g., 
  - The prompting method corresponding to "no CP" in Figure 2 is not described. 
  - Detailed information about the evaluation dataset for the commonsense reasoning task is not provided. 
- Potential errors in the content:
  - Line 190: the statement "The 8B model achieves scores of 0.7 across all prompting techniques. " appears incorrect, as "8B reflective CP" achieves an accuracy of approximately 0.73. 

References
- [1] Wei et al., Chain-of-thought Prompting Elicits Reasoning in Large Language
  Models, NeurIPS 2022.  
- [2] Kojima et al., Large Language Models Are Zero-shot Reasoners, NeurIPS 2022.
- [3] Yao et al., ReAct: Synergizing Reasoning and Acting in Language Models, ICLR 2023
- [4] Wang et al., Self-Consistency Improves Chain of Thought Reasoning in Language Models, ICLR 2023