# Summary: "Imagining and building wise machines: The centrality of AI metacognition" by Johnson, Karimi, Bengio, et al.

| > We may not know precisely what form wise AI will take—but it must surely be preferable to folly. |
| --- |

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/35f231f8947a13decff1eb1c127f2171fd1ecca678fb9b89.png)

Read the full, original paper

| **Paper Authors**: ------------------ [Samuel G. B. Johnson](https://arxiv.org/search/cs?searchtype=author&query=Johnson,+S+G+B), [Amir-Hossein Karimi](https://arxiv.org/search/cs?searchtype=author&query=Karimi,+A), [Yoshua Bengio](https://arxiv.org/search/cs?searchtype=author&query=Bengio,+Y), [Nick Chater](https://arxiv.org/search/cs?searchtype=author&query=Chater,+N), [Tobias Gerstenberg](https://arxiv.org/search/cs?searchtype=author&query=Gerstenberg,+T), [Kate Larson](https://arxiv.org/search/cs?searchtype=author&query=Larson,+K), [Sydney Levine](https://arxiv.org/search/cs?searchtype=author&query=Levine,+S), [Melanie Mitchell](https://arxiv.org/search/cs?searchtype=author&query=Mitchell,+M), [Iyad Rahwan](https://arxiv.org/search/cs?searchtype=author&query=Rahwan,+I), [Bernhard Schölkopf](https://arxiv.org/search/cs?searchtype=author&query=Sch%C3%B6lkopf,+B), [Igor Grossmann](https://arxiv.org/search/cs?searchtype=author&query=Grossmann,+I) +++ Highlights ---------- > • We examine the why and the how of building wise AI > • Wisdom helps humans to navigate intractable problems through object-level strategies (for managing problems) and metacognitive strategies (for managing object-level strategies) > • Wise AI, through improved metacognition, would be more robust to new environments, explainable to users, cooperative in pursuing shared goals, and safe in avoiding both prosaic and catastrophic failures > • We suggest several approaches to benchmarking wisdom, training wise reasoning strategies, and adapting AI architecture for metacognition ++++++ Abstract -------- > Recent advances in artificial intelligence (AI) have produced systems capable of increasingly sophisticated performance on cognitive tasks. However, AI systems still struggle in critical ways: unpredictable and novel environments (robustness), lack of transparency in their reasoning (explainability), challenges in communication and commitment (cooperation), and risks due to potential harmful actions (safety). We argue that these shortcomings stem from one overarching failure: AI systems lack wisdom. > > Drawing from cognitive and social sciences, we define wisdom as the ability to navigate intractable problems - those that are ambiguous, radically uncertain, novel, chaotic, or computationally explosive - through effective task-level and metacognitive strategies. While AI research has focused on task-level strategies, metacognition - the ability to reflect on and regulate one's thought processes - is underdeveloped in AI systems. In humans, metacognitive strategies such as recognizing the limits of one's knowledge, considering diverse perspectives, and adapting to context are essential for wise decision-making. We propose that integrating metacognitive capabilities into AI systems is crucial for enhancing their robustness, explainability, cooperation, and safety. > > By focusing on developing wise AI, we suggest an alternative to aligning AI with specific human values - a task fraught with conceptual and practical difficulties. Instead, wise AI systems can thoughtfully navigate complex situations, account for diverse human values, and avoid harmful actions. We discuss potential approaches to building wise AI, including benchmarking metacognitive abilities and training AI systems to employ wise reasoning. Prioritizing metacognition in AI research will lead to systems that act not only intelligently but also wisely in complex, real-world situations. ++++++ Notes on this summary --------------------- Some quotes have been reformatted (adding paragraph breaks, dot points, references changed to footnotes, ect.). I've made a few minor edits to the quotes. For example, I remove the word "secondly" when I'm only quoting one element of a list and adjusting the grammar when converting sentences into a list. Whilst I've tried as hard as possible to represent the views in the original (including making some updates in response to feedback from one of the authors), I can't guarantee perfect accuracy. By default, assume that quotes (indicated by a grey bar on the left) are from the paper. +++ |
| --- |

| **Why I Wrote This Summary** ---------------------------- Firstly, I thought the framing of metacognition as a key component of wisdom missing from current AI systems was insightful and the resulting analysis fruitful. Secondly, this paper contains some ideas similar to those I discussed in [Some Preliminary Notes on the Promise of a Wisdom Explosion](https://aiimpacts.org/some-preliminary-notes-on-the-promise-of-a-wisdom-explosion/). In particular, the authors talk about a "virtuous cycle" in relation to wisdom in the final paragraphs: > By simultaneously promoting robust, explainable, cooperative, and safe AI, these qualities are likely to amplify one another: > > * Robustness will facilitate cooperation (by improving confidence from counterparties in its long-term commitments) and safety (by avoiding novel failure modes[^uycflfu5vsl]). > * Explainability will facilitate robustness (by making it easier to human users to intervene in transparent processes) and cooperation (by communicating its reasoning in a way that is checkable by counterparties). > * Cooperation will facilitate explainability (by using accurate theory-of-mind about its users) and safety (by collaboratively implementing values shared within dyads, organizations, and societies). > > Wise reasoning, therefore, can lead to a virtuous cycle in AI agents, just as it does in humans. |
| --- |

* * *

Summary:
========

**What is wisdom?**
-------------------

### *Examples of Human Wisdom:*

> Willa’s children are bitterly arguing about money. Willa draws on her life experience to show them why they should instead compromise in the short term and prioritize their sibling relationship in the long term. 
> 
> "Daphne is a world-class cardiologist. Nonetheless, she consults with a much more junior colleague when she recognizes that the colleague knows more about a patient’s history than she does"
> 
> "Ron is a political consultant who formulates possible scenarios to ensure his candidate will win. To help generate scenarios, he not only imagines best case scenarios, but also imagines that his client has lost the election and considers possible reasons that might have contributed to the loss."

+++ ### What are some theories of human wisdom?

For a more detailed account, see the table on page 5 of the paper.

*Five component theories:*

• Balance theory: "Deploying knowledge and skills to achieve the common good by"  
• Berlin Wisdom Model: "Expertise in important and difficult matters of life"  
• MORE Life Experience Model: "Gaining psychological resources via reflection, to cope with life challenges"  
• Three-Dimensional Model: "Acquiring and reflecting on life experience to cultivate personality traits"  
• Wise Reasoning Model: "Using context-sensitive reasoning to manage important social challenges"

*Two consensus models*

• Common wisdom model: "A style of social-cognitive processing" involving morality and metacognition  
• Integrative Model: "A behavioural repertoire"

These consensus models attempt to find common themes.




++++++ ### Would AI wisdom resemble human wisdom?

*Potential differences:*

• AIs have different computational constraints. Humans need to "economize scarce cognitive resources" which incentivizes us to use heuristics more.  
• Humans exist in a society that allows us to "outsource... cognition to the social environment" such as through division of labor.

*Reasons why human and AI wisdom might converge:*

• Resource difference might be "more a matter of degree than kind"  
• Heuristics are often about handling a lack of information rather than computational constraints  
• AI's might "join our (social) milieu"




+++

### **Definition of wisdom**[^yb717et60ai]**:**

| > If life were a series of textbook problems, we would not need to be wise. There would be a correct answer, the requisite information for calculating it would be available, and natural selection would have ruthlessly driven humans to find those answers. |
| --- |

| > We define wisdom functionally as the ability to successfully navigate intractable problems— those that do not lend themselves to analytic techniques due to unlearnable probability distributions or incommensurable values[^p4mlpkwe8yk]. |
| --- |

+++ ### What kinds of intractable problems?

> **Incommensurable:** It features ambiguous goals or values that cannot be compared with one another[^7vsqtj26zvd].  
> **Transformative:** The outcome of the decision might change one’s preferences, leading to a clash between one’s present and future values  
> **Radically uncertain**. We might not be able to exhaustively list the possible outcomes or assign probabilities to them in a principled way[^t9wlmimd4h8].  
> **Chaotic**. The data-generating process may have a strong nonlinearity or dependency on initial conditions, making it fundamentally unpredictable[^47se0ry9yj9][^pjfiptu4ac].  
> **Non-stationary**. The underlying process may be changing over time, making the probability distribution unlearnable.  
> **Out-of-distribution**. The situation is novel, going beyond one’s experience or available data.  
> **Computationally explosive**. The optimal response could be calculated with infinite or infeasibly large computational resources, but this is not possible due to resource constraints




+++

This seems like a reasonable definition to use, though I have to admit I find the term "intractable problems" to be a bit strong for the examples they provided. For example, Daphne putting aside her ego to consult a junior colleague doesn't quite match what I'd describe as overcoming an "intractable" problem[^rxmhhh05aqh].

### **Two types of strategies for managing this:**

| **1) Task-level strategies** ("used to manage the problem itself") such as heuristics or narratives. **2) Metacognitive strategies** ("used to flexibly manage those task-level strategies")[^sluio9vrpt7] |
| --- |

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/235d9947a9c8e9ae959bc247c7f444c6e0573a579303267f.png)

They argue that although AI has made lots of progress with task-level strategies, it often neglects metacognitive strategies[^zgvv6onvhr]. For this reason, their paper focuses on the latter.

+++ ### Why do the authors believe current AI falls short in metacognition?

The authors provide some specific examples of where they believe AI systems fall short:  
  
1) Struggling to understand their goals (“mission awareness”[^o7wnadtsmiq])  
2) Exhibiting overconfidence[^v1ihrkj71hc]  
3) Failing to appreciate the limits of their capabilities and context (e.g., stating they can access real-time information or take actions in the physical world[^o7wnadtsmiq])  
  
They label this "metacognitive myopia"[^796pi2yx9no].




+++

**Why build wise AI?**:
-----------------------

| > First, it is not clear what the alternative is. Compared to halting all progress on AI, building wise AI may introduce added risks alongside added benefits. But compared to the status quo—advancing task-level capabilities at a breakneck pace with little effort to develop wise metacognition—the attempt to make machines intellectually humble, context-sensitive, and adept at balancing viewpoints seems clearly preferable. |
| --- |

### **Concrete Benefits**

|     | **Why the authors argue that wise AI might provide this benefit** |
| --- | --- |
| **Robustness** | **Reliability over similar inputs**: it'd be unwise to choose "excessively inconsistent" strategies: \\- *Comment*: I guess? Unclear how strong we should expect that effect to be though. **Bias**: identifying deficiencies in the data and either gathering more data or correcting for that bias **Inflexibility**: adjusting its confidence based on the situation |
| **Co-operation**[^oddfc59v0as] | *Slightly edited quotes* **Resolving conflicts among (object-level) strategies**: e.g., when accuracy cues diverge **Assessing the appropriateness (of object-level strategies)**: e.g., whether one can evaluate a chain of argumentation **Seeking appropriate inputs:** e.g., knowing the capabilities of the other counterparty This last point is particularly important for cooperative AI, which could overestimate the abilities of humans or lack common ground such as a shared emotional system. |
| **Safety** | The authors argue that wise reasoning provides an alternative to aligning AI to values[^w7j28jx8a][^mjjqj7s5neq]: > e.g., "one object-level strategy may be a bias toward inaction (not executing an action if it risks harm according to one of several possibly conflicting human norms), which in turn requires metacognitive regulation (learning what those conflicting perspectives are and avoiding overconfidence)" However, they argue that this isn't sufficient, as it doesn't address all the social questions of alignment, both in terms of design decisions ("*Who* should we align AI to? Should we increase the average human well-being, its sum, or care for the whole biosphere? Why assume today's values are the right ones?") and how these AI systems fit into a broader society (specifically how they can be channeled by institutions like governments and markets to allow our values to evolve towards a "shared reflective equilibrium". |
| **Explainability** | Metacognition seem to play a role in assisting humans to justifying their decisions. Presumably it should assist with helping AI to explain its decisions as well[^facv0pq75sc]? |

### **Comparison to Alignment**:

They identify three main conceptual problems for alignment:

1.  Humans don't uniformly prioritize following norms[^wojhuchi41e]
2.  Norms vary sharply across cultures
3.  Even if norms were uniform, they may not be morally correct

| > Given these conceptual problems, alignment may not be a feasible or even desirable engineering goal. The fundamental challenge is how AI agents can live among us—and for this, implementing wise AI reasoning may be a more promising approach. Aligning AI systems to the right metacognitive strategies rather than to the “right” values might be both conceptually cleaner and more practically feasible. |
| --- |

It seems plausible that there might be more agreement on meta-cognitive strategies than values, however, I still expect there to be sufficient disagreement to make this a challenge.

+++ ### Inaction example

"Task-level strategies may include heuristics such as a bias toward inaction: When in doubt about whether a candidate action could produce harm according to one of several possibly conflicting human norms, by default do not execute the action. Yet wise metacognitive monitoring and control will be crucial for regulating such task-level strategies. In the ‘inaction bias’ strategy, for example, a requirement is to learn what those conflicting perspectives are and to avoid overconfidence"




++++++ ### Possible effects on instrumental convergence

In the final section they suggest that building machines wiser than humans might prevent instrumental convergence[^e35uvf4ig9b] as "empirically, humans with wise metacognition show greater orientation toward the common good". I have to admit skepticism as I believe in the orthogonality thesis and I see no reason to believe it wouldn't apply to wisdom as well. That said, activating latents that improve wisdom might also improve alignment, even if it is far from a complete solution.




++++++ ### Further comments on alignment

"With respect to the broader goal of AI alignment, we are sympathetic to the goal but question this definition of the problem. Ultimately safe AI may be at least as much about constraining the power of AI systems within human institutions, rather than aligning their goals"




+++

**Benchmarking Wisdom**:
------------------------

The paper discusses potential for benchmarking AI wisdom. They seem to be in favor of starting with tasks that measure wise reasoning in humans and scoring their reflections based on predefined criteria. It's worth noting that these criteria can be about reasoning processes rather than the outcome they reach.

This could potentially be fruitful, however, I do worry that it might be fairly easy for AI's to learn to [Goodhart](https://www.splunk.com/en_us/blog/learn/goodharts-law.html) here - apply metacognition in a way that is fairly shallow, but sufficient to satisfy the human raters.

The author may not be in disagreement: whilst they see benchmarking as a "crucial start" they also assert that " there is no substitute for interaction with the real world". This leads them to suggest a slow rollout to give us time to evaluate whether their decisions really were wise.

One worry I have is that sometimes wisdom involves just knowing what to do without being able to explain it. This might be problematic for attempts to evaluate wisdom by evaluating the wisdom of a person's reasoning[^fdxex7tq5m6].

+++ ### Challenges with benchmarks

Memorization: Benchmark results can be inflated by memorizing patterns in a way that doesn't generalize outside of the training distribution

Evaluating the process is hard: They claim wisdom depends on the underlying reasoning rather than just success[^q6hthkp0onr]. Reasoning is harder to evaluate than the correct answer.

Producing a Realistic Context: It may be challenging to produce artificial examples as the AI might have access to much more information in the real world 




+++

**Building Wise AI**:
---------------------

| > *Proposal A*[^3f6nrflqrfk]* (two training steps): * > > 1. Train a model for wise strategy selection directly (e.g., to correctly identify when to be intellectually humble) > 2. Train them to use those strategies correctly (e.g., to carry out intellectual humble behavior). > > *Proposal B (one training step):* > > 1. Evaluate whether models are able to plausibly explain their metacognitive strategies in benchmark cases > 2. If this is the case, then simultaneously train strategies and outputs (e.g., training the model to identify the situation as one that calls for intellectual humility and to reason accordingly[^m9pqokpln3p]). |
| --- |

**Final Thoughts**
==================

The authors address the point that attempting to build wise AI  have negative externalities either due to malicious use or due to the project failing. In response they write:

> If the alternative were halting all AI progress, building wise AI would introduce added risks. But compared to the status quo—advancing capabilities at a breakneck pace without wise metacognition—the attempt to make machines intellectually humble, context-adaptable, and adept at balancing viewpoints seems clearly preferable. 

One final point of difference I'd like to mention: The authors seem to primarily imagine wise AIs acting directly in the world[^zr5bxcicrcs]. In contrast, my primary interest is in [wise AI advisors](https://www.lesswrong.com/posts/SbAofYCgKkaXReDy4/chris_leong-s-shortform?view=postCommentsNew&postId=SbAofYCgKkaXReDy4&commentId=Zcg9idTyY5rKMtYwo) working in concert with humans.

+++ ### Why am I focused specifically on wise AI advisors?

I'm personally focused on cybernetic/centaur systems that combine AI advisors with humans because this allows the humans to compensate for the weaknesses of the AI.

This has a few key advantages:

• It provides an additional layer of safety/security.

• It allows us to benefit from such systems earlier than we would be able to otherwise

• If we decide advisors are insufficient and that we want to train autonomously acting wise agents, AI advisors could help us with that.




+++

| Whilst the possibility of training wise AI has been previously discussed in the academic literature, I am hopeful that this paper will turn out to be a landmark. Given the credibility of the authors and the quality of the work, it's plausible to me that it will play a key role in causing artificial wisdom blossom into its own sub-field of ML. I really hope this is the case because I suspect that worlds where this happens are much more likely to result in good outcomes than worlds where this does not happen. To quote the authors: > We may not know precisely what form wise AI will take—but it must surely be preferable to folly. |
| --- |

Read the full, original paper

+++ ### Highlights of additional content in the full paper

• A table summarising psychological approaches to wisdom  
• A list of ideas of how to engineer wiser AI  
• A list of outstanding questions




+++

[^uycflfu5vsl]: Johnson, B. (2022). Metacognition for artificial intelligence system safety: An approach to safe and desired behavior. Safety Science, 151, 105743.  

[^yb717et60ai]: For the purposes of this paper... the authors aren't claiming to make a universal definition. 

[^p4mlpkwe8yk]: See the collapsable section immediately underneath for a larger list. 

[^7vsqtj26zvd]: Walasek, L., & Brown, G. D. (2023). Incomparability and incommensurability in choice: No common currency of value? Perspectives on Psychological Science, 17456916231192828.  

[^t9wlmimd4h8]: Kay, J., & King, M. (2020). Radical uncertainty: Decision-making beyond the numbers. New York, NY: Norton.  

[^47se0ry9yj9]: They seem to be pointing to Knightian uncertainty 

[^pjfiptu4ac]: Lorenz, E. (1993). The essence of chaos. Seattle, WA: University of Washington Press.  

[^rxmhhh05aqh]: Prof. Grossmann (personal correspondance): "You are right that the term "intractable problem" is complicated and our group has debated it for a while (different disciplines favoured different jargon). Our examples were chiefly for highlighting the metacognitive benefits for wise decision-making." 

[^sluio9vrpt7]: This table is copied from the paper. 

[^zgvv6onvhr]: They use examples we discussed earlier to help justify their focus on metacognition. Whilst the Willa example might not initially appear related to metacognition, I suspect that the authors see this as related to "perspective seeking", one of the six metacognitive processes they highlight. 

[^o7wnadtsmiq]: Li, Y., Huang, Y., Lin, Y., Wu, S., Wan, Y., & Sun, L. (2024). I think, therefore I am: Awareness in Large Language Models. arXiv preprint arXiv:2401.17882.  

[^v1ihrkj71hc]: Cash, T. N., Oppenheimer, D. M., & Christie, S. Quantifying UncertAInty: Testing the Accuracy of LLMs’ Confidence Judgments. Preprint.  

[^796pi2yx9no]: Scholten, F., Rebholz, T. R., & Hütter, M. (2024). Metacognitive myopia in Large Language Models. arXiv preprint arXiv:2408.05568.  

[^oddfc59v0as]: An older version of the paper suggested: "Wisdom could enable the design of structures (such as constitutions, markets, and organizations) that enhance cooperation in society". 

[^w7j28jx8a]: The original paper said: "It can be incredibly challenging to "exhaustively specify goals in advance". Humans handle this by using goal hierarchies and wisdom could assist AI's in navigating this" 

[^mjjqj7s5neq]: A previous version of the paper claimed: "Perhaps the greatest risk is currently systems not working well enough. Machine metacognition could be useful for this. In particular, "AIs with appropriately calibrated confidence can target the most likely safety risks; appropriate self-models would help AIs to anticipate potential failures; and continual monitoring of its performance would facilitate recognition of high-risk moments and permit learning from experience." 

[^facv0pq75sc]: I agree that metacognition seems important for explanability, but my intuition is that wise decisions are often challenging or even impossible to make legible. See Tentatively against making AIs 'wise', which won a runner up prize in the AI Impacts Essay competition on the Automation of Wisdom and Philosophy.The authors acknowledge the possibility that most attempts at introspection may fail to observe what really produced the decision, as opposed to merely producing an inference/story. Nonetheless, they assert that these inferences are in fact useful. 

[^wojhuchi41e]: The first sentence of this section reads "First, humans are not even aligned with each other". This is confusing since the second paragraph seems to suggest that their point is more about humans not always following norms, which is what I've summarised their point as. 

[^e35uvf4ig9b]: This paper doesn't use the term "instrumental convergence", so this statement involves a slight bit of interpretation on my part. 

[^fdxex7tq5m6]: Prof. Grossmann (personal correspondance): "I also don't think most philosophical or contemporary definitions of human wisdom in behavioural sciences would primarily focus on "intuition" - I even have evidence from a wide range of countries where most cultures consider a "wise" decision strategy to chiefly rely on deliberation" 

[^q6hthkp0onr]: This is less significant in my worldview as I see wisdom as often being just about knowing the right answer without knowing why you know. 

[^3f6nrflqrfk]: The labels "Proposal A" and "Proposal B" aren't in the paper. 

[^m9pqokpln3p]: For example, Lampinen, A. K., Roy, N., Dasgupta, I., Chan, S. C., Tam, A., Mcclelland, J., ... & Hill, F. (2022, June). Tell me why! explanations support learning relational and causal structure. In International Conference on Machine Learning (pp. 11868-11890).  

[^zr5bxcicrcs]: Prof. Grossmann (personal correspondance): "I like the idea of wise advisors. I don't think the argument in our paper is against it -it all depends on how humans will use the technology (and there are several papers on the role of metacognition for discerning when to rely on decision-aids/AI advisors, too)." 

[^m3kdbcud2f]: Eliezer Yudkowsky's view seems to be that this specification pretty much has to be exhaustive, though others are less pessimistic about partial alignment.