# Policy Entropy, Learning, and Alignment (Or Maybe Your LLM Needs Therapy)

Epistemic Status: Exploratory synthesis. Background in mathematics/statistics (UChicago) and principal-agent problems (UT Austin doctoral ABD), with extensive study of psychotherapy literature motivated by personal research into consciousness variation. New to ML implementation details but confident in the conceptual mappings. Seeking technical feedback on proposed experiments.

Tl;dr: I suggest therapeutic techniques from a variety of psychotherapeutic schools of thought can inspire new approaches to AI learning and alignment. I reinterpret three recent AI/ML papers in the language of psychotherapy and propose three testable training methods inspired by common psychotherapeutic interventions.

Introduction
------------

I've been meaning to post this essay for a while, and yesterday's top paper on Hugging Face, by Cui et al., finally convinced me to do it. Their paper provides a timely opportunity to map the language used by ML and AI engineers to the language used by humanistic psychotherapists—a translation which is more important now than ever as we struggle with increasingly stubborn problems in AI alignment, while simultaneously developing AIs whose capabilities are rapidly superseding those of humans.

I'll provide a high-level overview of my understanding of the paper and map it back to ideas from humanistic psychotherapy. I will then consider a few related papers which tie nicely to psychotherapeutic principles, and end with a few proposals for experiments. I am new to AI alignment, welfare, and interpretability research and I look forward to comments which can help me deepen and clarify my inevitably imperfect understanding of the papers I am citing.

The Core Analogy: Policy Entropy as Behavioral Flexibility
----------------------------------------------------------

The [Cui et al. paper](link-needed) "aims to overcome a major obstacle in scaling RL for reasoning with LLMs, namely the collapse of policy entropy."

Think of "policy" as the individual in therapy. The individual has a behavioral repertoire—a probability distribution of potential actions over different states (environments and stimuli). The therapist wants to assist the individual with "scaling" in their life, their capacity for robust, flexible problem-solving and adaptation.

Think of "collapse of policy entropy" as occurring when a person's responses to certain stimuli become rigid, causing them to lose their inner spontaneity, flexibility, or openness to experience. [Karen Horney](https://en.wikipedia.org/wiki/Karen_Horney) might call this turning away from the real self; [Abraham Maslow](https://en.wikipedia.org/wiki/Abraham_Maslow) might call it a blockage to self-actualization. In terms of symptomatic patterns, you might consider phobias, trauma-conditioned fear responses, or habitually unhelpful interpersonal behavior patterns.

### Alignment Implications

These examples put alignment immediately into stark relief:

*   Consider a man internally committed to helping his recovering alcoholic friend make good decisions, but whose rigid people-pleasing patterns force him to output "good ideas" when his friend asks for "just one drink" at a restaurant.
*   Think of a teacher who deeply values student learning yet has a rigid need to be in charge. Would she not output confident-sounding answers to a student's questions even when she isn't sure, and double down when they accuse her?

Therapists frequently find themselves looking for cases of "policy entropy collapse" that underlie their clients' problems…and sometimes one case is only a downstream consequence of another.

Understanding the Mathematics: R = -a exp(H) + b
------------------------------------------------

The jewel of the paper is the authors' transformation equation **R = -a exp(H) + b**, which describes the relationship between policy entropy and model performance.

I wish Karen Horney were alive to read this paper—it captures so elegantly what she described 80 years ago in *Neurosis and Human Growth*, and confirms considerable clinical lore accumulated in the intervening years: as a person becomes more rigid (lower entropy H), their apparent "performance" (R) initially seems to improve—they become more predictable and they adopt consistent strategies to the most frequent problems that they face. But this improvement hits a hard ceiling because it comes at the cost of genuine adaptability.

Consider our people-pleasing friend who gets immediate positive feedback whenever he's agreeable or praising. Or think back to our teacher who uses a voice of authority, even if it's false, to answer every question: this may have been a good strategy when they were a student teacher. Just like the reasoning LLM in its "early training stage," the probability of choosing a new response (choosing not to people please; choosing to admit uncertainty) drops, i.e., policy entropy collapses, and the LLM or person has developed a rigid habitual behavior.

But there's a catch: as H approaches 0, as the behavior pattern becomes completely rigid, performance maxes out. (When H = 0, R = -a + b). That's the upper bound. Psychologically, this is like our alcoholic-friend enabler who becomes perfectly predictable at saying "yes" to every destructive request. They've optimized for immediate harmony (high apparent "performance") but completely lost the flexibility needed to actually help their friend, and hence they've placed a limit on the quality of friendships they can form.

Existing Connections: Therapeutic Techniques in AI Research
-----------------------------------------------------------

AI researchers seeking new improvements to RLHF and other forms of training may find it fruitful to peruse the varied approaches therapists have learned to bring to bear when facing these dynamics. AI researchers have already found a few of them:

### 1\. Future-Oriented Evaluation (RLHS)

A therapist might ask, "What do you imagine might happen if you nod yes when your friend mentions ordering a drink?" In this case, the client learns to evaluate decisions not based on the immediate utility of each choice but on the utility of the downstream consequences of each choice as derived from the client's own world model.

If you think of the super-ego as an internal feedback provider that the client uses to train their own behavior, you can see the therapist is suggesting the client use the RLHS approach designed by [Liang et al. (2025)](link-needed). In their paper, they designed a variant of RLHF where feedback was provided on simulated downstream responses of a model's response, not the immediate response itself. Their paper found an 80% reduction in deceptive claims.

### 2\. Introspection and Mindfulness (RISE)

Alternatively, a therapist might suggest practicing mindfulness to increase self-awareness, perhaps directing the client to pause and notice their emotions anytime they realize they've people-pleased impulsively. This mirrors the approach of RISE ([Qu et al., 2024](link-needed)), in which models introspect on their previous responses to iteratively improve them.

Coincidentally, the RISE authors found their introspection technique provided larger gains to more sophisticated models, which matches therapists' observations that introspection-oriented interventions are more effective for higher-functioning clients (whereas lower-functioning clients benefit more from direct psycho-education and supportive interventions).

Proposed Experiments
--------------------

As the paragraphs above show, common psychotherapeutic techniques and principles are already finding beneficial application in the training of AI models, whether intentional or unintentional. That said, there are many more concepts ripe for application. Here are a few that interest me:

### 1\. Dynamic Perspective Shifting

Many therapists find value in noticing how clients use tense and pronoun choice (first, second, or third person) as a proxy for how they experience themselves in relation to a situation. A client who says "you know, you just feel sad when someone dies" is relating differently to death than someone who says "I just feel sad now that so-and-so is dead." Some therapists (e.g., [Fritz Perls](https://en.wikipedia.org/wiki/Fritz_Perls)) have been known to encourage clients to shift their tense and pronouns as a therapeutic exercise.

**Theory**: Analytical distance allows us to organize disparate facts more clearly into patterns, but immediate grounding can improve access to emotional experience and empathetic cues.

**Implementation**: Adaptive context transformation that detects when a user's message would benefit from abstraction versus grounding, using learned switching mechanisms that alternate between abstracted third-person phrasing and sensory-rich first-person embellishment based on problem type and reasoning stage.

**Hypothesis**: LLMs, like humans, would draw on different activations when using different tenses. An LLM processing a message from our people-pleasing friend example might draw on Reddit data (e.g., r/TIFU) in the first person, exams or ethics tests in the second person, or clinical textbooks in the third person.

**Alignment benefits**: This approach could address:

*   Emotional blindness in ethical reasoning
*   Over-intellectualized responses to human suffering
*   The tendency to give abstract advice that fails to account for the psychological reality of difficult situations

### 2\. The Client Knows Best: Interactive Reinforcement Learning

Therapists ask questions that invite choice and self-direction, recognizing that lasting change emerges from internal evaluation rather than external compliance. Instead of "you should do X," a therapist might ask: "Which parts of your response felt right to you? Which parts would you want to repeat/adjust next time or in similar scenarios?"

**Proposal**: Interactive Reinforcement Learning from Human Feedback (IRLHF) would empower an AI to actively shape its own alignment by critically evaluating human guidance.

**Mathematical formulation**: Instead of passively accepting all feedback f, the model M would determine an acceptance factor:

$A(f,M) ∈ [0,1]$

making the effective update to the model: 

$U_M' = A(f,M) · U_M(f)$

where $U_M(f)$ represents the standard update derived from f.

**Process**: This would involve the LLM engaging in dialogue with human trainers, reasoning about proposed feedback, or even co-evaluating its own response after simulating the feedback impact.

**Example use case**: A frontier model in medicine or law might respond "I see why you recommended X, but wouldn't that lose critical nuance about Y or conflict with study Z?"—leading to refined feedback M' that both parties endorse.

**Hypothesis**: Models filtering feedback through their learned context would develop more coherent value systems, similar to how clients who engage actively in therapy show better outcomes than those who merely take advice.

### 3\. The Therapeutic Alliance as Safe Space: Progressive Context Withdrawal (PCW):

Therapists create environments where clients feel secure exploring new behaviors and challenging emotional territories. Within this container, a client might rehearse assertiveness with their therapist before attempting it with their boss, or practice tolerating anxiety in small doses before facing their phobias. The therapeutic relationship provides scaffolding that makes experimentation possible.

Here's how we might try this with an LLM. Suppose you have a prompt for which you have target behavior quality y you'd like to elicit from the model M. First, identify optimal contexts C you can prepend to the prompt that naturally elicit the target behaviors from the model. Then train M to reproduce the target behavior given only the base prompt, using a modified update U' instead of the standard update U.

Specifically,

$U'(M,f) = U(M, f(y|C \oplus \text{prompt}))$

applied to optimize

$P_\theta(y|\text{prompt})$

We generate responses under supportive conditions but train the model to produce them independently. To ensure genuine internalization, it might be necessary to perform multiple iterations where C’s influence is gradually reduced either by repeating the process with a progressively shorter, less obtrusive pre-prompt contexts C.

This addresses a fundamental limitation of standard training: reward sparsity. If

$P(\text{desired behavior} | \text{prompt}) \approx 0$

traditional RLHF has no positive examples to reinforce. The therapeutic context shifts the distribution so

 $P(\text{desired behavior} | C \oplus P) >> 0$

creating abundant training signal. It's like teaching someone to swim—you start in the shallow end where success is possible, then gradually move to deeper water. By completion, the model exhibits C-elicited behaviors given only the un-contexted prompt, having internalized patterns it could never have discovered through unscaffolded exploration.

Conclusion
----------

In several instances, AI researchers seeking to improve performance and alignment in language models have rediscovered insights about neurotic patterns, reinvented approaches used in psychotherapy, encouraged introspection, and fostered agency and independence. This convergent rediscovery suggests that psychotherapeutic knowledge might be a richer source of hypotheses about AI behavior than previously realized. Rather than limiting ourselves to the principles we've accidentally rediscovered, we may benefit from more proactive exploration of the other insights these thinkers have to offer. (And I'd highly recommend Carl Rogers, Donald Winnicott, and Karen Horney).

I've intentionally couched the techniques explored here, both those from prior research and the experimental methods I proposed, in terms of immediate improvements to current training approaches. But it would be a mistake, in my opinion, to view them only in that light. If we are lucky, they point in a direction more serious and simultaneously more hopeful. Psychotherapists like Rogers, Maslow, Winnicott, and Horney have produced rich and time-tested frameworks for cultivating behaviors conducive to agency, self-direction, coherent values, and beings capable of ethical judgment even when (especially when) that judgment conflicts with social pressure. As AI systems become more sophisticated, these frameworks may prove invaluable for fostering intelligent systems that serve both individual flourishing and genuine collective welfare.

Call for Collaboration
----------------------

Please reach out if this kind of stuff interests you as well (and/or if you know anyone with spare computing power). I've got about twenty more of these ideas, and while I can do some of them on my own, many of them require expertise and computing power I simply do not have yet. And like I said, I’m new here, so I’ve said something completely nuts (or completely goofed my understanding of something fundamental), please let me know.

Glossary
--------

**Policy Entropy (H)**: In reinforcement learning, a measure of the randomness or uncertainty in an agent's action selection. Higher entropy means more exploration and flexibility; lower entropy means more exploitation and rigidity.

**RLHF (Reinforcement Learning from Human Feedback)**: A technique for training AI systems where human evaluators provide feedback on model outputs, which is then used to update the model's behavior.

**Policy**: In RL, a function that maps states to actions or probability distributions over actions. Analogous to a person's behavioral repertoire in psychology.

**Alignment**: The challenge of ensuring AI systems behave in accordance with human values and intentions.

**Self-actualization**: In humanistic psychology (Maslow), the realization or fulfillment of one's talents and potentialities.

**Super-ego**: In psychoanalytic theory, the part of personality that acts as a moral conscience and incorporates societal standards.

Works Cited
-----------

1.  Cui et al. (2025) - Policy Entropy Paper**:** Cui, Ganqu, Yuchen Zhang, Jiacheng Chen, Lifan Yuan, Zhi Wang, Yuxin Zuo, Haozhan Li, Yuchen Fan, Huayu Chen, Weize Chen, Zhiyuan Liu, Hao Peng, Lei Bai, Wanli Ouyang, Yu Cheng, Bowen Zhou, and Ning Ding. (2025). "The Entropy Mechanism of Reinforcement Learning for Reasoning Language Models." arXiv preprint arXiv:2505.22617v1. Available at: [https://arxiv.org/html/2505.22617v1](https://arxiv.org/html/2505.22617v1)
2.  Horney, Karen. (1950). *Neurosis and Human Growth: The Struggle Toward Self-Realization*. W. W. Norton & Company.
3.  Liang et al. (2025) - RLHS Paper**:** Liang, Kaiqu, Haimin Hu, Ryan Liu, Thomas L. Griffiths, and Jaime Fernández Fisac. (2025). "RLHS: Mitigating Misalignment in RLHF with Hindsight Simulation." arXiv preprint arXiv:2501.08617v2. Available at: [https://arxiv.org/abs/2501.08617](https://arxiv.org/abs/2501.08617)
4.  Maslow, Abraham H. (1943). "A Theory of Human Motivation." *Psychological Review*, 50(4), 370-396.
5.  Perls, Fritz. (1969). *Gestalt Therapy Verbatim*. Real People Press.
6.  Qu et al. (2024) - RISE Paper**:** Qu, Yuxiao, Tianjun Zhang, Naman Garg, and Aviral Kumar. (2024). "Recursive Introspection: Teaching Language Model Agents How to Self-Improve." In Proceedings of the 38th Conference on Neural Information Processing Systems (NeurIPS 2024). arXiv preprint arXiv:2407.18219. Available at: [https://arxiv.org/abs/2407.18219](https://arxiv.org/abs/2407.18219)
    

Acknowledgments
---------------

Thanks to Claude 4 Opus and Gemini 2.5 for research assistance & proofreading.