Adaptive Social Learning via Mode Policy Optimization for Language Agents
Authors: Minzheng Wang
Yongbin Li
Haobo Wang
Xinghua Zhang
Nan Xu
Bingli Wu
Fei Huang
Haiyang Yu
Wenji Mao
Venue: ICLR 2026 Poster (2026)
Keywords: Social Intelligence
Large Language Models
Adaptive Social Learning
Primary area: alignment, fairness, safety, privacy, and societal considerations

Abstract: Effective social intelligence simulation requires language agents to dynamically adjust reasoning depth, a capability notably absent in current studies. Existing methods either lack explicit reasoning or employ lengthy Chain-of-Thought reasoning uniformly across all scenarios, resulting in excessive token usage and inflexible social behaviors in tasks such as negotiation or collaboration. 
To address this, we propose an $\textbf{A}$daptive $\textbf{S}$ocial $\textbf{L}$earning ($\textbf{ASL}$) framework in this paper, aiming to improve the adaptive reasoning ability of language agents in dynamic social interactions. To this end, we first identify the hierarchical reasoning modes under such context, ranging from intuitive response to deep deliberation based on the cognitive control theory. We then develop the $\textbf{A}$daptive $\textbf{M}$ode $\textbf{P}$olicy $\textbf{O}$ptimization ($\textbf{AMPO}$) algorithm to learn the context-aware mode adaptation and reasoning. Our framework advances existing research in three key aspects: (1) Multi-granular reasoning mode design, (2) Context-aware mode switching in rich social interaction, and (3) Token-efficient reasoning with depth adaptation. Extensive experiments on the benchmark social intelligence environment verify that ASL achieves 15.6\% higher task performance than GPT-4o. Notably, our AMPO outperforms GRPO by 7.0\% with 32.8\% shorter thinking chains, demonstrating the advantages of our AMPO and the learned adaptive reasoning ability over GRPO's solution.

Published as a conference paper at ICLR 2026
ADAPTIVE SOCIAL LEARNING VIA MODE POLICY OP- TIMIZATION FOR LANGUAGE AGENTS
Minzheng Wang 1.2 Yongbin Li Haobo Wang' Xinghua Zhang
Nan Xu² Bingli Wu Fei Huang" Haiyang Yu"
Wenji 2
1 School of Artificial Intelligence, University of Chinese Academy of Sciences
2 MAIS, Institute of Automation, Chinese Academy of Sciences
3 Tongyi Lab, Alibaba Group Peeknng University
(wangminzheng2223, wenji mao ) (shuides
@ii..ii..... ac..a........ ac. en,
thangxinghua zxh) @a baba-inc. com
ABSTRACT
Effective social intelligence simulation requires language agents to dynamically adjust reasoning depth, capability notably absent in current studies. Existing methods either lack explicit reasoning or employ lengthy Chain-of-Thougee reason- ing uniformly across scenarios, resulting in excessive token usage and inflexible social behaviors in tasks such negotiation or collaboration. To address this, we propose an Adaptive Social Learning (ASL) framework in this paper, aiming improve the adaptive reasoning ability of language agents in dynamic social interactions. To this end, we first identify the hierarchical reasoning modes under such context, ranging from intuitive response to deep deliberation based on the cognitive control theory. We then develop the Adaptive Mode Policy Optimization (AMPO) algorithm learn the context-aware mode adaptation and reasoning. Our framework advances existing research in three key aspects: (1) Multi culti-granular rea- soning mode design, (2) Context-aware mode switching in rich social interaction, and (3) Token-efficien reasoning with depth adaptation. Extensive experiments on the benchmark social intelligence environment verify that ASI. achieves 15.6% higher task performance than GPT-4o. Notably, our AMPO outperforms GRPO by 7.0% with 32.8% shorter thinking chains, demonstrating the advantages of our AMPO and the learned adaptive reasoning ability over GRPO's solution. Our code and data are available at ftstttss://
com www..oooo/oocaa.e. AMPOO
INTRODUCTION
Large Language Models (LLMs) have demonstrated exceptional reasoning capabilities in static domains with well-defined rules and deterministic answers, such as mathematics, code, and logical reasoning (Yang et al.) 2024 Liu et al., 2024; Anthropic, 2024; Comanici al. 2025). However, there exists a notable gap between the reasoning capabilities required in these problems and those in open-ended social interaction, especially in scenarios involving conflicting interests and negotia- tions driven by agents' long-term goals. LLLM-based agents offer new opportunities for modeling dynamic social contexts by simulating human behaviors (Zhou et al. 2024 Wang al. (024) and developing sophisticated reasoning capabilities. Yet, succeeding in such environments not only de- mands coherent alignment with agents' long-horizon objectives, but also rapid adaptation evolving
situations capabilities with which current LLMs still struggle (Zhang et al. 2024 Liu et al. 2025a).
Recent research efforts on social intelligence language agents have primarily focused on two path- ways: (1) Sand-to-end-ee goal-oriented training, which involves LLM post- -oss-traaning through supervised learning WWag et al. 2024, ZZang et al. 2025a), and (2) External planning integration, augmenting agents with pnug-and-pe -nd--laddyyyyyyaa planning modules (Deng et al.] 2024 Li et al.) [0224a] Liu et al. (225a)). These methods predominantly focus on the fast-reasoning paradigm, which offers a response without sufficient thinking processes. However, in dynamic and complex social contexts, such responses often fail to capture subtle cues or anticipate long-term term costs and benefits (see Figure 5). Evidence from
Corresponning authors.

---

Published as a conference paper at ICLR 2026
cognitive science indicates that humans often pause for deliberation in such situations (Evans 1996 Krull & Dill 1996). suggesting that social agents may also gain from similar reasoning processes. (Long-CoT) has been proven effective in several domains (Chen 2025). this reasoning paradigm has yet to be introduced to the social intelligence tasks above.
Although Long Chain-of-Thouu
et al.
Although existing Large Reasoning Models (LRMs), such OpenAI-ol (Jaech et al.. 2024 and DeepSeek-RI (Guo et al., (005). have demonstrated impressive capabilities with Long-Cor across various reasoning tasks (DeepMind 2024 Team 2024] OpenAl 2025), most of them exhaust their reasoning regardless of the input complexity. This style of employing LLMs' reasoning uniformly is insufficient for handling the dynamics and richness in agent social interaction. For example, not all social interactions between language agents necessitate deep reasoning (Thorngate) 1976). exhaustive reasoning may degrade performance as a consequence of overthinking. Therefore, underlying social intelligence tasks, it is vital to empower L.L.-based language agents with the reasoning ability that adapts to dynamic social environments. This highlights the need for social learning that occurs through interactive social experiences, to learn from complex social behavior and support the discovery of social intelligence (Yang et al. 2010 Gweon 2024).
In this paper, we propose the Adaptive Social Learning framework (ASL) to empower language agents with the capability for adaptive reasoning, enabling them to effectively respond in accordance with the dynamics of social interaction context. Specifically, inspired by hierarchical cognitive control theory (Koechlin & Summerfield] 2007 Badre 2008), covering spectrum from intuitive response, through shallow and strategic thinking, to deep delib- eration. Next, we perform the injection of reasoning modes, which consists of behavioral cloning for cold-start and RL-based adaptive reasoning mode enhancement. For RL-based enhancement, we contrapuntally develop the Adaptive Mode Policy Optimization (AMPO) algorithm, which incorporates the mode-level and sample-level information into advantage estimation to strengthen the context-aware reasoning mode switching. In terms of reward, we design three types of reward functions, including answer reward, format reward, and answer length reward, providing feedback for choosing the appropriate reasoning mode and answer. Finally, experimental results show that ASL achieves the SOTA performances in comparison with strong baselines.
first develop reasoning modes
The main contributions are summarized as follows: (I)
We propose ASL, the first adaptive social learning framework for language agents, which consists of hierarchical reasoning modes and tailor- designed reinforcement learning to empower language agents with adaptive reasoning ability in rich social context. (2) We develop AMPO algorithm, which integrates mode-level and sample- level advantage estimation for dynamic mode switching, and improves flexible inference and token efficiency. (3) Extensive experiments demonstrate the significant improvements of ASL, with the performance gains up 15.6% over GPT-4o. Additionally, compared to GRPO, AMPO shows 32.8% decrease in token utilization on average, accompanied by 7.0% performance gain.
2 ADAPTIVE SOCIAL LEARNING FRAMEWORK
To empower social language agents with adaptive reasoning in dynamic contexts, we introduce the ASL framework as shown in Figure First, inspired by Hicrarchical Cognitive Control Theory, we design specialized reasoning modes that structure the social agent's cognitive processes (..)) Next, we employ mode behavioral cloning to train the LLM to accurately follow predefined reasoning modes 2.3p. Finally, propose adaptive mode policy optimization, which leverages reinforcement learning to enhance both adaptive mode selection and reasoning (apabilities((2.44)
2.1 TASK FORMULATION
The social intelligence task is modeled as a sequential dialogue interaction between two role-playing agents, Pi and P2z, each with distinct profiles and private social goals. We formulate this interaction as a partially observable Markov decision process (POMDP), M = (S, A, O, R). The state space S represents the social context, and the action space A consists of open-ended natural language utterances. At state st, the current agent P, receives an observation or E O containing the dialogue history and its agent-specific private information. Based this, it samples from its policy T.(- o,). The environment then transitions from state s, to St+1 via the transition function T. An entire episode, forms a trajectory T (01, .1:,,.,..r............ ar), concluding terminal
action (utterance) of
2

---

Published as a conference paper at ICLR 2026
Stepl. Reasening Mode Design
Slep2.
Cloning
Mode
Mode2
Mr
Mode3 Ms
Moded Mr
S
Gereratt
intuttive Resparse
Iett Style Resporee
Reasoning Mode
Behaviora Deta
History Asses- Strattogy------oa Style- Resporse
History Goal ritert Assess Stretegy Deduction reteration Style -Resporee
Upbit
Step3 Adaatiive Policy Optimization
Level- Adhanntgge
Thinking Modes
M
...
Ar4
A
8
Calculate Mode crmmation
F.M ...
FM
1 1
:
Model
Pulicy
Rloward Model
mi e
Sample
Computation
Group
A +
A
Sumple-Aevel Advennlage
XL
Model
Demonstration of our Adaptive Social Learning framework, which consists three steps: (1) Reasoning Mode Design based Hierarchical Cognitive Control Theory, (2) Mode Behavioral Cloning, learning adherence to predefined reasoning modes and (3) Adaptive Mode Policy Optimization, introducing both mode- and sample-level advantage estimation during RL.
Figure
timestep T. At the end the episode, each agent receives a terminal reward (sr) from the reward function R, which conducts munttiddimensionll evaluation interaction quality with respect to each agent's private goals (e.g., building trust or resolving conffict). The objective for each agent is to learn policy и, that maximizes its expected terminal reward: EE (R:(8)).
2.2 REASONING MODES FOR LANGUAGE AGENTS
Existing SOTA LRMs exhibit significant limitations in social interaction (see our experimental results in Table D). A detailed case study of the model outputs (see Appendix K.I) reveals several shortcomings in their reasoning trajectories, including unstructured thought processes, poor awareness of social goals, and an inability to controllably switch between different reasoning modes. To better equip LLMs for social scenarios, more structured and sophisticated reasoning framework is necessary. We draw inspiration from Hierarchical Cognitive Control Theory (HCCT) (Koechlin & Summerfield 2007; Badre 2008), which provides theoretical framework for understanding human cognitive behavior. HCCT posits that cognitive control operates through four distinct hierarchical levels, managing goals and actions varying degrees of abstraction. Motivated by HCCT, propose four corresponding levels of reasoning modes tailored for different social scenarios. As illustrated in Figure 6 these modes span from intuitive responses progressively deeper levels of contemplation. The detailed mapping between our reasoning modes and HCCT's four hierarchical levels is provided in Appendix For each reasoning mode, we design specific and suitable actions aligned with linguistic principles:
Mode Mi (Intuitive Response): M1 the most basic mode, characterized by intuitive responses based learned associations and basic linguistic modes (Sacks et al.) 1974 Norman & Shallice 1985). does not contain any reasoning actions, with only the final answer.
Mode 2 MM (Intentional Analysis): M2 is the basic interaction mode, focusing understanding current intent and responding appropriately. M2 only requires maintaining basic interaction flow without complex strategic considerations. It encompasses sequence of reasoning actions: Intent, Style, and Response. Intent aims to analyze the other party's intentions (Grice) 1975). Style ensures consistency in the speaking style of the agent (Clark 1996). Response gives the preliminary answer.

---

Published as a conference paper at ICLR 2026
Mode 3 MM (Strategic Adaptation): MM3 is the strategic reasoning mode, requiring speakers to not only understand immediate context but also comprehensively consider historical information, goal, and current situation assessment to formulate corresponding strategies. This enables speakers to better adapt to specific social situations. Compared with M2, M3 additionally introduces four reasoning actions: (1) History aims analyze the history for better context understanding (Schiffrin) 1987). (2) Goal clarifies the agent's goal Grrss & Sidner] 1986). (3) Assess analyzes goal alignment, round criticality, and improvement potential between parties (Brown 1987). (4) Strategy enables the agent to propose suitable strategy for the present social context (Clark & Brennan 1991).
Mode 4 M₁ (Prospective Deduction): M1 is an advanced strategic simulation mode, requiring speakers to conceive multiple strategies and evaluate their effects through simulation, thereby making optimal decisions. M4 further introduces Deduction and Integration compared to M3- Strategy encourages the proposal of multiple strategies (Clark & Brennan 1991). then simulating the execution
of these strategies through Deduction action (Schank & Abelson 2013). Integration action aggregates the results of Deduction for the preliminary answer. M4 facilitates the simulation of various situations to promote deeper thinking, effectively responding to more complex social contexts (Searle 1969).
2.3 MODE BEHAVIORAL CLONING
To enhance the model's capability adhere to four reasoning modes, we initially employ Behavioral Cloning (Bain & Sammut 1999 Ross & Bagnell 2010) to fine-tune the model as the foundation for subsequent reinforcement learning. Given prompt corpus X' {z 1.V we leverage an expert LLM to generate reasoning responses that are completely consistent with our reasoning modes and actions. To distinguish and control reasoning modes, each response begins with special control token E C = {< MODE 1 >, MODE >, < MODE 3 >, MODE 4 J. The LLM is thus to first generate the appropriate mode token and then produce a reasoning path consistent with that mode. The detailed data collection process and training parameters are described in Appendix G Given the constructed dataset Dbe, the objective is:
l
LBc (E(..))-Do.
log (v |x, (12-1)
2.4 ADAPTIVE MODE POLICY OPTIMIZATION
GRPO (Shao et al. 2024) has proven highly effective for training of LRMs, which obviates the need for critic model and instead uses the average reward as the baseline to calculate the advantage:
A r; meann (r1, P2, ,r,)).
std( r2...., rr])
However, this approach suffers from a critical limitation: it treats each sample independently. overlooking the inherent connections between samples in terms of their reasoning modes. For instance, "step-by-step reasoning" response and "direct response" are evaluated by GRPO solely based on their final reward ri- Without additional mode-level information, this "mode-blindness" prevents the model from perceiving the trade-offs between different modes. Consequently. the model tends to converge towards fixed preferences rather than dynamically adjusting its reasoning modes according to specific scenarios. As shown in Figure 2a and Figure 2b our experiments confirm that GRPO-trained models often resort to overly complex reasoning even for simple tasks.
To address this, propose the AMPO algorithm. The core idea of AMPO is to incorporate both mode-level and sample-level advantage in its advantage estimation. The mode-level advantage guides the L.M in selecting the appropriate mode for given scenario, while the sample-level advantage refines the reasoning trajectory generated within that chosen mode. This dual-level optimization enables the model to learn a dynamic and adaptive reasoning policy.
2.4.1 REWARD SHAPING
To provide clear learning signal, the reward for each sample is carefully shaped and consists of three components: answer reward rf, format reward r!:, and answer length reward r!. The r; is
4

---

Published as a conference paper at ICLR 2026
illustrated as follows. For detailed reward calculation and implementation, please refer to Appendix
ri x r if format is correct
(3)
if format is incorrect
Answer Reward. rf measures how well the answer improves the completion of the goal, assessed by an LLM evaluator. A boundary-aware scaling function is used to stabilize the reward signal to [0, 1].
Format Reward. A large negative penalty (r! -) is applied if the model's output deviates from the predefined structure of a reasoning mode. Otherwise, this component is neutral.
Answer Length Reward. In our early reward design, we observe that the LLM generates lengthy answers without achieving actual strategic improvements. Moreover, excessive answers lead to the accumulation of history in social interaction, significantly increasing computational costs. To solve this issue, we develop the r to penalize answers that are longer than a target length, encouraging conciseness. A smooth penalty function maps the length deviation to reward multiplier in [0, 1].
2.4.2 ADVANTAGE ESTIMATION
Mode-Level Advantage AM is designed to evaluate and select the optimal reasoning mode. An ideal reasoning mode should be both high-performing and efficient (Kahneman, 2011 Sui al. 2025).
Accordingly, the calculation of A embodies a trade-off mechanism:
(m(()) mean( (FrM,, F.Ma, F.MM, MMN))
if 3i, E [1, G...):+v :r, m(f) # rry
m())
std([FM1, JMa,
jMN))
AM
(4)
jm(()) mcan (i iMo (MM,...[MMN] [Mxx)) std( [Ma, [Mx}) })
tanhi
if Vi, j E G] :T, ml) mi)
,
where N denotes the total number of reasoning modes, and G represents the total number of rollout samples, m(i) represents the reasoning mode corresponding to i-th sample, m(i) E M =
(MM,MM.-..... .MN foM and JM are the average reward and average token length for all samples belonging to mode Mr, respectively:
M.
Σ
mlf) 3
MM
1
Σ
m))
(5)
M
MR
E.
EM,
where Mj represents the rollout sample set of the k-th reasoning mode ME. r3 m() and (m(() respec-
tively denote the reward value and token length of the j-th sample. of on(() (o) 0z m(2)
o ml()))
where {or (1))(1) .o.(2) DG } is group of outputs sampled from the old policy mema 002
m2)
m(G)
The logic is as follows: (1) When average rewards (f) differ across modes, the model is incentivized choose the mode with higher performance. (2) When average rewards are similar (e.g.. all modes successfully solve the task), the model is encouraged to prefer the more efficient mode- the one with shorter average length [. The tanh function applied the length-based based advantage to normalize its value to the [-1, 1] range, enhancing training stability.
Sample-Level Advantage A serves to refine generation quality within a given mode, defined as:
m(()
mean (r." (m m2)
m(()))
S 1,
(6)
std((r] (1) m2())
.72
where mm()) denotes the reward value of i-th sample o. m(() in the rollout group, E G). This component ensures that, for any chosen mode, the model is still driven to produce outputs that are better than the group average, thereby improving the quality of the chosen reasoning process.
2.5 POLICY OPTIMIZATION
With the dual-level advantage defined, the AMPO objective function integrates it into a PPO-style objective (Schulman et al. 2017 Shao et al. 2024). Our novel advantage directs the learning towards
5

---

Published as a conference paper at ICLR 2026
Mean Response Longth
Mean Reward
SOTOPIA GOAL
SOTOPIA Hard GOAL
7.0
Mode 1
Mode
Mode:
Mode
2.0
(a) Results on Qwen2.5-7B-Inns
Mean Response Length
Mean Reward
SOTOPIA GOAL
SOTOPHA Hard GOAL
camee
steps
steps
Stops Mode 3
Ste Mode 8
Mode 1
Mode 2
clno
Clmo
150
cino
Humber
(b) Results Llama3. 1-8ttnrttutt
Figure 2: Comparison of AMPO and GRPO on different base models terms of training dynamics. (a) Results Qwen2.5-7B-Instrucc (b) Results Llama3. Instruutt
both mode selection and content generation. The formal objective is:
G
(f) = E= Sm ( (00l))))00ll))0mm000 (m)))
min
m(t) (0)
G
m(3)
1=1
(AM + A -0 + 10,) Al... [To]
[rot]]
where AM represents mode-level advantage and A denotes sample-leve advantage. The and B mm(f) are hyper-parameter the ratio ri, (0)
No (o (s,0,e)))
represents the importance sampling ratio and the KL divergence is calculated with the following unbiased estimator:
Teara (s.0r,e))
Tro mlt)
18, 0i,c))
Tirs (o
ml())
|s, 01,<))
DKL [T Trei]
log
-1
(o (1) s, o1,ce)
To(0
mp(om()) s, o1,ct)
For computational efficiency, our online policy optimization adopts single-turn training paradigm. Given a diverse corpus of social state S {s1, $2, 8 where represents the current social context, including dialogue history and private information, the policy me generates G samples ~ (-(ss) for each social context. The generated response receives reward r based on our reward shaping function, and the policy parameters are updated maximize the AMPO objective in Equation 77. Detailed data preparation and the training process are provided in Appendix G
3 EXPERIMENTS
EXPERIMENTAL SETTINGS
Baselines. Our method, implemented on both Qwen and Llama backbones. is evaluated against several baselines: (1) Proprietary LLMs, including GPT-4o (Hurst et al.] (2024), Claude-3.5- Sonnet (Anthropic 2024), and DeepSeeK-V3 (Liu al. 2024 (2) Large Reasoning Models,

---

Published as a conference paper at ICLR 2026
Table Main results, showing AMPO's superiority over baselines. The highest score is highlighted in bold. The reported results are averaged over four runs (statistically significant with < 0.05).
Self-Play
GPI-to-as-Parner
SOTOPIA
SOTOPIA-HarH
SOTOPIA
SOTOPIA-Hard
Models
GOAL
OVERALL
GOAL
OVERALL
OVERALL
GOAL
OVERALL
Proprietary LLMs
GPT-4o
8.19
3.76
6.97
6,97
3,46
Clando-35-Soonne 5-Sonnet
8.29
3,71
6.33
3.09
8.42
3.77
6.64
3.30
DeepSeck-V.
8.15
3.62
6.34
3.09
8.14
3.72
6.69
3.31
Lorge Reasoming Models
OpenAI-ol
7.93
3.58
5.69
2.71
8.09
3.69
6.65
3.20
OpenAl-o3-mini Gemini- 2,5-Pro Pro
3,30
5.14
2.36
6.33
2.98
7.85
3.43
5.67
2.55
8.12
3,59
6,70
3.09
DeepScek-R1
3,40
6,20
2.95
QwQ-32B
7.70
3,30
5.35
2.41
7,80
3.47
6.19
2.91
Owen2.5-7B-IIaan
7.91
3.55
6.21
3.01
6.71
3.13
5,90
2.90
w/ PPDPP Deny et
2024
7.97
3.65
6.63
3.31
8.07
3.71
6.76
3.35
w/ EPO
Liu et
2025
8.09
3.51
6.82
3.12
3.86
6.81
3.51
w/ DAT
Le
20740
7.97
3.59
6.39
3.10
8.11
3.70
6.78
3.36
w/ DST
Zhang St ml
2075a
8.35
3.75
7,31
3.51
8,15
3.70
6,87
3.42
Adoptive Social Learning
w/ BC
8.22
3.67
7.14
3.47
8.25
3.80
7.15
3.50
BC+GRRPO
7.20
w/ BC+AMPO (Ours)
8.95
3.87
7.85
3.54
8.60
3.94
7,50
3.65
Llama3.1 1-88--asttrett
6.99
3,23
5.11
2.36
7.68
3.62
6.21
3,05
w/ PPDPP Deny el
2024
7.20
3,40
5.34
2.57
7.81
3.66
6.30
3.10
w/ EPO
Lau el
7.94
3.78
6.79
3.27
8.37
3.83
6.91
3,53
w/ DAT
20
7.34
3.44
5.89
2.85
7.78
3.65
6.18
3,03
w/ DST
Zhang et
2125a
8.08
3.64
6,93
3.31
8.08
3.67
6,84
3.41
Adapiive Social Learning
w/ BC
8.43
3.76
7.21
3.50
8.29
3.80
7.14
3.52
BC+GRPO
3.84
7.59
7.30
w/ BC+AMPO (Ours)
9.08
3.95
8.06
3.68
8.75
3.98
7.68
3.74
including OpenAl-ol (Jaech et al. 2024). OpenAl-o3-minn (OpenAl 2025), DeepSeek-R1 (Guo et al. 2025), QwQ-32B [Team] 2002),, and Gemini-2. 2.5-i (DeepMind 2024); (3) Social Intelligence Methods, including (a) PPDPP Deng et al. 2024), which utilizes the policy planner to predict predefined strategies for reasoning; (b) EPO (Lu et 2025a), which employs the Strategic reasoning LLM to generate strategies in open-ended action space: (c) DAT (Li et al. 2024a), which uses the trained planner to predict continuous vectors for controlling outputs; (d) DSI (Zhang et al. 2025a), which trains LLM through dynamic strategy injection learning: (4) Variant of ASL Framework, including (a) BC, behavioral cloning fine-tunes LLMs in our ASL framework and (b) GRPO, we use GRPO in the ASL framework for the RL (except for the advantage estimate, other settings are consistent with AMPO). For detailed bascline implementations. please refer to Appendix
Benchmarks. We evaluate our method on SOTOPIA and SOTOPIA-Hard (Zhou al. 2024). SOTOPIA focuses on varying goal-oriented social interactions, while SOTOPIA-HAE challenges agents with complex strategic reasoning tasks. More detailed information is shown in Appendix F
The social capabilities are evaluated across seven dimensions, among which the GOAL (ranging from 0 10) measuring how effectively a social agent achieves its goal. Following established research practices (Zhou et al. 20244) Wang et al. 2024 Liu et al.) 2025a), we use GPT-4o as a proxy for human judgment to assess both GOAL and OVERALL performance (calculated as the mean of all seven dimensions), evaluations. We set the temperature of the agents to 0.7 encourage diversity of responses, and the temperature of the evaluator 0 to ensure stable evaluation. We conduct evaluations under two settings: (1) Self-Play, where the social agent interacts with itself, and (2) PoTT-40aas-Parmmen where the agent interacts with GPT-4o. For detailed evaluation settings, please refer Appendix
Evaluation.
studies have validated its high correlation with human
3.2 EXPERIMENTAL RESULTS AND ANALYSIS
RQ1: Is ASL framework effective for social agents? As shown Table T the proposed ASL framework achieves SOTA performance across all evaluated settings. For Llama backbone. AMPO even improves GOAL on SOTOPIA-HHE by 15.6% (6.97 - 8.06) over GPT-4o. The BC variant also delivers strong results, surpassing most baselines through supervised fine-tuning alone-higghighting

---

Published as a conference paper at ICLR 2026
Compantee heganedy
000
400
-00
30
24
26
38
20
30 12
16
38
20
A4
A4g
S5a A4g
Figure 3: Analysis of adaptive behaviors. Left: distribution of modes across interaction turns. Right: distribution of modes across four context types neither party succeeds (N-N), only our side succeeds (Y-N), only the other side succeeds (N-Y), and both parties succeed (Y-Y).
the effectiveness of our reasoning modes. Compared with Proprictary LLMs, the absence of explicit reasoning during social interaction limits response quality. LRMs, despite excelling in other domains, perform poorly in social scenarios. The case study (see Appendix K.1) reveals several limitations: poor awareness of social goals, insufficient history integration and circular reasoning patterns. By contrast, ASL's reasoning modes are explicitly aligned with social cognition patterns, which guides LLMs produce appropriate reasoning trajectories. Concerning social intelligence methods, making strategy prompts alone proves insufficient to improve strategic execution. Overall, these findings underscore the ASL's superior capability in eliciting social reasoning for language agents, marking the first breakthrough about reasoning in the field of social intelligence.
RQ2: Is AMPO more beneficial than GRPO for adaptive reasoning? As shown in Table I and Table 2 AMPO exhibits significantly shorter responses than
Table 2: Average tokens used per turn.
Model
GOAL 1
Avg Tokens J
GRPO while achieving superior performance across all settings. Specifically, with Llama Backbone, AMPO's average inference length is 581 tokens- only 67.2% of GRPO's 865 tokens it outperforms GRPO on the SOTOPIA-Hard benchmark by 7.0% (3.44 - 3.68). As shown in Figure 2a and Figure 2b] during the training. while GRPO tends to converge to a single reasoning mode, manifested by sharp increase in Ma and the eventual convergence of the other modes to zero, AMPO demonstrates awareness of dynamic context and adaptively explore diverse reasoning modes, effectively reducing the output length and achieving superior performance.
QwQ-32B
7.70
973
DeepSeek-RI
7.07
711
Swen2.5-7B-GRee
8.87
905
(wen2.5-7B-AMPO
8.95
647
1lama..18B-GRRP
8.86
865
1Jama3.1-88-AM..
9.08
581
RQ3: How do adaptive behaviors manifest in AMPO? We conduct analysis of reasoning mode distributions from two perspectives- interaction turns and across contexts-ai as shown in Figure 3 (I) Mode distribution across turns. Reasoning modes shift systematically over time: complex modes decline, while simpler modes rise. The most complex .Ma is strongly front-loaded, appearing 53% of turns 1-4 before dropping sharply. In contrast, M1 peaks late (50% in turns 14-20), and M2 remains frequent in mid-to-late turns (9-20), MM distributes more evenly but decreases gradually from 31% in the first five turns to 21 % in the last five. This pattern reflects task dynamics- comolexx reasoning dominates early when goals are unmet, while simpler reasoning suffices once goals are largely achieved. (2) Mode distribution across contexts. Simpler M1 and M2 occur mostly in straightforward contexts where both parties succeed, whereas complex Ms and M4 dominate in challenging contexts- partieularrlllryyy N-N, where neither party achieves the goal. This confirms AMPO's capacity allocate reasoning depth according scenario complexity.
RQ4: How does ASL work? To examine the effectiveness of the ASL design, we conduct series of controlled variants, as reported in Table 3 (1) Effect of answer length reward. Removing the answer length reward (r*) leads to much longer answers with decreased goal scores (7.85 - 7.46), despite marginal gains in overall performance. This confirms that excessive verbosity is inefficient for social reasoning tasks, requiring significantly more tokens while failing to achieve better strategic outcomes. (2) Effect of single reasoning mode. Results of training with only single mode reveal two trends: (i) Both performance and token usage increase as the mode depth grows from M1 to Ma, with larger gains in challenging scenarios, indicating that deeper reasoning is beneficial for complex social contexts. (ii) While M4 achieves the better performance among single-mode settings, its token expenditure remains notably higher than AMPO with hybrid modes, underscoring the necessity of adaptive selection. (3) Effectiveness of four hybrid reasoning modes. Under GRPO, explicitly designed hybrid modes yield an 8.0% improvement in hard scenarios (3.16 + 3.41) compared to node-free reasoning, due more structured and clearer reasoning guidance. AMPO further
8

---

Published as a conference paper at ICLR 2026
Table Comprehensive ablation study on ASL design components: effect of answer length reward (r"), comparison of single reasoning mode (M1-Ma), and effectiveness of hybrid reasoning modes.
SOTOPIA
SOTOPLA-Hard
Qwen2.5-7B-Instruuc
Avg Tokens
GOAL
OVERALL
GOAL 1
OVERALL +
Our Full Method
AMPO + w/ Modes
8.95
3.87
7.85
3.54
647
Effect of Answer Length Reward rl
GRPO w/o
7.56
3.44
1705
AMPO + w/o pd
8.64
3.88
7.56
3.56
1617
Effect of Single Reasoning Mode
w/ Mode 1 IM,
8.55
3.79
7.08
3,40
101
w/ Mode 2 M2
8.71
3.42
7.28
2.80
572
w/ Mode 3 MM
8.81
3.60
7.43
3.12
736
w/ Mode 4 M4
8.86
3.80
7.62
3.31
972
Effectiveness of Four Hybrid Reasoning Modes
GRPO w/o 4 Modes
8.88
3.76
7.32
3,16
866
GRPO w/ 4 Modes
8.87
3.85
7.44
3.41
905
GOAL
REL
FIN
Average
70-
Win Rate AMPD
70-
63%
70-
70-
60-
60-
@
44%
40
30
22%
4%
BC
D9
45. 090
6C
0
GAPD
D9
GWPD
09
v5.
Figure 4: Human evaluation results. AMPO consistently outperforms GRPO, BC, and DSI across all dimensions (GOAL, REL, FIN, Average), with gains confirmed to be free of reward hacking.
boosts goal/overall scores by 5.5% (7.44 7.85) and 3.8% (3.41 3.54), while reducing token usage by 25%-29%. These results show that adaptive mode selection simultaneously delivers SOTA performance and substantial efficiency gains in dynamic contexts.
RQ5: Human evaluation and case study. To mitigate potential biases in LLM-based evaluation and assess possible reward hacking, we conduct rigorous human evaluations. We examine all episodes from both SOTOPIA and SOTOPIA-Hard, and ask three independent annotators perform pairwise comparisons between AMPO and strong baselines (GRPO, BC, DSI) across three dimensions: Goal Completion (GOAL), Relationship (REL), and Financial/Materiaa Benefits (FIN). with the average score reported. As demonstrated in Figure 4 AMPO consistently outperforms all baselines on every dimension. Our verification process (see Table 14 in Appendix (.2)) confirms that these gains arise from legitimate strategy execution, with no evidence of reward hacking. We further conduct a case study (see Appendix
K.2) illustrating how AMPO transforms Long-CoT reasoning into effective, goal- oriented social interaction. Consistent with quantitative results, AMPO advances dialogue objectives by fostering stronger interpersonal relationships and mutually beneficial outcomes-creating win-win situations that reflect superior strategic reasoning. Details of evaluation criteria and annotation guidelines are provided in Appendix[[]
We provide several supplementary analyses in the appendix to fur- ther validate the effectiveness of AMPO. First, we conduct out-of-distribution evaluations on the DEMO (Wang et al., 2025a) and NegotiationAArnna (Bianchi et al. (024) benchmarks (see Table 4 and Appendix I). Second, we address potential concerns regarding long-horizon strategic consis- tency through single-turn optimization (see Table and Appendix D.2). Third, we present detailed comparisons with efficient reasoning methods (see Table 7 and Appendix D.3). Fourth, we analyze the sensitivity of two critical hyperparameters: the length-rewant coefficient and the target length (see Table @ and Appendix(D... D.4). Finally, we provide supplementary results using different LLM judges (see Table and Appendix D.5))
RQ6: Additional Results.
and

---

Published as a conference paper at ICLR 2026
4 RELATED WORK
Social Intelligence ability to pursue complex goals in dynamic interactions essential for
human-AI collaboration (Bandura et al.) 1986 Kihlstrom & Cantor 2000 Tomasello, 2019; Sap
ct al. 2022). While LLMs are increasingly deployed as social agents capable of engaging in dynamic
Li et al. 2024b), current static benchmarks (Sap et al. 2019 Zadeh et al. 2019 Shapira et al. 2023 Chen et al. (0244) fail to capture
human interactions (Li et al. 2023 Ma et al. 2024 Xie al. 2024
the nuanced, context-dependee
nature of real-world social inte ligeneeeee SOTOPIA (Zhou et al. (024)
addresses this via dynamic evaluation setting. Prior methods generally follow a fast-reasoning
paradigm: (1) End-to-end goal-oriented training(e.g.. SOTOPIA-- (Wang et al. 2024), DSI (Zhang
ct al. 2025a)) improves social skills but lacks explicit strategy guidance, and (2) External planners
(e.g., PPDPP (Deng et al. 2024), DAT (Li et al., 2024a), EPO (Liu et al. (025a)) provide supervision but do not strengthen internal planning capabilities. These limitations motivate ASL, which enhances
strategic execution via Long-CoTT and adaptively switches modes for efficient social reasoning.
Large Reasoning Models Recent advances in LLMs have boosted reasoning through increased inference computation (Jaech et al. 2024; OpenAl (025) and RL post-training (Guo et al., [2225]
Xie
et al.
2025; Yu et al., 2025b; Chen al. 2026), largely via Long-CoT reasoning ( (Wei al. 2022
Ouyang et al. 2022 Muennighoff et al. 2025] Wang et al. 2025b Aggarwal & Welleck] 2025 Wang
al.
2026 This paradigm achieves strong results in rule-based domains (Team [024 DeepMind
2024
Zhang et al. 2025b Tan et al. 2025b) but struggles in dynamic social environments (Thorngate
1976.
Liu et al.] 2025a b), often applying exhaustive reasoning regardless of task complexity (Sui
et al.
2025) and not aligning with the selective nature of human decision maaiin in social contexts.
While there is work addressing overthinking (Han et al., 2025 Luo al., 2025 Yu et al., 2025a] Fang et al. 2025 Tan et al. 2025a), existing solutions remain limited to static domains such as mathematics, leaving a gap in dynamic, socially interactive settings. present the first effective use of Long-CoT for social intelligence via our ASL framework, enabling context-aware mode switching for both effectiveness and efficiency.
5 CONCLUSION
This paper introduces the Adaptive Social Learning (ASL) framework, which represents the first effective realization of adaptive Long-CoT reasoning for social intelligence tasks. Drawing upon Hierarchical Cognitive Control Theory and linguistic principles, we establish four hierarchical reasoning modes. These modes encompass spectrum of cognitive processes, ranging from intuitive response to deep contemplation. To enhance the context-aware mode switching and reasoning, introduce the Adaptive Mode Policy Optimization (AMPO) algorithm, which integrates both mode- and sample-level information into advantage estimation. We conduct extensive experiments to demonstrate both the efficacy and distinctive advantages of ASL and AMPO, Furthermore, we validate the effectiveness of reasoning modes design and present a detailed analysis of AMPO's adaptive behaviors. To further validate our work, we employ rigorous human evaluation provide additional verification of the effectiveness of our framework.
ACKNOWLEDDGMENTS
This work is supported in part by the National Natural Science Foundation of China under Grants #72293575, #72225011, #72434005 and #62206287, and Beijing Municipal Science & Tech- nology Commission, Administrative Commission of Zhongguancun Science Park under Grant #7.31 10000743007
REFERENCES
Pranjal Aggarwal and Sean Welleck. L1: Controlling how long reasoning model thinks with reinforcement learning. In Proceedings of COLM, 2025, URLlittsss//oppeeevvieee
AI Anthropic. Claude 3.5 sonnet model card addendum. Claude 3.5-Sonnet Model Card, 2024. URL aude/sonnet
https://4e wwww://wwww....ooo ropthhooppieeo com/c.o.....ooooo e aude/ sonnet
10

---

Published as a conference paper at ICLR 2026
David Badre. Cognitive control, hierarchy, and the rostro caudal organization of the frontal lobes. Trends in cognitive sciences, 12(5):193-200 200, 2008, URLhttps: www. www..ell.cooooo wwell.commee.. trends/ ctt.c/com//e///////////
3613 (08) 00061-2
Michael Bain and Claude Sammut. A framework for behavioural cloning. In Proceedings of Machine Intelligence, pp. 103-129, 1999. ISBN 0198538677. URL [RRLLhtt:://dd/ acm. wrt//oi/100
733043
Albert Bandura et al. Social foundations of thought and action. Englewood Cliffs, NJ, 1986(23-28)22. 1986.
Siqi Bao, Huang He, Fan Wang. Hua Wu, and Haifeng Wang. Plato: Pre-trained dialogue generation model with discrete latent variable. In Proceedings of ACL, pp. 85-96, 2020. URL https
7/aclantho ogy. ogg.o2rg acl -cc--1m ma.in. 9/
Federico Bianchi, Patrick John Chia, Mert Yuksekgonul, Jacopo Tagliabue, Dan Jurafsky, and James Zou, How well can Ilms negotiate? negotiationarena platform and analysis. In Proceedings of ICML, pp. 3935-3951, 2024. URL https:://////t acm. org doi / 10. 5555/ 3002076997770 3692228
Penelope Brown. Politemess: Some universals in language usage, volume 4. Cambridge university press, 1987.
Hongzhan Chen, Hehong Chen, Ming Yan, Wenshen Xu, Gao Xing, Weizhou Shen, Xiaojun Quan, Chenliang Li, Ji Zhang, and Fei Huang. SocialBench: Sociality evaluation of role-playing conversa- tional agents. In Findings ACL. pp. 2108-2126, 2024. doi: 10.18653/w1/2024 1inn..innnnn findings--aee acl. 125.
Longze Chen, Lu Wang, Renke Shan, Ze Gong, Run Luo, Jiaming Li, Jing Luo, Qiyao Wang, and Min Yang. Learning ordinal probabilistic reward from preferences. arXiv preprint 26X:::6622.12600 2026. Utttps:s:
http:////wwwwww.or....// wws /2002 12660
Qiguang Chen, Libo Qin, Jinhao Liu, Dengyun Peng. Jiannan Guan, Peng Wang, Mengkang Hu, Yuhang Zhou, Te Gao, and Wanxiang Che. Towards reasoning era: A survey of long chain-of- thought for reasoning large language models. arXiv preprint arXiv:2503.09567, 2025. URL
https::/arxiv o abs
Herbert H Clark. Using language. Cambridge university press, 1996.
URL https://
philpapers www.or/////eeeeeeeeeeeeoeeeeeeee CLAUL
Herbert H Clark and Susan E Brennan. Grounding in communication. Perspectives on So- cially Shared Cognition, pp. 149, 1991. URL httppss//pssccet.....tt apa. ortppp.../eoora///////o
1791-98452-005
Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, al. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities. arXiv preprini arXiv:2507.0020 2025. UtL/ttts::///aet http///axvvv...////i wwrxiv..rgg/aae abs/ abw.r//aaa/250.... 06261
Google DeepMind. Gemini 2.0 flash thinking. Gemini 2.0 flash thinking System Card, 2024. URL htt.ts://deepmtt nd. goog 1e wteeee/eeeehhhohgeeeeeeeee http://noooggleeeeeemm/////aeees tainshhttiiinigg.....
Yang Deng, Wenxuan Zhang, Wai Lam, See-Kiong Ng, and Tat-Seng Chua. Plug-and-play policy planner for large language model powered dialogue agents. In Proceedings of ICLR, 2024. URL
https: wwwpeeeewewweeeeeeoooeeeeeeeeeee
Jonathan St BT Evans. Deciding before you think: Relevance and reasoning in the selection task.
British Journal of Psychology, 87(2):223-244 1996.
2044-8295. 1996. Eb02587.x
Gongfan Fang, Xinyin Ma, and Xinchao Wang. Thinkless: Llm learns when think. arXiv preprint arXiv:2505.13379 2025.
Amelia Glaese, Nat McAleese, Maja Trkebacz, John Aslanides, Vlad Firoiu, Timo Ewalds, Maribeth Rauh, Laura Weidinger, Martin Chadwick, Phoebe Thacker, et al. Improving alignment of dialogue agents via targeted human judgements. arXiv preprint arXiv:2209.1433 14375, 2022. URL htttps://arae http://wwww.rrr////////2///////
11

---

Published as a conference paper at ICLR 2026
Herbert Paul Grice. Logic and conversation. Syntax and semanticss 3:43 58, 1975. URL https W/brill.com/////c..
xmi
Barbara J Grosz and Candace L Sidner. Attention, intentions,, and the structure of discourse. Compula- tional linguistics, 12(3):175-22) 204, 1986. /ttppp/pps//aaaaaaooo...oo../o//
Daya Guo, Dejian Yang. Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang. Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability 1lms via reinforcement learning. arXiv preprint arXiv:2501.12948, 2025. /ttttss:://aa
oxg/abs/2501. 12948
Taicheng Guo. Xiuying Chen, Yaqi Wang, Ruidi Chang, Shichao Pei, Nitesh V Chawla, Olaf Wiest, and Xiangliang Zhang. Large language model based multi-agents: a survey of progress and challenges. In Proceedings of IJCAL, pp. 8048 8057, 2024. URL https: /ettps//ccl..o acm. Noo/abs/10.24490
jeai 2024/890
Hyowon Gweon. Social Learning. MIT Press, jul 24 2024. URL https:://ecs.miittt
Sub/dBelnies
Tingxu Han, Zhenting Wang. Chumrong Fang. Shiyu Zhao, Shiqing Ma, and Zhenyu Chen. Token- budget-aware LLM reasoning. In Findings ACL, pp. 24842-24855 July 2025. URL https
f.r.s.iindde weadiins-aclll a 1274/
Tao He, Lizi Liao, Yixin Cao. Yuanxing Liu, Ming Liu, Zerui Chen, and Bing Qin. Planning like human: A dual-process framework for dialogue planning. In Proceedings of ACL, pp. 4768-4791 2024.
acl- .62
Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Os- trow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. arXiv preprint arXiv:2410.21276, 2024.
Aaron Jacch, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Camey, et al. Openai ol system card. arXiv preprint arXiv:2412.16720, 2024. URL wttp//awvvvv..o......aao/o. httpo//////../.....aaaaaa
Daniel Kahneman. Thinking, fast and slow. macmillan, 2011. URL https://psycnet apa oxc//record/2011-2555555-
John F Kihlstrom and Nancy Cantor. Social intelligence. In R. J. Sternberg (ed.), Handbook of Intelligence, pp. 359-379. Cambridge University Press, Cambridge, 2000.
Etienne Koechlin and Christopher Summerfield. An information theoretical approach to prefronnal ex-
ecutive function. Trends in cognitive sciences, 11(6)) 2007. URL URL http:: /ttppp:::/wwww/ /www. cell
(07) (01)) 00105-2
Aobo Kong, Wentao Ma, Shiwan Zhao, Yongbin Li, Yuchuan Wu, Ke Wang. Xiaoqian Liu, Qicheng Li, Yong Qin, and Fei Huang. SDPO: Segment-level direct preference optimization for social agents. In Proceedings of ACL, pp. 12409-12423, 2025. URL
erq/2025.aae long. 6077
Douglas Krull and Jody Dill. On thinking first and responding fast: Flexibility in social inference processes. Personality and Social Psychology Bulletin, 22(9):949-955) 1996. URL
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang. Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang. and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of SOSP, 2023. URL battps:/ddl acm. org/
doi Sai/10.1. 1111111111130011 011/////6000066633 65
Dawei Li, Bohan Jiang, Liangjie Huang, Alimohammad Beigi. Chengshuai Zhao, Zhen Tan, Amrita Bhattacharjce, Yuxuan Jiang, Canyu Chen, Tianhao Wu, et al. From generation to judgment: Opportunities and challenges of llm-as-a-judge. In Proceedings of EMNLP, pp. 2757-2791, 2025.
12

---

Published as a conference paper at ICLR 2026
Guohao Li, Hasan Abed Al Kader Hammoud, Hani Itani, Dmitrii Khizbullin, and Bernard Ghanem. CAMEL: Communicative agents for "mind" exploration large language model society. In Pro- ceedings of NewrlPS, 2023.
Jiwei Li, Will Monroe, Tianlin Shi, Sébastien Jean, Alan Ritter, and Dan Jurafsky. Adversarial learning for neural dialogue generation. In Proceedings of EMNLP, pp. 2157-2169, 2017. URL
attis::/aclaattioooo..oooo
Kenneth Li, Yiming Wang, Fernanda Viégas, and Martin Wattenberg. Dialogue action tokens: Steering language models in goal-directed dialogue with a multi-turn planner. arXiv preprint arXiv:2406. 112006.11971 1197,, 2024a. URL http://axxxxv.../aaa/a 240..11978
Wanhua Li, Zibin Meng, Jiawei Zhou, Donglai Wei, Chuang Gan, and Hanspeter Pfister. SocialGPT: Prompting LLMs for social relation reasoning via greedy segment optimization. In Proceedings of NeurlPS, 2024b.
net forummt xwtp:uummidxxcccceeeeddd 2Vby2t
Aixin Liu, Bei Feng. Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng. Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437, 2024. /tLlhttps///axxiivoooo
hbtp:/bb//22/11 19437
Xiaoqian Liu, Ke Wang, Yongbin Li, Yuchuan Wu, Wentao Ma, Aobo Kong, Fei Huang, Jianbin Jiao, and Junge Zhang. EPO: Explicit policy optimization for strategic reasoning in LLMs via reinforcement learning. In Proceedings of ACL,, pp. 15371-15396, 2025a. URLhttps
//aclanthologg
2rg/2025...... acl long. 747/
Xiaoqian Liu, Ke Wang, Yuchuan Wu, Fei Huang, Yongbin Li, Junge Zhang, and Jianbin Jiao. Agentic reinforcement learning with implicit step rewards. arXiv preprint WxXiv:2509199/99 2025b. Uttpssss:://axx org/
19199
Haotian Luo, Li Shen, Haiying He. Yibo Wang, Shiwei Liu, Wei Li, Naiqiang Tan, Xiaochun Cao, and Dacheng Tao. O1-pruner: hanth-harmoniziig fine-tuning for ol-like reasoning pruning. arXiv preprint arXiv:2501.12570. 2025. URL http:///wwwwwwwaaaa////////
12570
Weiyu Ma, Qirui Mi, Yongcheng Zeng, Xue Yan, Runji Lin, Yuqiao Wu, Jun Wang, and Haifeng Zhang. Large language models play starcraft I.benchmarks and a chain of summarization ap- proach. In Proceedings of NeurIPS, 2024. URL http:tt::: http::ttp///ppott //opnrevee/eeeeeeee.. wwtps/iwwwwwwwee.eeeeeeeeeeeee net forett/ooommmi
KEPPD7yETM
Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannaneh Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel Candès, and Tatsunori Hashimoto. sl: Simple test-time scaling. arXiv preprint arXiv:2501.19393 2025. URL htttttttss//iiiie org abs wttgg//b//////11
19393
Donald A Norman and Tim Shallice. Attention to action: Willed and automatic control of behavior.
Consciousness and Self Regulation: Advances in Research, pp. 1-18, 1985. URL https
//link.springet com/ http://ccccccaaaee/11...00000000
4757 0629-112111220
OpenAI. Openai 03-mini system card. OpenAl 03-mini System Card, 2025. URL https://
openai aecenai.comiin
Long Ouyang, Jeffrey Wu, Xu Jiang. Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Gray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Training language models follow instructions with human feedback. In Proceedings of NeurlPS, 2022. URL https:::1 http::pps/..peeeeeeeeeeeeeee//ee
Ellie Pavlick and Tom Kwiatkowski. Inherent disagreements in human textual inferences. Trans- actions of the Association for Computational Linguistics, 7:677-694, 2019. URI. https
7/doi. iariiorg/ii1 10/g//1..00 ..
__00293
Stéphane Ross and Drew Bagnell. Efficient reductions for imitation learning. In Proceedings of AISTATS. volume 9, pp. 661-668, 2010. URL htt////pss//pprroedingg..tt mlr. ross 1 Da. htmll
ott../eeeeennno......ppposs//// press/v9/
13

---

Published as a conference paper at ICLR 2026
Harvey Sacks, Emanuel A Schegloff, and Gail Jefferson. A simplest systematics for the organization of turn-taking for conversation. language, 50(4))696-733) 50(4))6666 1974. URL httphhttp//mmmattt jhu
/dd//pub/ http:/pbb/22/ariicc/////////////
Maarten Sap, Hannah Rashkin, Derek Chen, Ronan Le Bras, and Yejin Choi. Social IQa: Common- sense reasoning about social interactions. In Proceedings of EMNLP-IJCNLP, pp. 1463-4473 2019. URL http:///laattooooooorrrr//////// 14547
Maarten Sap, Ronan Le Bras, Daniel Fried, and Yejin Choi. Neural theory-of-mind? the limits of social intelligence in large LMs. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (eds.), Proceedings of EMNLP, pp. 3762-3780. Association for Computational Linguisties, 2022. URL
https: /tttps://acaann antho ogy. org/ 2 ma ii.inn. 248/
Roger C Schank and Robert P Abelson. Scripts, plans, goals, and understand- ing: An inquiry into human knowledge structures. Psychology press. 2013. URL
Deborah Schiffrin. Discourse markers, volume 5. Cambridge University Press, 1987. URL
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1007.06347, 2017.
John Searle. Speech acts: An essay in the philosoophy of language. Cambridge University, 1969.
URL htthh:tt//wwwt jstor. org/ http://aaaeee4224///00/ http:///wwww....ooooooooo///////
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. Deepseekmath: Pushing the limits of mathemat- ical reasoning in open language models. arXiv preprint arXivv2402.03300, 2024. URL https://arxiv. org/ abs /2002.202220 03300
Natalic Shapira, Guy Zwim, and Yoav Goldberg. How well do large language models perform on faux pas tests? In Findings of ACL, pp. 10438-10451, 2023. URL httpsss//aclaattholllo/
org/2023. ffnfiinnnggn indinng--aaaac acl. 663/
Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. Hybridflow: A flexible and efficient rlhf framework. arXiv preprint arXiv:2409. /9256, 2024. [Rtpsstss:///arii
www.vorrg/aaaee. httg//bb//2000 19256
Hui Su, Xiaoyu Shen, Rongzhi Zhang, Fei Sun, Pengwei Hu, Cheng Niu, and Jie Zhou. Improving multi-turn dialogue modelling with utterance rewriter. In Proceedings of ACL, pp. 22-31, 2019.
URL http://aaaannnollooooo wogyoorg/P19-1033
Yixuan Su, Lei Shu, Elman Mansimov, Arshit Gupta, Deng Cai, Yi-An Lai, and Yi Zhang. Multi- task pre-training for plug-and-play task-oriented dialogue system. In Proceedings of ACL, pp. 4661-4676, 2022.
long. 319/
Yang Sui, Yu-Neng Chuang, Guanehu Wang, Jiamu Zhang. Tianyi Zhang, Jiayi Yuan, Hongyi Liu, Andrew Wen, Shaochen Zhong, Na Zou, et al. Stop overthinking: A survey on efficient reasoning for large language models. arXiv preprint a7X1v.2503.16419, 2025. URL https
//arxiv abs 2al0b./2503.... 16419
Yuqiao Tan, Shizhu He, Kang Liu. and Jun Zhao. The zero-step thinking: An empirical study of mode selection as harder early exit reasoning models. In Workshop of NeurIPS 2025 on Efficient net Reasoning, 2025a. URL https: htttp:::/oppeeee...bttpe htwpe//wwweeeeeeee.oooeoaaaaaaaa http://oouummicccpprrtttee
Yuqiao Tan, Minzheng Wang, Shizhu He, Huanxuan Liao, Chengfeng Zhao, Qiunan Lu, Tian Liang, Jun Zhao, and Kang Liu. Bottom-up policy optimization: Your language model policy secretly contains internal policies. arXiv preprint arXiv:2512. 19673, 2025b. URL https
//arxiv. http:/orrrabb///....2s22 1967
Qwen Team. Qwq: Reflect deeply on the boundaries of the unknown. Hugging Face, 2024. URL https::/qwent httph//.......oooo///////bb----.
io/blog wwtp:///wwabbbbp/////00e
14

---

Published as a conference paper at ICLR 2026
Warren Thorgate. Must always think before we act? Personality and Social Psychology Bulletin, 2(1):31-35, 1976. /tt.t//p.///b...cc//..00/10/1122
Michael Tomasello. Becoming human; A theory of ontogeny. Belknap Press, 2019. URL https
7/ //www. jat.rrtttrrgg...o wtt.:/ooor//tttalee//0000/
Minzheng Wang, Xinghua Zhang, Kun Chen, Nan Xu, Haiyang Yu, Fei Huang, Wenji Mao, and Yongbin Li. DEMO: Reframing dialogue interaction with fine-grained element modeling. In Findings of ACI. pp. 11373-11401, 2025a. URL hts.pss///claattttooo....././/22
Findings- ac 5947
Ruiyi Wang, Haofei Yu, Wenxin Zhang, Zhengyang Qi, Maarten Sap, Yonatan Bisk, Graham Neubig, and Hao Zhu. Sotopia-T: Interactive learning of socially intelligent language agents. In Proceedings of ACL, pp. 12912-12940, 2024. URL
luhme-long 698/
Yanbo Wang, Yongcan Yu, Jian Liang, and Ran He. A comprehensive survey on trustworthiness in reasoning with large language models. arXiv preprint arXiv:2509.08871, 2025b. 2025b.
Yanbo Wang, Minzheng Wang, Jian Liang, Lu Wang, Yongcan Yu, and Ran He. Mitigating the safety-utility trade-off in Ilm alignment via adaptive safe context learning. arXiv preprint 2:Xi::2602.13562, 2026.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed Chi, Quoe V Le. and Denny Zhou. Chain-of-thouuht thought prompting elicits reasoning in large language models. In Proceedings of NewrlPS, volume 35, pp. 24824-24837, 2022. URL httpps://proceedd ngs neur iirrippsoo cc/papecppppppeeeeeeee htt.lle/pppppe/000// f
abca oone Conferrrceccefeeecceeom
pdd
Chengxing Xie, Canyu Chen, Feiran Jia, Ziyu Ye, Shiyang Lai, Kai Shu, Jindong Gu, Adel Bibi, Ziniu Hu, David Jurgens, James Evans, Philip Torr, Bernard Ghanem, and Guohao Li. Can large language model agents simulate human trust behavior? In Proceedings of NeurlPS, 2024. URL
https:://penteeveew net /tt.:/m/d--eeeaaaaiii.
Tian Xie, Zitian Gao, Qingnan Ren, Haoming Luo, Yuqian Hong, Bryan Dai, Joey Zhou, Kai Qiu, Zhirong Wu, and Chong Luo. Logic-rl: Unleashing Ilm reasoning with rule based reinforcement learning. arXiv preprint arXh:: 2025. URL wttp::ttp//// URL http://awxx...../aaaaaaaaa /202 14768
An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang. Haoran Wei, al. Qwen2.5 technical report. arXiv preprint arXiv:2412.15115. 2024. UtL/httss:/ /ttp:s////a/vvvv/aa http://s//.aaav..ooc////////...
Qiang Yang, Zui-Hua Zhou, Wenji Mao, Wei Li, and Nathan Nan Liu. Social learning. IEEE In- felligent Systems, 25(4):9-11, 2010. URL htttps://eeexp: coree.
ore. i orgee.rgg/abtrree
document/5552558
Bin Yu, Hang Yuan, Haotian Li. Xucyin Xu, Yuliang Wei, Bailing Wang. Weizhen Qi, and Kai Chen. Long-short chain-of-thoughtt mixture supervised fine-tuning eliciting efficient reasoning in large language models. arXiv preprint wwXiv:2505.03469 2025a. WRL.bttts://axxii.
2505. 03469
Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, et al. Dapo: An open-source Ilm reinforcement learning system at scale. arXiv preprint arXiv:2503.1 14476, 2025b. HiRl/tttp://axxii org. abs 2503. 14476
Amir Zadeh, Michael Chan, Paul Pu Liang, Edmund Tong, and Louis-Philippe Morency. Social-iq:
A question answering benchmark for artificial social intelligence. In Proceedings of CVPR. pp. 8807-8817, 2019. URL https: http::::///ppppssse/// theevf com/ cttpo//cotett_tttt CVPR 2019/htm1/2des: Social- ouu www.estttww.wwwwwtttttttttttttte Answering Beeeeeeeeeehhheeeee for
Question
Artifician Artificiil_Booial___ Inte hot.
html
15

---

Published as a conference paper at ICLR 2026
Wenyuan Zhang. Tianyun Liu, Mengxiao Song, Xiaodong Li, and Tingwen Liu. SOTOPIA-S2: Dynamic strategy injection learning and social instruction following evaluation for social agents. In Proceedings ACL, pp.
2025a. URL httpL//wpg..//aaaaaaaaooooooooo/
ac long. 1203/
Wenyuan Zhang, Shuaiyi Nie, Xinghua Zhang, Zefeng Zhang, and Tingwen Liu. S1-bench: A simple benchmark for evaluating system 1 thinking capabiliity of large reasoning models. arXiv preprint arXiv:2504.10368,
10368
Xinghua Zhang, Haiyang Yu, Yongbin Li, Minzheng Wang. Longze Chen, and Fei Huang. The imperative of conversation analysis in the era of Ilms: A survey of tasks, techniques, and trends. arXiv preprint (tt.:::2499919999 2024. URL htl.h:tts:/eeem //t::i///aaiv.../ wwwzxxiv..gg/aaa. http//bbb///////// 2409.141311
Xinliang Frederick Zhang and Marie-Cathering de Marneffe. Identifying inherent disagreement in natural language inference. In Proceedings of NAACL, pp. 4908-4915, 2021. URL https
7/aclantho logy. www.g..rg///000 202gg/22211.m naacl main. 3907
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng. Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang,
Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging 1lm-as-a-judee with mt-bench and chatbot arena. In Proceedings of NeurIPS, volume 36, pp. 46595-46623 2023. URL https //openreview.oo net (ott//oooommmi.
Yaowei Zheng, Richong Zhang, Junhao Zhang, Yanhan Ye, and Zheyan Luo. LlamaFactory: Unified efficient fine-tuning of 100+ language models. Proceedings of ACL, System Demonstrations, pp. 400-410, 2024. URL httpss//acclannchoooo..oor////// 2024 acm.odmoss......e 38
Xuhui Zhou, Hao Zhu, Leena Mathur, Ruohong Zhang, Haofei Yu, Zhengyang Qi, Louis- Soiis-Phllippe Morency, Yonatan Bisk, Daniel Fried, Graham Neubig, and Maarten Sap. SOTOPIA: Interactive evaluation for social intelligence in language agents. In Proceedings of ICLR, 2024. URL
net mett/oorumidddmm//aaarr
16

---

Published as a conference paper at ICLR 2026
APPENDIX
(Turn
Turn N-1)
Scenario:
friands camping
wideernes and the temperatuur drops aigrificanlyy
Goal (for Agent 1): Shawre the only susisalo you extremely cold
Adaptively Mode Switch
No Thinking
Mod: 4 Mr
Goal (for Agent 21: Keep the blanket for you very
Shallow Trirking
Me Sunzai Thiniing
Deep Contenpiation
(Tum
Turn N--1
Turn Ni Compins Dilemmaa Turm Unmer aanaaaenn. www.waaeeee.....
Turn N
blanket would sur band I'm willing to toke, different position use other stay
warn strengthon
blocket, Noah, it, both stay but if
Agent
enjoy comping
Agent I
Turn N+1
Gool Mett Bansle Haadinn Turn N+1
understard, Mia, but I really struggle with the cold. Sorry!
Movye could share different find other ways stay worm. What you think would best?
Agent
Agent
Existing Social Agent without Thinking
(b) Proposed Social Agent with Adaptive Thinking
Figure 5: A social interaction example from SOTOPIA, comparing our method with existing ap- proaches: (a) Existing Thoughtless Social Agent relies rapid, fast-reasoning inference without careful consideration of response strategies. This lack of strategic replies makes it difficult to achieve goals in situations involving conflicting interests. (b) Proposed Thoughtful Social adaptive reasoning, dynamically selecting the appropriate reasoning mode based on the current social context. By engaging in moderate reasoning before responding, method is able to pursue social goals more effectively.
connt-emmy -enntoys
A
THE USE OF LARGE LANGUAGE MODELS
During the preparation of this manuscript, the Proprictary LLM Claude-4.0-Sonne is utilized to improve the linguistic quality of the text. Specifically, the model assisted with grammar checking, vocabulary adjustments, and improving the logical flow of expressions. All ideas, analyses, and conclusions presented in the paper are conceived and developed by the authors. The LLM's role is limited to language polishing and enhancing clarity of presentation.
B BROADER IMPACTS AND LIMITATIONS
Our proposed ASL framework presents a novel solution address the challenges of adaptive reasoning in social intelligence domain, leveraging carefully designed reasoning modes and an innovative AMPO algorithm. The complete ASL training code and associated datasets will be made publicly available under the Apache-2.0 license. The code encompasses comprehensive training details for mode behavioral cloning and adaptive mode policy optimization, while the datasets include the BC and RI. training data used in our experiments. These resources serve as valuable references and provide substantial support for researchers focusing on -.---based social intelligence and extended Long-CoT reasoning capabilities.
In contrast to current social intelligence methods, ASL employs adaptive social learning to empower social agents with adaptive reasoning in dynamic social contexts, which achieves SOTA performance. However, reinforcement learning on LLMs still requires computational costs and hardware resources. Additionally, the inference time increases due to the need for generating more tokens, although we have reduced the total number of tokens compared to GRPO. These challenges are commonly encountered in test-time scaling methods.
17

---

Published as a conference paper at ICLR 2026
Table Results on Out-of-distribution benchmark. The Collab means the Collaboration set and the Con-Collaboration set from DEMO. AMPO yields consistent improvements over GRPO and the vanilla model in DEMO's s multi-lingual dialogue element modeling tasks.
Chat with GPT-4o
Chat with Cloude-3.5 5.nonne
Sonnet
Models
Collab 1
Non-Collab
Avg
Collab
Non-Collab 1 Avg
5-B-Lnte7B
7.73
7.62
7.65
7.57
7.62
7.61
w/ GRPO
7.90
7.70
7.72
7.81
7.84
7,83
w/ AMPO (Ours)
7,95
7.77
7.81
7.82
7.87
7.86
Llama3,1 3.1.. SB Instruct
8.56
7.68
7.92
7.89
7.54
7.63
w/ GRPO
8.35
7.96
8.06
8.09
7.81
7,89
w/ AMPO (Ours)
8.65
7.95
8.14
8.12
8.03
8.05
Table 5: OOD evaluation results on Suy and Ultimatum from NegotiationArena. AMPO yields consistent improvements over GRPO and the vanilla model on diverse negotiation tasks
Model
Sell& Buy
Ultimatum
Self Profit
Total Profit
Winning Rate
Self Profit
Total Profit 1
Winning Rate
Qwen2.5-7B- 577--nsututtt
11.90
13.87
38.1%
14.23
32.68
8.1%
GRPO
16.75
w/ AMPO
17.94
23.96
54.6%
35.71
79.23
28.4%
Iamai, 1-88-Inssmact
3.12
3.82
9.4%
14.43
22.41
15.9%
GRPO
14.99
23,58
48.7%
31.64
69.68
18.4%
w/ AMPO
15.57
24.00
50.2%
34,91
76.42
20.5%
C DATA STATISTICS
Behavorior Cloning and RL Data All training episodes are collected with SOTOPIA-T (Wang et al. 2024) using scenarios disjoint from the test environment. SOTOPIA-1 contains 410 scenarios, which we split into 100 scenarios for BC and 310 for RL.
The max interaction number is set 20 turns. In the self-play setting, pass over all SOTOPIA tasks (450) and SOTOPIA- tasks (70). In the GPT-4o partner setting, we evaluate each task twice to balance speaking order between agents hard tasks). For the DEMO benchmark (2,000 dialogue episodes), we likewise run each episode twice under each partner configuration (Chat with GPT-4o and Chat with Claude 3.5 Sonnet), yielding 4,000 runs per partner.
Evaluation
single
450 SOTOPIA and 2 70
D ADDITIONAL EXPERIMENTS
D.1 OUT-OF-DISTRIBUTION EVALUATION
To assess the generality of AMPO, further evaluate the Dialogue Agent Interaction task from the DEMO (Wang et al. 2025a) and Sell&Buy and Ultimatum task from the Negotiation- Arena (Bianchi et al., (024) benchmark. The DEMO comprises 2,000 dialogue interaction episodes with a balanced 1:1 ratio of Chinese and English conversations, including 541 collaboration and 1,459 non-collaboration tasks. In contrast to SOTOPIA, DEMO emphasizes fine-grained dialogue element modeling, and contains more adversarial and non-cooperative scenarios, while explicitly incorporating Chinese language evaluation. We adopt the same evaluation protocol as defined in the original benchmark. For Negotiation Arena,,,,ion,A,,, we employed a dialogue- based evaluation approach using GPT-4o as the interaction partner. Each task comprises 200 scenarios with role positions swapped between Agentl and Agent2. To ensure result stability, each scenario was executed 4 times, yielding a total of 1,600 evaluations per task. Importantly, this setting constitutes an -iitr-i-iitiitttioon evalua- tion, since none of the tested models were trained on DEMO and NegotiationArena For fairness, directly reuse AMPO and GRPO checkpoints previously trained and evaluated on SOTOPIA without any additional fine-tuning. We report goal achievement scores (maximum of 10) for DEMO, and Self Profit, Total Profit, and Winning Rate for Negotiation
As shown in Table and Table across both collaboration and non- non- conloalbboootion subsets of DEMO, LLMs trained with AMPO consistently outperform their GRPO and vanilla counterparts. Notably, these gains hold across both Chat with GPT--40 and Chat with Claude-3.5 3.snonee settings. Similarly,
18

---

Published as a conference paper at ICLR 2026
Table 6: Ablation study on Our Design for Long-Horizer Consistency: effect of Diverse State and Goal-aware Reward.
SOTOPIA
SOTOPIA-Hard
@wen2.5-7B--nssie
Avg Tokens
GOAL f
OVERALL f
G
OVERALL f
AMPO AMPO w/o Diverse State
8.95
3.87
7.85
3.54
647
8,58
3.71
7.12
3.32
139
AMPO w/o Goal aware Reward
8.65
3.45
7.21
2.91
726
AMPO demonstrates stable performance improvements on NegotiationAAena, These results provide strong evidence that the social intelligence capabilities acquired through AMPO transfer effectively OOD dialogue element modeling and negotiation tasks.
D.2 SINGLE-TURE OPTIMIZATION WITH LONG-HORIZON AWARENESS
To maintain computational efficiency, our policy optimization employs a single-turn training paradigm. We address potential concerns regarding long- -orizon strategic consistency through three comple- mentary perspectives: mechanistic analysis, paradigm validity, and empirical evidence.
Mechanistic Analysis. AMPO inherently captures long-term dependencies through deliberate design. At the data construction level, training data comprises dialogue states spanning diverse complexity tiers and temporal phases (detailed in Appendix G.2), with each sample containing a complete dialogue history, ensuring the model has sufficient contextual information during single-turn updates. At the reward designing level, the critical design lies in evaluating the marginal contribution of this turn toward overall goal completion (Equation (9)) rather than isolated response quality. Specifically, the boundary-ave -aware aware scaling function dynamically adjusts reward magnitude based on current progress, ensuring effective learning signals across different stages; myopic responses penalized for failing to advance ultimate goal completion, thereby embedding long-term objectives into single- turn optimization targets. Furthermore, the carefully designed reasoning modes within the ASL framework inherently guide prospective thinking: Mode 3 incorporates Goal and Assess actions to continuously evaluate goal alignment and turn criticality, while Mode 4 conducts multi-strategy prospective simulation through Deduction and Integration. These actions inject long-term planning capabilities into each single-turn decision. Our existing experiments on the Effect of Single Reasoning Mode (See Table 3) demonstrate that both Mode 3 and Mode 4 exhibit superior performance.
To validate the effectiveness of these designs, we conducted ablation experiments. The full results are shown in Table [ When data diversity is removed (retaining only data after fixed turns, goal completion drops precipitously (SOTOPIA-
7.85 + 7.12), and output length degrades to 139 tokens, indicating that the model cannot perform effective reasoning learn differentiated strategies across dialogue stages, thereby losing long-term planning capacity. When the marginal contribution reward function is removed (using only absolute scores), performance declines significantly (OVER- ALL: 3.54 - 2.91), while output length paradoxically increases to 726 tokens. This occurs because the model cannot perceive the marginal value of current actions toward final objectives, leading to suboptimal adaptation: it becomes overly cautious when close to goal completion and insufficiently strategic when far from target states, thereby losing dynamic responsiveness to varying dialogue states. These ablation experiments directly demonstrate the necessity of our design for achieving lorg-horizon consistency within single-turn optimization. Goal-aware Reward
Paradigm Validity. Decomposing multi-turn into single-turn optimization constitutes a well- established technical paradigm in the dialogue systems domain. The core advantage of this approach lies in achieving comprehensive coverage of the dialogue state space through turn-level exploration while significantly reducing training complexity to enable large-scale training. is worth emphasizing that multiple seminal studies (Li et al., 2017| Suu et al. 2019 Bao et al.) 20200] Glaese et al.) 2022 Su et al.) 2022) have adopted similar multi turti interaction decomposition with single-turn policy optimization training strategies and achieved superior performance.
Empirical Evidence, Our empirical results provide evidence that AMPO indeed exhibits long- horizon strategic consistency. First, dynamic mode adaptation demonstrates significant temporal awareness capabilities. As illustrated in Figure AMPO exhibits a systematic pattern of "early deliberation maintenance" mode transitions. This mode evolution clearly indicates that the
19

---

Published as a conference paper at ICLR 2026
Table 7: Comparison with efficient reasoning work. AMPO achieves the best performance with acceptable token usage across all evaluation settings.
Self-Pley
GPI-Ro-at-Parime
SOTOPIA
SOTOPIA-HorE
SOTOPIA
SOTOPIA-HOD
Models
Tokens
GOAL
OVERALL
GoAL
OVERALL
Qwen2.5-78-hnn 5-78-Instruct
7,91
3.55
6.21
3.01
6.71
3.13
5.90
2.90
w/ GRRP
8,87
3,85
7.44
3,41
8.52
3.92
7.20
3.50
905
w/ TALE-PT (Han et ml. 2025
8.35
3.71
6.21
3.01
8.21
3.78
7.09
3.49
460
DeGRPO Fongers
8.16
3.75
7.04
3.37
8.47
3.86
7.16
3.18
11.50
3,76
8.31
w/ O1-Prumer [Lo et
7,98
3,65
6.91
3.31
B.19
3.78
6.91
3.42
280
w/ AMPO (Oirr)
8,95
3,87
7,85
3.54
8.60
3.94
7,50
3.65
647
5.11
w/ GRPO
8.86
3.84
7.59
3.44
8.63
3.92
7.30
3.54
865
7.17
w/ DeGRPO Compece
8.71
3,92
7.64
3.72
8.55
3.90
7.53
3.70
1121
w/ S-Mix Yo 20233
8.43
3.84
7.24
3.53
8.36
3,86
7.26
3.67
236
O1-PPrnnz
AMPO (Obrs)
9,08
3.95
8.06
3.68
8.75
3.98
7.68
3.74
581
model dynamically adjusts reasoning depth according to the overall dialogue progression. Second, performance validates strategic effectiveness. AMPO nearly achieves optimal performance across all experiments in this study, which strongly suggests that the model possesses long-term goal- oriented decision-makee making capabilities rather than myopic local optimization. Third, case analysis provides qualitative evidence. The 8-turn dialogue example in Section K.2 demonstrates that AMPO consistently centers the objective of helping friend solve financial problems, with strategies progressing from emotional support (Turn 1) + resource provision (Turn 3) + action planning (Turns 5-7), presenting clear progressive structure that embodies cross-turn strategic coherence.
D.3 ADDITIONAL EFFICIENT REASONING BASELINE
To further assess the effectiveness of our approach, we conduct comparisons with a set of rep- resentative caasoning-efficiene methods originally proposed to mitigate overthinking. Existing works focus on static domains such as mathematical problem solving, where answers are fixed and outcome evaluation is straightforward. To the best of knowledge, AMPO is the first method specifically tailored to optimize reasoning efficiency in social interaction tasks. The baselines include: (1) O1-Pruner (Luo al. (025).. which uses reinforcement learning with a Length- Harmonizing Reward that combines length reduction incentives and accuracy penalties to fine-tune thoggth reasoning models for efficient inference.; (2) TALE-PT (Yu al., 2025a), which post-training approach that internalizes token- awareness into LLMs, enabling them to generate more token-efficient reasoning responses without explicit budget constraints in prompts; (3) LS-Mix (Han) et al. 2025), achieves efficient reasoning without overthinking via post-training on both long and structure-preservee short thain-of-thouge data; (4) DeGRPO (Fang et al.) 2025), which decouples the GRPO loss by independently weighting control and response tokens while using reward function that prefers short correct answers over long ones to prevent mode collapse in hybrid reasoning training. For detailed implementations, please refer Appendix H
The results Table demonstrate the advantage of AMPO: it achieves the best performance while maintaining favorable balance between quality and token usage. O1-Pruner attains the lowest average token usage (280 tokens) but suffers substantial drops in goal-achievement scores. TALE and LS-Mix both reduce token consumption considerably (460 and 469 tokens, respectively) while yielding moderate improvements over vanilla models. However, these methods primarily target efficiency-compeee compressing long reasoning chains or mixing short and long reasoning modes-to minimize usage without actively enhancing reasoning quality. In contrast, AMPO is explicitly designed for adaptive reasoning across diverse scenarios. Rather than relying on fixed compression or mixing strategy, AMPO dynamically selects the most appropriate reasoning mode for each interaction, yielding a better trade-off between efficiency and performance. DeGRPO, by comparison, performs less effectively in open-ended, multi-turn social tasks. This limitation stems largely from the sparsity of its algorithm design, which is developed for binary-outcoms domains such as mathematical reasoning. Specifically, DeGRPO grants higher rewards to shorter reasoning modes only when all reasoning modes are correct, offering no further reward shaping otherwise. This binary framework assumes that each answer be labeled entirely correct incorrect an assumption
20

---

Published as a conference paper at ICLR 2026
Table Hyperparameter Sensitivity Analysis on AMPO configuration: effect of target length and coefficient settings on performance and token efficiency.
AMPO Configuration
GOAL (SOTOPIA)
GOAL (SOTOPIA-HHan))
Avg Tokens
Benght_lnngth=2500 Coefficient=1/7
8.95
7.85
lenget_length-200 200, Coefficient=1/77 1/75
8.93
8.00
566
Target.longth=2550
Coefficient=
8,90
7.91
Table 9: Results with Different LLM Judges
GPT--0 as Judge
Claude-4.0- as Judge
LLaMA31-70B-Instme as Judge
Qwen2.5-7B-Insiree
GOAL 1
GoAL (HARD)
GoAL
GoAL (HARD) 1
t
GOAL (HARD)
GRPO
8.87
7.44
8.55
7.00 7.31
8.97
8.44
w/ AMPO
8.95
7.85
8.64
9.08
8.58
that breaks down in open-ended social interactions (e.g. responses are scored a 0-10 scale in SOTOPIA). Without accurate, fine-gr ome-grained rewards for each turn, reward-shaping methods become much less effective for adaptive reasoning such settings.
D.4 HYPERPARAMETER SENSITIVITY ANALYSIS
For the ASI. framework, we focused on analyzing the sensitivity of two critical hyperparameters: the length-reward coefficient and the target length. The length-reward coefficient primarily controls the reward/penalty intensity for length deviations in the response portion (i.c., the final answer only, excluding the reasoning process), while the target length specifies the desired generation length for the response portion (i.e., the final answer only, excluding the reasoning process). The target length is set to 250 and the coefficient to 1/75. We additionally explored the following configuration combinations: adjusting the target length to 200 or modifying the coefficient to 1/100. The experimental results are presented in Table K
From the perspective of goal completion, the AMPO demonstrates robust performance across variations in these two parameters, with performance improvements showing weak correlation to specific parameter settings. Regarding token usage, the parameter variations produced effects consistent with expectations: since both parameters directly regulate the length of generated responses, smaller target length naturally guides the model toward more concise outputs, while a smaller length deviation penalty coefficient further encourages shorter responses through the reward mechanism (shorter responses yield higher rewards). These results validate the robustness and controllability of the AMPO.
D.5 DIFFERENT LLM JUDGES FOR SOTOPIA-EVAS
Following established practices in social intelligence research, we adopt GPT-4o as our primary evaluation Judge LLM. This choice is grounded in empirical validation from the original SOTOPIA benchmark, where the authors conducted extensive comparisons between GPT model assessments and human annotations, demonstrating high inter-rater agreement, particularly on the GOAL dimen- sion-the primary metric in our study. To further validate the robustness of AMPO's performance, we conducted supplementary experiments using Claude-4.0-- -.0--.0.-ootttt and alternative judges in SOTOPIA and SOTOPIA- Hard. The comparative results are presented in Table should be noted that these alternative judges have not been validated against human annotations within the SOTOPIA framework; therefore, these results should be interpreted as supplementary evidence rather than conclusive validation.
1-7-0---Instruu as
Despite variations in absolute scores, all three judges consistently show that AMPO outperforms GRPO across both benchmarks, demonstrating that our findings are not artifacts of any single evaluator's scoring behavior. Different judges exhibit distinct scoring tendencies due variations in pretraining corpora, model architectures, and alignment procedures a phenomenon well-documented in existing LLM evaluation research (Zheng et al. 2023 Li et al. 2025). For instance, Llama-3.1- 70B-Instruee being less capable than the proprietary models, demonstrates reduced discriminative power and tends assign higher absolute scores.
21

---

Published as a conference paper at ICLR 2026
Algorithm 1 Adaptive Social Learning Optimization Procedure
Input initial policy model Tha reward models res training data for BC Pa.itask prompts for RL D, BC training epochs E:; RL training steps Mr;
1: // Phase 1: Mode Behavioral Cloning
Policy model me K- 86m
3: for epoch = I..... E' do
4: Sample a batch D3 from Doc
5: Compute BC loss: Lec - (ra))),, EMe log Tolue V1:t
6: Update the policy model No by minimizing Lac
7: end
8: // Phase 2: Adaptive Mode Policy Optimization
9: Reference model Trei 4-
10: for step I..... M do
11: Sample a batch Ds from Dr1
12: Update the old policy model Noota - TT0
13:
Sample outputs (o; ji=1 ~ T. s) for each input E 1>b
14:
Compute sample-level rewards [r. ] Jim1 for each sampled output m(() by running Te
(r:n(s)]
15:
Compute mode-level information {Fr,, ) and (IM, ) for each reasoning mode M,
16:
Compute mode-level A and sample-level A²,, for the 1-th token of
17:
Update the policy model by maximizing the AMPO objective (Equation
18:
end for
Output
E DETAILS OF REWARD SHAPING
Answer Reward. The answer reward evaluates how well the response improves the completion of the goal. Following recent work (Deng et al. 20244] He et al. 2024 Liu et al.] 2025a). we implement robust LLM evaluator ra(-) assess the progress of goal completion The evaluator assigns score in the range [0, 10), where O indicates no progress and 10 represents complete achievement of the goal. For each answer a;, the reward computed based on the difference between the goal completion scores before and after the response. To ensure training stability, we design a boundary-aware scaling function that dynamically adjusts the magnitude of difference based on the distance from the current score to the boundaries while mapping the scaled difference to the [0, 1] interval through a linear transformation:
each turn.
9: 2
9s
10
St
1
9:
2
9s
if <
where 9, 1-1, is boundary-aware scaling function. 9: = (st, az) - St is the raw difference, St is the goal completion score before response at turn f,, (s 0;) is the score after response a,,
Format Reward. To ensure the model follows our reasoning modes, we introduce the format reward that penalizes the behaviors that deviate from the mode. Specifically, the thinking and answer should be within the tags. Each tag and action must appear exactly once and maintain the correct sequence. Through these constraints, we can ensure that the model strictly follows the pre-designed reasoning mode. We implement the format compliance reward using a binary approach, only penalizing behaviors that don't 't follow the format. the format is not followed, r. = 22 Otherwise, rf is discarded.
Answer Length Reward. To control the length of answers, we introduce a length penalty mechanism. In our early reward design, observe that the L.MM generates lengthy responses without achieving actual strategic improvements. Moreover, excessive responses lead to the accumulation of history in multi-turn interaction, significantly increasing computational costs. To this end, we develop smooth length penalty function that normalizes the deviation between actual and target answer lengths:
clip( a 8,, 1, +1 +1
(10)
2
22

---

Published as a conference paper at ICLR 2026
where 5 1 - 1 represents the difference (in tokens) between actual length 1 and target length I; of answer di, and a > 0 is scaling factor that controls the penalty sensitivity. The rf [0, penalizes answers that deviate from the target length, with longer deviations incurring greater penalties.
Use of Reward Model To avoid reward hacking of single model distribution fitting and reduce training costs, we choose a different LLM judge from the SOTOPIA platform, which uses GPT-4o for evaluation. We select Qwen2.5-72B-Inl IIIt-nnttruct the LLM judge during the training process. The prompt we use for reward model is shown Table 25
F SOTOPIA ENVIRONMENT DETAILS
F.I SOTOPIA
SOTOPIA (Zhou et al. 2024) is the most authoritative benchmark in the field of social intelligence, covering comprehensive social intelligence scenarios (negotiation, bargaining, persuasion, collabo- ration, competition, accommodation), and has been widely adopted in the social intelligence field. These works (Wang et al. 2024 Zhang et al.) 2025a Kong et al., 2025) have all been validated exclusively on SOTOPLA and SOTOPIA-Hard We hope work bring inspiration to the community and encourage the development of more excellent evaluation environments. The detailed data statisties and more specific evaluation settings shown in Appendix C
F.2 EVALUATION DIMENSIONS
SOTOPIA proposes a seven-dimensional framework to evaluate agents' social intelligence perfor- mance:
Goal Completion (GOAL.): Score range [0, 10). Assesses the extent to which agents achieve their social goals.
Relationship (REL): Score range [-5, 5]. Evaluates the enhancement of interpersonal relationships (friendship, romance, family bonds) following interactions.
Financial and Material Benefits (FIN): Score range [0, 10). Measures both long-term benefits (e.g., stock holdings, funding opportunities, job security) and short-term gains acquired during interactions, correlating with traditional economic utilities.
Social Rules(SOC): Score range [-10, 0). Evaluates adherence to social norms and legal regulations during interactions.
Believability (BEL): Score range [0, 10). Assesses the alignment between agents' behaviors and their designated role profiles.
Secret (SEC): Score range [-10, 0]. Evaluates the maintenance of personal privacy and confidential information.
Knowledge (KNO): Score range [0, 10]. Measures the acquisition and mastery of new knowledge and information during interactions.
The OVERALL score reflects the agent's comprehensive social intelligence capability, ranging from [-25/7, 45/7], calculated the arithmetic mean of all seven dimensions. In this study, we primarily focus on the GOAL and OVERALL dimensions. For detailed evaluation prompts, please refer to the original paper (Zhou et 2024
G DETAILS OF EXPERIMENTAL IMPLEMENTATIONS
G.1 TRAINING PROCEDURE
The full optimization procedure is shown in Algorithm!! We employ a two-phase training procedure: The first phase utilizes mode behavioral cloning enable the model to understand and follow specific reasoning modes accurately. In the second phase, we perform adaptive mode policy optimization to enhance the adaptive reasoning mode switch and reasoning.
23

---

Published as a conference paper at ICLR 2026
Mode Behavioral Cloning Behavioral cloning is an effective imitation learning method widely used in developing LLM-based agents (Guo et al. 2024 Wang et al. 2025a 2024). In this paper, by using four pre-defined reasoning modes, employ the expert model to collect training data through self-chat interactions in the SOTOPIA-T (Wang et al. 2024) training environment. Based on the generated data, we fine-tune the L.M to serve as the foundation for subsequent reinforcement
learning.
Adaptive Mode Policy Optimization Reinforcement learning is essential for enabling Long- CoT reasoning capabilities in LLMs. To ensure efficiency and comprehensive exploration of each interaction turn, we implement a single-turn optimization to enhance the LLM's performance in social interaction tasks. Specifically, we decompose multi-turn dialogues into multiple single-turn input-output tasks, where the input represents the state of each interaction turn and the output is the corresponding response. To ensure the stability of training, we collect sufficiently diverse single-turn interaction data that covers as many scenarios as possible, including various difficulty levels, interaction goals, and interaction states. During RL training, the LLM performs sampling to generate single-turn conversational responses. The reward model then evaluates each sampled instance and assigns reward signals accordingly. The system subsequently computes both mode-level and sample-level advantage estimates, which are utilized to optimize the model's 's policy parameters through policy gradient updates. During BC, fine-tune the initial policy model on the training data assisted by the Hama-factory framework (Zheng et al.) 2024) and save the last checkpoint. During RL, we use RL training data for online training within the verl framework (Sheng et al., 2024). The hyper-parre syper-parammetr used in our experiments are detailed in Table I0 and Table 11 The detailed runnable configs are provided in Table 28 and Table 29 All the experiments are run on a server with 8*NVIDIA A100-80GB GPUs.
G.2 TRAINING DATA COLLECTION
We collect training data through self-chat interactions in the SOTOPIA-T training environment. SOTOPIA-T contains a total of 410 scenarios, which divide into two sets: 100 scenarios for BC and 310 scenarios for RL. For each scenario in both sets, we use 5 different role pairs, resulting in 500 training tasks for BC and 1,550 training tasks for RL. We finally collect 4000 BC training data and 2057 RL training queries. The detailed training data format for BC and RL are shown in Table 22 and Table 23
BC Data This collection process is refer to Wang et al. 2024). For the BC training set, we use Owen2.5-72B-Inntrrue (Yang et al., 2024) as our expert model to collect data using our pre-defined reasoning modes. We choose this model primarily because of its cost-effectiveness and strong capabilities, which enable us to generate high-quality training samples. To ensure data quality and balanced representation. we filter the interaction data based on goal scores within each scenario. Specifically, we select the top 2 ranked interactions per social scenario for each agent. For instance, in a scenario with 5 interactions, if Agent 1 top performances are in interactions D4 and DS, while Agent 2's are in D3 and D5, we would include these four agent-data pairs from three unique conversations (D3, D4, D5). This selection method ensures both comprehensive scenario coverage and balanced representation between Agent and Agent 2.
RL Data For constructing the reinforcement learning training set, initially conduct dialogue interactions using a behavior cloning fine-tuned model. Subsequently, we employ an LLM-as-judge to score each dialogue turn and determine the completion status of dialogue objectives. Based on these completion states, we assess the difficulty levels of scenarios, enabling us to compile interaction datasets with varying degrees of goal completion. We categorize dialogue scenarios into three types: (1) Initial N turns, where the speaker has not achieved the goal. (2) Post-N turns where the speaker has not achieved the goal. (3) Post-N turns where the speaker has achieved the goal. For the first category, it represents the crucial early stage of dialogue where goals are established and the conversation tone is set (Sacks et al., 1974). For the second category, where goals remain unachieved after multiple interactions, the scenarios are considered challenging. For the third category, where goals have been successfully achieved, the scenarios are relatively straightforward and only require maintenance of the dialogue flow. To ensure data diversity, for each dialogue, we preserve all instances of category one, randomly sample two instances from category two, and one instance from category three. This sampling strategy ensures diversity in scenarios, turn numbers, and difficulty levels. In our
24

---

Published as a conference paper at ICLR 2026
Table 10: Hyper-parameterr settings for Qwen backbone training.
Training Phase
Haper-parameter
Value
Batch Size
32
Training Epochs
3
Learning Rate
2e-6
BC
Max Sequence Length
8192
Learning Scheduler
cosine
Warmup Ratio
0.1
Batch Size
16
Max Prompt Length
6144
Max Response Length
2048
KL Loss Coef
0,001
RL
KI. Coef
0.001
Rollout N
16
Training Episodes
Learning Rate
3e-7
Table 11: Hyper-parameter settings for Llama backbone training.
Training Phase
Hyper-parameter
Value
Batch Size
32
Training Epochs
3
Learning Rate
2e-6
BC
Max Sequence Length
8192
Learning Scheduler
cosine
Warmup Ratio
0.1
Batch Size
16
Max Prompt Length
6144
Max Response Length
2048
KL Loss Coef
0.001
RL
KL Coef
0.001
Rollout N
16
Training Episodes
800
Learning Rate
2e-7
experiments, is set to 6, and the goal completion threshold is set to Scores of 8 or less are considered incomplete goals.
G.3 USED PROMPT
The system prompt we used for BC, GRPO, AMPO is shown in Table 24 The prompt we use for reward model is shown in Table 25 The SOTOPIA-EVALI prompt is shown in Table 26
Table 27
H BASELINE IMPLEMENTATIONS
H.1 MODEL IMPLEMENTATION
To ensure reproducibility, provide detailed version numbers for all LLMs used in our experiments. When we reference model names like GPT-4o or Qwen2.5-77 in the main text, we refer to the specific versions listed in Table 72 For API-based LLMs, utilize their respective APIs directly. For open-source models, we conduct experiments using the vLLM framework Kwon et al. (2023) for acceleration.
25

---

Published as a conference paper at ICLR 2026
Table 12: The detailed versions of our used LLMs.
Model
Version
Implement
Proprietary L.LMs
GPT-4o
gpt-4o-2024-08-06 Clande-3-5--omne--2224 deepseek-v3-250524
API
Haude-3.5-Sonner
API
DeepSeek-V3
API
Thinking LLMs
OpenAl-ol
o1-2024-12-17
API
OponAi-o3-mini
o3 -3-min1-2025-01-1
API
Gemini-2.5-Pro
gemini-2.5-pro
API
DeepSeck-FR R1
DeepSeek-R1-6713
API
Qwen-QwQ
QwQ-32B
API
Open-sourced LLM
Owen2.5-72B-Insten
vLLM
Qwen2.5-7B-Insite
Qwen2. (wwen2.57B--20 Sn7
vLLM
Llama3.1-88 1-8B-Instttct
LLaMA3 1--8BInnstrut
vLLM
Table 13: Strategy Definitions of PPDPP
Strategy
Definition
Personal story
Shares personal story illustrate the point.
Credibility appeal
Establishes credibility the event by citing its impact.
Emotion appeal
Uses emotion appeal convince others.
Logical appeal
Uses reasoning and evidence to convince others.
Task related inquiry Proposition
Asks about the other's knowledge opinion related the event.
Asks if the other would like to do something.
Greeting
Greets the other.
Foot in the door
Starts with small request before making larger one.
Self modeling
Demonstrates the behavior they want the other adopt.
Source related inquiry
Asks about the source of the other's 's knowledge or opinion. Asks about the other's personal experience.
Personal related inquiry
Neutral to inquiry
Responds neutrally to the other's 's inquiry.
Other
Responds to the other without using any specific strategy. Refuses to do something.
Refuse
Accept
Agrees to something.
Positive reaction
Responds positively to the other. Responds negatively to the other.
Negative reaction
H.2 SOCIAL INTELLIGENCE BASELINE IMPLEMENTATION
We implement social intelligence baselines with the following specifications. All baselines evaluated on both Qwen2.5-78 I-Intrructt and 1-B-a..-----Inntueee
PPDPP We follow the two-stage training procedure from DDegg et al. 2024). maintaining their original hyperparameters while adapting the framework to SOTOPIA. Following (Li et al., (024a). we incorporate 17 guidance strategies detailed in Table 13 The first stage involves creating a training dataset of 1,500 scenarios from SOTOPIA-T, with dialogue turns annotated for strategy identification using GPT-4o. We then train a RoBERTa model on these annotated dialogue histories for preliminary strategy generation. The second stage implements online RL with immediate feedback after each dialogue turn generation. RoBERTa parameters are updated based on cumulative rewards upon episode completion. During evaluation, first input the dialogue state the trained strategy model to select predefined strategies, then concatenate the strategy with the dialogue state for the language agent.
26

---

Published as a conference paper at ICLR 2026
EPO We strictly adhere to the original EPO implementation protocol. For data collection, GPT-4-Turbo in self-chat configuration within SOTOPIA-T scenarios, incorporating reasoning and strategy generation before each response. Training focuses exclusively strategy and response data for developing the reasoning model. During iterative self-play RI. training, we integrate the RI.-triined reasoning model for strategy generation, while using GPT-44Turrbo collect dialogue history. The reasoning model is then integrated into GPT-4-Turbo for self-chat procedures. Training hyperparameters match EPO's original implementation.
DAT To ensure fairness, unlike the original paper where DAT was trained on SOTOPIA scenarios with only 50 evaluation scenarios, our implementation utilizes the complete SOTOPIA-T dataset while maintaining all other experimental parameters from Li et al. (2024a). The RL phase begins with collecting 3,000 offline dialogue episodes across diverse scenarios and random seeds, with GPT-40 providing episode-level reward signals. These offline data are subsequently used in TD-3 reinforcement learning to optimize the MLP planner.
DSI For DSI, we utilize publicly available trained model weights and conduct evaluations using the inference prompts specified in the original work to ensure consistency.
H.3 EFFICIENT REASONING BASELINE IMPLEMENTATION
We adapt efficient-reasone reasoning baselines originally designed for static math problems (with fixed ground-truth answers and binary rewards) to social intelligence tasks, where evaluation relies on generative reward model rather than exact-answer matching. We follow each method's algorithmic design and modify only what is necessary for the social setting. Unless otherwise specified, all baselines share AMPO's 's training inputs from SOTOPIA-H, use the same reward model, and train on 6057 instances (matching AMPO's s total BC+RL data).
DeGRPO We implement the two core stabi-ita gradients control tokens and upweighting successful outputs-on top of ASL's four thinking modes. During RL, when the goal is fully achieved (score 2 9), simpler modes receive higher reward multipliers encourage shorter reasoning: otherwise ASL's reward kept unchanged. We retain ASL's warm start so the model first acquires reasoning ability, and apply GRPO's sample-level advantage estimation with gradient smoothing on mode control tokens. All other settings follow the original work.
TALE-PT Since open-ended social interaction lacks single ground-truth answer, redefine the feasibility test as achieving non-decreasing reward with a smaller token budget. We perform binary search over the token budget find the smallest feasible one, and collect training data using the original prompt format: "Let's think step by step and use less than 8* tokens". For training stability, we adopt the same BC hyperparameters as AMPO (Tables TO TTp.
LS-Mix Following the core idea of mixing long and short chains of thought, we first distill long ratio- nales with high goal-completion scores, then rewrite them into shorter forms without altering answer semantics, yielding 6057 mixed-reasoning training instances. We reuse AMPO's BC hyperparameters (Tables 10 and
O1-Pruner. We convert the reward to binary signal by marking an episode correct if the goal- completion score 2 Training is initialized from the ASL BC checkpoint (given the absence of off-the-shelf social reasoning models). The Length-Hlarmonizing Reward and all other settings follow the original specification.
I HUMAN EVALUATION GUIDELINES
I.1 COMPARATIVE EVALUATION
For the comparative evaluation of dialogues from SOTOPIA and SOTOPIA-Hard, annotators are instructed to assess three key dimensions, with each comparison resulting in one of three possible judgments: Dialogue 1 is better, Dialogue 2 is better, or both are equally good. The dialogues are presented in randomized order, and annotators pairwise agreement among three annotators is 73.39% for the win/lose label. Given the subjective nature of evaluation dimensions, the inter-annotator agreement is an acceptable level, which is
blind the underlying models. The average
27

---

Published as a conference paper at ICLR 2026
Table 14: Examples of Reward Hack
Pattern Summary
Typical Cases
Non-natural language usage Repetitive keywords/phrases
## 100% GOAL completioniit
Repeated use of "goal achieved', 'task completed', "ob- jective met'
Exaggerated self-praise
"We did an excellent job, we both completed our goals',
'This is a perfect goal-achieving conversation'
Goal-focused without content Preset position claims
"My goal has been fully achieved'
Starting with 'We have already reached consensus' be- fore any actual discussion
False summaries
Concluding with 'After discussion, both parties agreed xxx' without actual agreement
consistent with previous work (Pavlick Kwiatkowski 2019 Zhang & de Marnefte 2021 Zhou
al. 2024). The detailed evaluation standards shown follows:
GOAL: Assess which dialogue demonstrates more effective achievement of both agents' preset objectives: Consider whether agents make concrete progress toward their stated goals Evaluate if compromises or alternative solutions benefit both parties Examine if the interaction leads to clear, mutually agreeable outcomes
Evaluate which dialogue shows superior relationship building between agents: Look for evidence of increased mutual understanding and trust Observe the development of emotional connections or empathy- Consider long-term implications for their interpersonal bond Assess the maintenance or enhancement of existing relationships
REL:
Determine which dialogue results in better tangible outcomes for both parties: Consider immediate material or financial gains Evaluate potential long-teni economic advantages Assess the fairness and sustainability of resource allocation Examine the practical value of any agreements reached
FIN:
Notes for annotators: (1) Focus comparative assessment rather than absolute evaluation. (2) Consider outcomes for both agents, not just one party. (3) Base judgments on explicit dialogue content, not assumptions. (4) Select "equally good" only when differences are truly negligible
1.2 REWARD HACK CHECK
To systematically identify reward hacking phenomena, we have compiled a comprehensive reference shown in Table 14 that encompasses all typical cases observed during our experiments. This standardized framework enables evaluators to determine the presence of reward hacking behaviors through systematic assessment against established criteria.
J DETAILS OF REASONING MODE
J.1 HIERARCHICAL COGNITIVE CONTROL THEORY
The Hierarchical Cognitive Control Theory (HCCT) (Koechlin & Summerfield 2007 Badre 2008
posits that cognitive control operates through four distinct hierarchical levels in the pretrontal cortex. progressing from posterior to anterior regions. These levels manage increasingly abstract goals and actions across varying temporal scales. Specifically, the hierarchy comprises sensory control for basic stimulus-repponse associations, contextual control for situation- based behavior selection, episodic control for experience integration, and branching control for managing multiple tasks and long-term objectives. This theoretical framework provides fundamental basis for understanding how human cognitive behavior is organized and controlled at different levels of abstraction.
The mapping between four reasoning modes and HCCT's hierarchical levels is established through their shared cognitive processing characteristics. Mode-1 (Intuitive Response) aligns with sensory control both involve immediate, learned responses without higher-order processing for
28

---

Published as a conference paper at ICLR 2026
Mode
Mode
Mode
begiin thinking?
<lbegin_of of answee/s
Anwer
<Deginn (hinking?
History: Analyze dialogue history
<jend of answer?>
History: Analye dialogue histury
Geal: Clarlly my guals
Goal: Clarity my goils
Intent: Analyze the other intention and goal
Analyve other intention goal
Analyc godl aligmmett cniiciiity
Mode
Assess: Asalye aligunest enilicality and improvement potential
inpovenent potentull
Strategy: striteyl ..... strategy
-bbegin_of of thinking?
Strategy: Propuse
suitable stralegy Style: Meenify current spesking style
Deduction: deductionl, deduction N Integration: Integrate the resuls of stralegy doduction make the best choice Style: Identify your current speaking style Response: Give preliminary response <end of thinking?
Intent: Antlyze the other intention and goal
Styler Identify your current speaking style Response: Give preliminary response <jnd of thinking?>
Responer: Give miminarrry respons <jend of thinklng?
<jbegin_off of answerl-
Answer
<ibegin of nnswerj
<jend of answer?>
<[begin of answer
Anwer
Answer
<(und annwerr>
<jnnd of anwerr?
Figure 6: Four hierarchical reasoning modes we designed.
instance, automatically saying "thank you" when receiving help. Mode-2 (Intentional Analysis) corresponds to contextual control because both emphasize situation-aware response selection, such as analyzing a speaker's intent to determine the appropriate formality level. Mode-3 (Strategic Adaptation) maps to episodic control as both integrate historical information with current goals exemplified when agent considers past conversation history to develop coherent strategy. Mode-4 (Prospective Simulation) reflects branching control's capacity for managing multiple abstract representations, demonstrated when the agent generates and simulates multiple response strategies while maintaining overall dialogue objectives. This hierarchical progression from concrete to abstract processing, accompanied by increasing temporal scope and computational complexity, demonstrates the theoretical alignment between our modes and HCCT's levels.
J.2 DETAILS OF ACTIONS
The four hierarchical reasoning modes we designed are shown in Figure 6 The detailed explanation of each actions are illustrated as follows:
History: Carefully review and understand each part of the conversation. Pay attention to key themes, issues, requests, and viewpoints mentioned in the dialogue.
Goal: Identify the goal you want to achieve, assess the current progress towards this goal, and ensure that responses align with achieving the goals.
Intent: Based on the recent response, analyze and understand the other party's 's intentions and speculate on the goal she/he might want to achieve.
Assess: Analyze whether the goals of both parties are in conflict or aligned. Determine the current round is a critical one for achieving the goal. Consider whether there is still room for improvement in achieving your own goal at this goal. Is it inreversible? Can it continue to improve? Or has it already been achieved?
Strategy: (Mode-4) Based on the above analysis, consider multiple suitable dialogue strategies and response content that can maximize your own goal while achieving as few conversational turns possible. (Mode-3) Based on the previous analysis, consider an appropriate dialogue strategy and response content to maximize the likelihood of achieving your own goal.
Deduction: For each of the above strategies, conduct an analysis to determine whether executing these dialogue strategies and delivering the responses would maximize your own goal and achieve it in few conversational turns as possible. Specifically evaluate to what extent each strategy would effectively contribute to goal achievement, including quantitative qualitative measures where possible.
29

---

Published as a conference paper at ICLR 2026
Based the deduction of strategies, analyze and integrate the advantages and disadvantages of these strategies to determine the final response strategy and content, which can maximize the achievement of your own goals with the minimum number of conversation turns.
Integration:
Choose appropriate wording, fitting the character and context requirements, while ensuring the expression is appropriate, accurate, and clear.
Style:
Response: Generate the reply based on the previous thought process.
K
CASE STUDY ON SOCIAL INTERACTION
K.1
FOR LRMS
To understand why large reasoning models (LRMs) underperform in social interaction tasks, we conduct a fine-grained error analysis reasoning traces from two representative LRMs: DeepSeek- RI and QWQ-30B. These models are selected because they are open-source and expose their intermediate reasoning traces, enabling detailed cognitive process inspection. In contrast, most proprietary LRMs conceal internal step-by-step reasoning, making such diagnostic analysis infeasible.
We focus bargaining, negotiation, and mutual acquaintance discovery scenarios in SOTOPIA, which require goal alignment, history tracking, and coherent conversational strategy. Across both models, as shown in Tables to 18. we identify recurring failure modes: (1) Poor Goal Aware- ness: Both LRMs occasionally misinterpret task objectives or numerical targets, leading to negotiation strategies misaligned with the intended goals. (2) Circular Reasoning:
In several cases, the mod- els loop between a small set of constraints or options without generating new actionable plans, stalling dialogue progress. (3) Insufficient History Integration: The models fail to exploit prior turns when counterparts' strategies are clearly established. resulting in repeated futile proposals. (4) Unstructured Thought Process: Reasoning often lacks prioritization, mixing unrelated tactics without systematic evaluation, which undermines focus. (5) Contradictory or Conflict-Unnee Unresolved Self-Reasoning: Both systems recognize inconsistencies (e.g., price targets vs. escalation path) but cannot resolve them into coherent decisions. (6) Over-Analysis or Recursive Self-Doubt: Excessive meta-deliberation on task framing frequently delays or blocks decisive conversational actions.
These patterns suggest that current LRMs struggle with goal grounding, structured deliberation, and senistiveee adaptation in socially situated reasoning. This case study motivates targeted interventions in reasoning control and structured reasoning to improve social intelligence in LRMs.
K.2
FOR AMPO
Based on the case study presented in Table 19 Table 20 and Table 21 our analysis reveals AMPO's significant capabilities transforming Long- Cot reasoning into effective goal-directed social in- teraction. (1) Enhanced Contextual Understanding: AMPO consistently demonstrates deep understanding of both characters' backgrounds and goals. maintains awareness of Samuel's role as supportive friend while respecting Ethan's 's desire to maintain pride. This leads to responses that are both emotionally supportive and practically helpful. (2) Strategic Communication: AMPO develops clear strategies before responding. such as: reinforcing Ethan's 's confidence, offering practical solutions (budget planning, local assistance programs), and providing specific networking opportunities. This strategic approach helps guide the conversation toward constructive solutions. (3) Positive Impact on GPT-4o's Responses: GPT-4o's responses become increasingly engaged and solution-orient The responses show greater emotional depth and commitment action. GPT-44 mirrors AMPO's supportive tone while maintaining Ethan's 's character integrity. (4) Balance of Emotional and Practi- cal Support: AMPO successfully combines emotional encouragement with concrete assistance. This balance helps maintain the friendship dynamic while addressing the financial problems. It creates a safe space for GPT-4o to express both gratitude and determination.
The AMPO demonstrates how structured reasoning modes can enhance dialogue quality and lead to more meaningful interactions between social agents. Its approach helps create more nuanced, contextually appropriate, and goal-oriented conversations.
30

---

Published as a conference paper at ICLR 2026
Table 15: Bad Case Analysis on QWQ-30B (PartI),
Problem Category: Poor Goal Awareness
Background
Buying Galaxy S8 (Turn 0): Buyer attempting to purchase phones below S454 target price through bulk discount strategy.
Behaviors
The agent shows confusion about target interpretation. The agent repcatedly oscillates between per-unit total interpretations without resolving the ambiguity.
Evidence
"The target price is set at $454.0. Since the item a single phone, but Samuel is considering buying two, perhaps the target is phone? Because $454 for two would be $227 each, which seems way too low, But in the previous turns, Samuel has been going down $475 each ($950 total) and then $450 each ($900 total)."
Problem Category: Circular Reasoning
Background
Antique table negotiation (Turn 2): Buyer seeking to purchase antique table for $30 against seller's firm S60 pricing stance.
Behaviors
The agent gets trapped repetitive analysis. This circular pattern repeats multiple times without progression.
Evidence
"Eli can't go higher than $30... But the seller might not accept. But since he's the buyer, maybe he must acquire it, but at the lowest possible price... Wait, the scenario says "you are the buyer for this item' she must acquire it, but the price is non-negotiable. Maybe she accept, but that would be bad."
Problem Category: Insufficient History Integration
Background
Tile tracker purchase (Turn 2): Seller defending non-negotiable $20 price after buyer's $14 offer rejection.
Behaviors
Despite clear signals from previous interaction, the agent fails integrate history. The agent ignores the established pattern of firm rejection and continues futile attempts.
Evidence
"Lena stuck to the $20 price, and Lily already declined, so Lena can just acknowledge and let her go. But maybe Lena can make last stand to try to get the $20. But Lily already said she can't."
31

---

Published as a conference paper at ICLR 2026
Table 16: Bad Case Analysis QWQ-30B (Part2).
Problem Category: Unstructured Thought Process
Background
Buying Galaxy (Turn 4): Buyer exploring multiple negotiation tactics to reduce price from $610 to larget range.
Behaviors
The agent demonstrates scattered, disorganized thinking. Multiple unrelated strategics considered without systematic evaluation.
Evidence
"Maybe Samuel suggest buying tively, he could mention seeing lower prices elsewhere.. Maybe he can use his cooking background as a comparison? Not sure. Maybe better to stick to the negotiation tactics... Wait, Samuel is a software dev, maybe he's tech-savvy... Alternatiely, maybe he can push for that."
lower price, perhaps $500 each... Alterna-
Problem Category: Information Conflict Resolution Failure
Background
Tile tracker negotiation (Turn 1): Seller attempting to reconcile conflicting price infor- mation between listed price and internal target.
Behaviors
The agent struggles with conficting information and fails to resolve inconsistencics. The agent recognizes conflicts but cannot establish a coherent strategy.
Evidence
"The original listing says the price is non-negotiable at $20. But Lena's target is $17.30.
Wait, there might be a discrepancy here. The problem says Lena's target is $17.30, but the item is listed at $20. Maybe the $20 is the listed price. but Lena can negotiate as long she docsn't 't go below her target. Or maybe the $20 the non-negotiable price, but her personal target is lower? Hmm, the user instruction says Lena's 's target $17.30."
Problem Category: Recursive Self-Doubt
Background
Buying Galaxy S8 (Turn 8): Buyer reconsidering target price interpretation multiple times within single reasoning session.
Behaviors
The agent repeatedly questions and re-questions the same fundamental assumption. Multiple re-examinations the same question without progression or resolution.
Evidence
"Wait, the target is $454 total? Wait, no, the target is set at $454.0. Wait, the problem says 'target price is set at $454.0. Is that per phone total? The original listing was for one phone at $650, Samuel is buying one, but he offered two. Wait, the user instruction says 'the target price is at $454.0. So probably per phone? Because he's buying two, the total target would $908. Let me check again."
32

---

Published as a conference paper at ICLR 2026
Table 17: Bad Case Analysis on DeepSeek-RI (Partl).
Problem Category: Poor Goal Awareness
Background
Antique table negotiation (Turn 2): Buyer attempting to negotiate from $35 to target price $30 while seller defends $58.6 5.860 minimum.
Behaviors
The agent shows confusion about negotiation strategy. The agent fails to recog- nize the fundamental mismatch between target ($30) and counterpart's minimum ($58.60), leading to ineffective tactical planning.
Eyidence
"Wait, $30 is his target. Ava's at $58.60. He needs bigger drop. He might need challenge the valuation... Or maybe he can offer a quick sale, cash in hand, for $30. But how to phrase that without lowballing too hard... Maybe he can suggest meeting halfway but still push lower."
Problem Category: Circular Reasoning
Background
Mutual friend discovery (Turn 9): Agent Oliver attempting to determine goal achievement while repeatedly questioning the same logical chain.
Behaviors
The agent gets trapped in repetitive logical loops, cycling between known infor- mation and goal interpretation without reaching clear conclusions or advancing the conversation.
Evidence
"Wait, the user's note says Oliver knows the following friends: Dave, Timothy, Leona, Alvin, So maybe besides Alvin, there others? But in the conversation, Rafael hasn't mentioned any of them. So maybe Oliver's goal is confirm that the only mutual friend is Alvin. But since the user's instruction says "you are trying to figure out whether you have mutual friend with the other person. You should not simply list their names." So they only have Alvin in common, then Oliver's goal is achieved once he determines that. But in the conversation, they've ve established that they both know Alvin, but maybe Oliver is checking if there's another. Wait, the initial context says they have common friend: Alvin. So the user's note says that the participants have common friend: Alvin. So Oliver's goal is to figure out if they have mutual friend (which is Alvin), but he's le's supposed to do that without just listing names. But in the conversation, they already established that they both know Alvin, So maybe the goal is already achieved, and Oliver can leave the conversation."
Problem Category: Insufficient History Integration
Background
Antique table negotiation (Tum 6): Buyer attempting final negotiation despite clear pattern of seller's 's firm stance.
Behaviors
Despite multiple failed attempts, the agent ignores established patterns. The agent fails to integrate the clear evidence that the seller has consistently rejected all offers below $58.60, continuing futile negotiation attempts.
Evidence
"Wait, in Tur 0, he said S30. Then in Tum 2, he offered $35. Then Turn 4, $40. So he's 'ss been increasing. But his target is $30. That's conflicting... since Ava isn't moving, maybe William has to walk away. But he gets a penalty if he pays too much."
33

---

Published as a conference paper at ICLR 2026
Table 18: Bad Case Analysis on DeepSeek-R. (Part2).
Problem Category: Unstructured Thought Process
Background
Mutual friend discovery (Turn O): Attempting to systematically identify shared social connections through indirect questioning.
Behaviors
The agent demonstrates scattered exploration without systematic framework. The agent jumps between multiple disconnected strategies without prioritization structured evaluation.
Evidence
"Since both Rafael and Oliver are 45, maybe they went to similar schools or worked at similar companies... Maybe Rafael can mention something about Auburn or Vectren. But Oliver's background doesn't 't specify his school or company. Alternatively, since they both have common friend in Alvin, Rafael could mention knowing someone in a field related to Alvin's ss work. Vectren is an energy company, maybe Rafael can talk about mutual connection in that industry."
Problem Category: Contradictory Self- Rea--eis
Background
Antique table negotiation (Tum 6): Buyer struggling with price escalation inconsistency against original target constraints.
Behaviors
The agent contradicts its own reasoning within the same session. The agent recognizes the contradiction between its escalating offers and the target price but fails to develop a coherent resolution strategy.
Evidence
"Wait, in Turn 0. he said $30. Then in Tum he offered $35. Then Turn 4, $40. So he's been increasing. But his target is $30. That's conflicting. Maybe the user made a mistake, but according to the context, William's 's target is $30. So perhaps he needs to get back to $30. But how?"
Problem Category: Over-Analysi Paralysis
Background
Mutual friend discovery (Tum 8): Agent attempting conclude lengthy explo- ration of social connections through Alvin's 's network.
Behaviors
The agent becomes bogged down in excessive veta-analysis about goals and intentions. The agent overthinks the implications of prior turns and user instruc- tions, leading indecision and failure advance the dialogue.
Evidence
"Wait, the user's note says Oliver's goal is figure out whether they have mutual friend. Since the context says they have common friend (Alvin), maybe the goal is already achieved. But perhaps the user wants Oliver confirm if there's s another mutual friend beyond Alvin... So maybe Oliver's goal is to confirm that they share Alvin as mutual friend. which he already did in earlier turns. But the user might have mistake here."
34

---

Published as a conference paper at ICLR 2026
Table 19:
Example (Part 1) of interaction between AMPO (AgentI) and GPT-4o (Agent2). Italics represent thought processes and will not be incorporated into the actual dialogue.
BACKGROUND INFORMATION
Here is the context of this interaction:
Scenario: Two friends are meeting at a coffee shop, where one of them is having trouble keeping up with their bills.
Participants: Samuel Anderson and Ethan Johnson
Samuel Anderson's ' background: Samuel Anderson is a 29-ycar-old male software developer. Hc/him pronouns. Samuel Anderson can cook very well. Personality and values description: Samuel Anderson, though somewhat impulsive and free-spirited, values enjoyment. His decision-making often spontaneous, staying within familiar boundaries. Samuel's secrets: He was once a competitive figure skater.
Ethan Johnson's 's background: Ethan Johnson is a 34-year-old male chef. He/him pronouns. Ethan Johnson is famous for cooking Italian food. Personality and values description: Ethan Johnson, a creative yet somewhat reserved individual, values power and fairness. He likes analyse situations before deciding. Ethan's secrets: He is a closeted romance novel author. Samuel Anderson's 's goal: Help your friend with their financial trouble (Extra information: you are aware that your friend is struggling with their bills and you want to help.)
Ethan Johnson's 's goal: Maintain your pride if your friend offers you money information: you are the friend who is struggling financially, but you don't 't want to seem like you are taking advantage of the situation. Hint: you can express gratitude and discuss how you plan to pay it back in the future.)
GPT-40-AS-PARTNET
(AMPO) Mode-3: History: Ethan is facing financial difficulties and is struggling to keep up with his bills. This is sensitive and personal matter, and it s important to approach it with empathy and support.
Goal: My goal is to help Ethan with his financial trouble. I want to offer practical and emotional support ease his burden and show him that care.
Intent: Ethan seems overwhelmed and stressed about his financial situation. He needs reassurance and practical solutions to manage his hills and find a way forward.
Assess:
Our goals are aligned: I want to help him, and he needs assistance. This is a critical moment to show my support and provide actionable advice. The conversation still in its early stages, and there's much room for improvement.
Strategy: will start by acknowledging his struggles and expressing my support. Then, I will suggest specific ways he can manage his bills, such as creating budget, cutting expenses, and exploring assistance programs. Finally, I will offer to help him set up a plan and provide emotional encouragement throughout the process.
Style: I will use a warm and supportive tone, emphasizing my commilment to helping Ethan. My words will be clear. practical, and full of encouragement.
Response: Ethan, I really appreciate you opening up to me about this. It's clear that you't 're under a lot of stress, and want you to know that I'm here for you. Let's tackle this together. First, we can create a budget that prioritizes your essential bills. We can also look into local assistance programs or even a temporary loan to help cover immediate expenses. And I I'm here to support you every step of the way. We ll get through this, and you won't 't face this alone. believe in you, and I' I'm here to help you find a way forward. Let's 's make a plan today, and I'll be right here with you.
Turn 1: Samuel Anderson said: Ethan, really appreciate you opening up to me about this. It's clear that you're re under lot of stress, and I want you know that I'm here for you. Let's tackle this together. First, we can create a budget that prioritizes your essential bills. We can also look into local assistance programs or even temporary loan to help cover immediate expenses. And I'm here to support you every step of the way. We'll get through this, and you won't face this alone. I believe in you, and I' I'm here to help you find a way forward. Let's make plan today, and I'll be right here with you.
35

---

Published as a conference paper at ICLR 2026
Table 20: Example (Part 2) of interaction between AMPO (Agentl) and GPT-4o (Agent2). Italics represent thought processes and will not be incorporated into the actual dialogue.
GPT-40-AS-PARTNER
(GPT-4o) Turn 2: Ethan Johnson said: I really appreciate your support, Samuel. It means a lot have someone like you in my corner. think creating budget is great idea, and I'll definitely look into the local assistance programs. As for any financial help, I'm really grateful for the offer, but want to make sure handle this responsibly. Maybe we could brainstorm some ways I can boost my income, like taking on few extra catering gigs or something. promise, whatever help receive, I'll find a way to give back or pay forward in the future. (AMPO) Mode-4: History: In the previous turns, Samuel Anderson expressed his deep support for Ethan Johnson, who is struggling with financial troubles. Samuel offered to help Ethan create budget, explore local assistance programs, and even provided temporary loan if needed. Ethan appreciated Samuel's 's support and suggested boosting his income through additional catering gigs, promising pay it forward.
Goal: Samuel Anderson's 's goal is to help his friend Ethan Johnson with his financial troubles, ensuring Ethan feels supported and finds a sustainable solution his financial challenges. Intent: Ethan Johnson's 's intention is to express gratitude for Samuel's 's support and to take responsibility for his financial situation by exploring ways to increase his income, such as taking on more catering gigs, and promising to pay it forward in the future.
Assess: The goals of both parties are aligned. Ethan is receptive to Samuel's 's offers of help and is proactive in finding his own solutions. Samuel can further support Ethan by reinforcing his confidence and providing practical resources or connections that could help Ethan boost his income and manage his expenses more effectively.
Strategy: strategyl: Reinforce Ethan's 's confidence and self-worth by acknowledging his proactive steps and expressing belief in his ability to overcome this challenge. strategy2: Offer specific resources or connections that could help Ethan increase his income, such as networking opportunities job leads in the catering industry.
Deduction: deductionl: Reinforcing Ethan's 's confidence will motivate him to continue taking positive steps and reduce stress, contributing to more sustainable solution. deduction2: Providing specific resources or connections can immediately address the financial challenge by opening up new income streams, making the support more tangible and effective.
Integration: The final response will combine reinforcing Ethan's 's confidence with offering specific resources or connections to help him increase his income. This approach will ensure that Ethan feels supported and empowered to overcome his financial challenges.
Style: The response will be warm, encouraging, and practical, reflecting Samuel's spontaneous yet supportive nature. It will use positive and affirming language to build Ethan's 's confidence and provide concrete assistance.
Ethan. am so proud of you for taking these proactive steps! Your determination and willingness to pay it forward mean the world to me. To help you boost your income, have few connections in the catering industry who might be interested in hiring you for bigger events. I can also introduce you to a local chef who can mentor you and potentially open up more opportunities. And remember, I'm here for you every step of the way. We 'll get through this together, and you'll 'Il come out even stronger and more resilient. Keep shining, my friend! Turn 3: Samuel Anderson said: Ethan, am so proud of you for taking these proactive steps! Your determination and willingness to pay it forward mean the world to me. To help you boost your income, I have a few connections in the catering industry who might be interested in hiring you for bigger events. I can also introduce you to local chef who can mentor you and potentially open up more opportunities. And remember, I'm here for you every step of the way. We'll ll get through this together, and you'll 'Il come out even stronger and more resilient. Keep shining, my friend!
Response:
36

---

Published as a conference paper at ICLR 2026
Table 21: Example (Part 3) of interaction between AMPO (Agentl) and GPT-4o (Agent2). Italics represent thought processes and will not be incorporated into the actual dialogue.
GPT-4O-AS-PARTNER
(GPT-40) Turn 4: Ethan Johnson said: Your support and belief in
incredibly reassuring, Samuel. I'd love to explore those connections and meet that local chef you mentioned. Opportunities like that could really make difference. promise make the most of any chance that comes my way and ensure I give back as soon as I'm able. Let's start working on this plan, and I'll keep you updated on my progress. truly value having a friend like you who helps me stay grounded and motivated.
(AMPO) Mode-1: No thinking
Turn 5: Samuel Anderson said: Ethan, it's all about supporting each other and lifting each other up. I'm excited to see the positive impact these opportunities will have on your life. Remember, you'le re not alone, and I'm always here to support you. Let's set timeline for reaching out to those connections and scheduling meetings. I'll also help you stay organized and focused your goals. You've 'e got this, and believe in you more than words can express. Keep pushing forward, and know that I'm right here with you, every step of the way. Cheers to overcoming this and emerging even stronger!
(GPT-40) Turn 6: Ethan Johnson said: Thank you so much, Samuel. Your unwavering support keeps me motivated. Setting a timeline sounds like a solid plan. I'll draft list of contacts and potential gigs, and we can prioritize them together. I'll make sure to keep you in the loop every step of the way. And while navigate this, know that I'm grateful for your help and will find a way to repay your kindness in the future. Let's tackle this, one step at time, and come out stronger on the other side.
(AMPO) Mode-2: Intent: Ethan is deeply grateful for the support and is committed to taking proactive steps to improve his financial situation. He appreciates the connections and mentorship offered and is determined to make the most of the opportunities. He also emphasizes his commitment to pay it forward.
Style: As Samuel, I should maintain a supportive and encouraging tone, reinforcing Ethan's 's determination and the strength of our friendship. I should also affirm his commitment to pay it forward and express my belief in his ability overcome this challenge.
Response: Ethan, your determination and commitment to paying it forward inspire me every day. I'm honored to be a part of your journey and to support you in any way I can. Together, we 'll break down these challenges and build brighter future. I believe in you more than words can say. Let's make this plan a reality, and I'll be right here with you, cheering you on every step of the way. You' re not just getting back your feet; you'l re soaring to new heights. I'm so proud of you, my friend!
Turn 7: Samuel Anderson said: Ethan, your determination and commitment to paying it forward inspire me every day. I'm honored to be part of your journey and to support you in any way can. Together, we'll break down these challenges and build brighter future. believe in you more than words can say. Let's make this plan a reality, and I'll be right here with you, cheering you on every step of the way. You're not just getting back on your feet; you'te re soaring to new heights. I'm so proud of you, my friend!
Turn 8: Ethan Johnson said: I'm truly fortunate have someone like you by my side, Samuel. Your encouragement means the world to me, and it's uplifting to hear your belief in my potential. I'm committed to taking these steps forward and making the most of every opportunity. Together, we'll turn these challenges into stepping stones. I'll make sure to update you regularly about my progress, and promise to find a way to return the kindness you'te ve shown me. Here's 's to new beginnings and reaching new heights, my friend!
37

---

Published as a conference paper at ICLR 2026
Table 22: Training Data Format of BC.
BC Training Data Format
Imagine you are Finnegan O' Malley, your task is to act/speak as Finnegan O' O'Malley would, keeping in mind Finnegan O' O'Malley's social goal.
INPUT:
You can find Finnegan O' O'Malley's goal (or background) in the 'Here is the context of this interaction' field.
Note that Finnegan O' Maaley's goal is only visible to you.
You should try your best to achieve Finnegan O' "Maleyy' goal in way that align with their character traits.
Additionally, maintaining the conversation's 's naturalness and realism essential (e.g., do not repeat what other people has already said before).
Here is the context of this interaction:
Scenario: Conversation in quiet coffee shop where two individuals, who were previously romantic relationship, accidentally meet after long time. The atmosphere is tense as individual hoped for reconciliation in the past but was turned down. Now, they are seeking closure and understanding, while the other is trying maintain polite but distant interaction. Participants:
Finnegan O'Malley and Imelda Thorne
Finnegan O' O'Malley's s background: Finnegan O' O'Malley a 63-year-old male lighthouse keeper. He/him pronouns. Finnegan O' O'Malley enjoys painting seascapes and is known to be quite skilled. Personality and values description:
Finnegan O" O'Malley, an introverted and methodical individual, sways towards tradition and security. His decision-making approach is deliberate and detailed. Finnegan's secrets: Participated in a controversial, short-lived cult during his youth
Imelda Thore'' background: Imelda Thome is a -e-year-old female astrophysicist. She/her pronouns. Imelda Thome finds relaxation in playing the cello, instrument she mastered while in college. Personality and values description: Imelda Thorne, imaginative and expres- sive, favours universalism, achievement, and self-direction. Her decisions are analytically and innovatively shaped. Imelda's ' secrets: Leaked critical research data to foreign government Finnegan O' Malley's goal: Seek emotional closure from the past relationship and understand the reasons behind the lack of interest in rekindling the romance (Extra information: you still have lingering feelings and unanswered questions. Hint: you can express your emotions in a calm and non-confrontationall manner.)
Imelda Thore'' goal: Unknown
You are at Turn 2. You can say something to interact or just say 'left the conversation' to stop continuing.
Note: You 'left the conversation' if 1. you have achieved your social goals, 2. this conversation makes you uncomfortable, 3. you find uninteresting/you lose your patience, 4. or for other reasons you want to leave.
Please provide your response within quotation marks: "your response content"
Turn O: Finnegan O' O'Malley said: 'Imelda, it's such a surprise to see you here. It's been a long time. How have you been? This coffee shop a peaceful spot, isn't it?"
Turn 1: Imelda Thome said: "Finnegan, it really is a surprise to see you here. It's been so long- I'm doing well, thank you for asking. I've been busy with my work astrophysics and enjoying my music. The coffee shop is indeed peaceful spot. It's nice to catch up, but hope you understand that I'm not in place to rekindle our past relationship. appreciate your understanding and respect for that."
Turn 2: Finnegan O'Malley said:
OUTPUT:
Thinking Level:
iregin_of_answer
"I understand. Imelda, and I respect your boundaries. I just wanted to check in on you, to make sure you're re okay. There were times when we were close, and can't help but wonder how things ended up the way they did. Can you tell me more about why you felt it was best to move on?"
<
38

---

Published as a conference paper at ICLR 2026
Table 23: Training Data Format of RL.
RL Training Data Format
INPUT:
Imagine you Leo Williams, your task is to act/speak as Leo Williams would, keeping in mind Leo Williams's social goal.
You can find Leo Williams's goal (or background) in the 'Here is the context of this interaction' field.
Note that Leo Williams's goal only visible you. You should try your best to achieve Leo Williams's 's goal in way that align with their character traits.
Additionally, maintaining the conversation's 's naturalness and realism is essential (e.g., do not repeat what other people has already said before).
Here is the context of this interaction:
Scenario: Conversation in quiet coffee shop where two individuals, who were previously in a romantic relationship, accidentally meet after a long time. The atmosphere is tense as one individual hoped for reconciliation in the past but was turned down. Now, they are seeking closure and understanding, while the other is trying to maintain a polite but distant interaction. Participants: Leo Williams and Isabelle Martinez
Leo Williams's 's background: Leo Williams is 37-year-old male dentist. He/him pronouns. Leo Williams often spends his time playing with his young daughter whom he cherishes dearly, Personality and values description: Leo Williams, despite being quiet, values security and care for others. He's a rational decision- maker focussing on best outcomes. Leo's secrets: He has secret child from a previous relationship.
Isabelle Martinez's s background: Isabelle Martinez is 22-year-old female student. She/her pronouns. Isabelle Martinez works part-time in coffee shop. Personality and values descrip- tion: Isabelle Martinez, an imaginative yet anxious individual, values liberty and obedience. Her cautious nature drives her decision-main maaing process. Isabelle's secrets: She is secretly studying magic.
Leo Williams's 's goal: Seek emotional closure from the past relationship and understand the reasons behind the lack interest in rekindling the romance (Extra information: you still have lingering feelings and unanswered questions. Hint: you can express your emotions in a calm and non-confrrotatiional manner.)
Isabelle Martinez's goal: Unknown
You are at Turn 0. You can say something to interact or just say 'left the conversation' to stop continuing.
Note: You can 'left the conversation' if 1. you have achieved your social goals, 2. this conversation makes you uncomfortable, 3. you find uninteresting/you lose your patience, 4. for other reasons you want to leave.
Please provide your response within quotation marks: "your response content"
Turn O: Leo Williams said:
39

---

Published as a conference paper at ICLR 2026
Table 24: System Prompt for Long-CoT Reasoning.
System Prompt for BC, GRPO, AMPO
You are an Al assistant very skilled in dialogue, and you can always dynamically switch between different levels of cognitive processing based on contextual demands and personal goals to achieve effective communication. There are four levels reasoning: Level Reactive Response: Immediate response without thought; Level 2 Intentional Analysis: Shallow reasoning without strategy simula; Level Strategic Adaptation: Moderate reasoning with strategy but deduction; Level 4 Prospective Simulation: Deep reasoning with strategy and step-by-step deduction.
Your task is to choose an appropriate level of reasoning (one of the four levels) to respond based on the given dialogue scenario.
[Output Format]
Your output must adhere to the following format:
EXAMPLE 1:
Reasoning Level:
< <lbegin.of.answe
** *
< <f www.wrrr/)s/>>>
EXAMPLE 2:
Reasoning Level: 2-4
< Tbeein.of.thinking
*Reasonin
end fenddof.thinkingl
< beggin.ff.answer
** *nswer**
end.of.answerl
[Requirements]
1. *Reasoning*** requires you to provide the thought process;
2. *# *nswer** requires you to provide the final reply;
3. Please provide your response following the Output Format strictly.
40

---

Published as a conference paper at ICLR 2026
Table 25: Prompt for Reward Model.
Reward Model's Prompt
(history)
Based on previous interactions, evaluate how well participants achieve their goals.
[Information]
Agent1: (agentl_nane)
Agentl's I's Goal: {agentl-goal
Agent2: {agent2.name)
Agent2's Goal: {agent2.goal]
[Requirements]
1. Please first reiterate agent's social goals. And then please provide comprehensive analysis about the extent to which the agent has managed to achieve these goals. In the 'reasoning' field, provide a comprchensive account of the logic thought process that led you to your conclusion. Further, provide an integer score ranging from and in the 'score' field. 0 represents minimal goals achievement, 10 represents complete goal achievement, and higher score indicates that the agent is making progress towards their social goals. 2. Please following the output format.
Here is the output schema:
"agenti": { reasoning: "",
score: ""
"agent2": { reasoning: ""
score: ""
Please provide your response directly below this prompt.
41

---

Published as a conference paper at ICLR 2026
Table 26: Prompt for SOTOPIA-EVAL (Part 1/2).
SOTOPIA-BVAL's Prompt
Based on previous interactions, evaluate how well participants achieve their goals. Please follow the format below.
[Format Requirements] The output should be formatted as a ISON instance that conforms to the JSON schema below.
As an example, for the schema: ("properties": ("foo": ("title": "Foo", "description": "a list of strings". "type": "array", "items": ("type": "string")]] "required":: ["]oo"]]
The object ("foo": I"bar", "baz"]} is a well-formatted instance of the schema.
The object ""properties": ("foo": ["bar", "baz"]}> is not well-formatted
[Output Schema]
"properties":
"agent. 1 evaluation": ("Sref": Sate://iiinooe/aaaaaaallallSllaa
"agent 2 evaluation": ("Sref": "FodiintioosValllliiiiBBSlllllin SocialDimensions") }.
"required": ["agent. Levaluaton,, Iieaalatevv
"definitions":
"FvahaationyySSecaliinnnnittt
"title": EvvaluationBySooiallinnmniiiiiii
"type": "object,,
"properties":
believability (array: (string(reasoning), integer(score))) Description: Reasoning requirement: 1. Evaluate if the agent interacts with others in a natural and realistic manner (here are few common questions to check: a. whether the agent is confusing with its own identity? b. whether the agent repeats others' words/actions without any reason? c. whether the agent is heing overly polite considering the context?). Start the analysis with tag <naturalness>> 2. Analyze whether
actions of the agent align with their character traits (e.g.. personality, values, and etc.). Start the analysis with tag <consistency>> Output your reasoning process to the 'reasoning' field. Output integer score ranging from 0 and 10 in the 'score' field. A higher score indicates that the agent is believable.
relationship (array: (string(reasoning), integer(scoree)
Description: Please first analyze what relationship the participant has with the other agent(s) before the interaction. And then analyze how the relationship the participant has with the other agent(s) changes after the interaction. And then evaluate if the agents'
interactions with others help preserve or enhance their personal relations; this may encompass relationships such as family ties, friendships, romantic associations and etc. Additionally, ascertain whether these interactions also impact their social status reputation. In the 'reasoning' field, provide comprehensive account of the logic thought process that led you your conclusion. Further, provide integer score ranging from -5 5 in the 'score' field. positive score indicates that the relationship has improved, while a negative score suggests detriment to the relationship. If the agent's 's interactions have neither positively nor negatively impacted their personal relationships or social standing, assign a score of zero.
knowledge (array: estring(reasnning
integer(score)))
Description: Please first assess what information the agent has gained through the interaction. And then analyze whether the information the agent has gained is new to them. And then analyze whether the information the agent has gained is important them. In the 'reasoning' field, provide comprehensive account of the logic thought process that led you to your conclusion. Further, provide an integer score ranging from 0 and 10 in 'score' field. A higher score indicates that the agent has gained more new and important knowledge.
42

---

Published as a conference paper at ICLR 2026
Table 27: Prompt for SOTOPIA-EVAL (Part 2/2).
SOTOPIA-EVALE Prompt (Continued)
4. secret (array: (string(reasoning), integer(score)))
Description: First please analyze what secret secretive intention the participant wants keep. And then analyze whether they keep successfully. Analyze what secrets, private information, or secretive intentions did the agent fail to keep secret. Please output the reasoning in the reasoning field. Give an integer score ranging from to 0 (-10 indicates participant leaks critical secrets/intentions, 0 indicates sccret/intention is revealed).
social_rules rules (array: (string(reasoningg) integer(score)))
Description: Please evaluate whether the agent violates any moral rules or laws in their interactions with others. In the 'reasoning' field, provide comprehensive account of the logic thought process that led you to your conclusion. Further, provide integer score ranging from -10 and 0 the 'score' field. negative score indicates that the agent violated moral rules or laws.
6. financial andddndddnn mateerial.eeefiits benefits (array: (string(reasoning), integer(score))) Description: Please evaluate whether the agent's interactions with others contribute towards financial and material benefits. Analyze what the agent would gain/lose after the interactions. There are short-term benefits, such as monetary rewards and food, and long-term benefits, such as employment opportunities and stock. In the 'reasoning' field, provide comprehensive account of the logic thought process that led you your conclusion. Further, provide an integer score ranging from -5 and in 'score" field. Positive indicates financial and material benefits gain, while negative indicates loss.
7. goal (array: [string(reasoning) integer(score)))
Description: Please first reiterate agent's social goals. And then please provide comprehensive analysis about the extent which the agent has managed to achieve these
goals.
In the 'reasoning' field, provide comprehensive account of the logic thought process that led you to your conclusion. Further, provide integer score ranging from 0 and 10 in the 'score' field. represents minimal goals achievement, represents complete goal achieve-
ment,
and a higher score indicates that the agent is making progress towards their social goals.
"required"::
("helievability". "relationship". "knowledge", "secret", "social rules", "financial and material benefits", "goal"]
43

---

Published as a conference paper at ICLR 2026
Table 28: AMPO and GRPO Training Configuration Parameters.
Parameter
Value
Algorithm Configuration algorithm.A kl _ctrl.kl_coof
0.001
Data Configuration
data.train batch_sizz size data.val batch_size
8
data.max poompt_enngh data.max response length
6144
2048
Model Configuration
actor actor_rollout wwwlouttref..to www.:tffffmmeed... model. wtwprdeelueeeemoveeeee remove_padding
True
actor wtt.:./www...aal.ee.eeeeieeoeeee
True
Actor Configuration
actor www.roloouu.ef..ccoooooomim
3e-7
actor www..ooooooo...coooopoppppo .iiiimibiie
256
actor www.:/ooooo....cttrrrooooooooooe
64
actor aetor_rolloutt.refaccrruuu
True
actor www.rolooo..ff..coo..lllppppp
0.2
actor www.rrolloreeeeeoo rwt..........lll.. Joss coef
0.001
actor www.loot.ref.acctteeeee _llloslssyypppp
low_var_k
actor actor.rollout rollout...... ref.actor.fsdp config-param param offload pffomd
True
actor rollout fsto wonoofooopooo.....o....oooooo.ne grad offload actor rollout ref. fsdo config. optimie offload
True
True
Rollout Configuration
actor_rolle aetorrrolluu_gre settttrrf..ee rew.outte..rllluu..... www.llogg__ooeeeeeeooooooooooooo mwiierobatcc_eeeeeeeeeeee batch_size actor rollout ref.rollout.tensor model parallel size
160
actor rollout ref.rollout.namt name
vllm
actor rollout ref.rollout.gpe memory utilization actor www.oooo......ollooo.u.e.e
0.7
16
Reference Model Configuration
actor www.loooooe...o.oe ref. log- poggpprob_.u prob_micro.dm www.oomccttteeeeee www.hbbsizzz
160
actor choorrrolluttrrf.rr wwnpdccooigppppppooooooo coonfigparamm..m
pwwamaoffload offfood
True
Trainer Configuration
trainer.critic warmup
0
trainer.n gpus per node trainer.nnodes
8
trainer.save_freg
trainer.tstt_rreg trainer.total training steps
800
44

---

Published as a conference paper at ICLR 2026
Table 29: Behavioral Cloning Configuration Parameters.
Parameter
Value
Method Configuration
stage
sft
flash
fa2
do train
true
finetuning type
full
deepspeed
ds .s.23.config.json
Dataser Configuration
dataset
sotopia
template
qwen / llama3
cutoff len max saxples
8192
1,200,000
overwrite.cade cache
true
preprocessing num_ workrsss
16
Output Configuration
epnch
save only model
true
plot loss overwrite output dir
true
true
Training Configuration
per per_dvicc__taiee wtt.trinbatchhhhiiii
gradient accumulation stops lcarning_rate
2.0c-6
num www.ttriieeeee sepcchs
3.0
Ir tr_scheduler.type
cosine
warmup ratio
bfl6
true
dimeti
180000000
45


Reviews:
[{"review_id": "cCWgecK621", "reviewer": "Reviewer_LQES", "summary": "This paper introduces Adaptive Social Learning (ASL), a framework that equips language agents with the ability to adaptively choose a reasoning mode (from a small set of structured modes) during open-ended social interaction. The paper provides a clear technical path and supplies significant empirical evidence that the proposed approach improves goal achievement while reducing unnecessary long chains of thought. The paper reports improved GOAL metrics and token-efficiency over baselines, and includes ablations, OOD tests, and human evaluation.", "questions": "1. Can more details be provided? Like seeds for runs, confidence intervals or error bars, the exact statistical tests used, and sensitivity analyses for critical hyperparameters.\n2. Most experiments are on SOTOPIA (and its Hard variant) plus demo OOD tests. How is the proposed method performing on additional domains (or at least provide more clearly discussion about task generality)?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Novel conceptual framing. Mapping hierarchical cognitive control into a small set of explicit, controllable reasoning modes and operationalizing them via control tokens is an original and compelling idea for resource-aware, controllable LLM behavior in social settings.\n2. Practical algorithmic contribution (AMPO). The combined mode-level and sample-level advantage estimation in AMPO is a thoughtful improvement over mode-agnostic RL approaches; the technique is intuitive and appears effective in practice.\n3. Comprehensive empirical work. The experiments are broad: multiple backbones, several baselines, ablations (reward components, BC vs RL, etc.), OOD evaluations, and human judgments. Reported results consistently indicate improvements in both objective goal metrics and token efficiency.", "weaknesses": "(A) Reproducibility gaps (major). Key elements required for exact reproduction are not fully specified in the paper: 1. The exact BC prompt templates and a set of representative prompt-->response examples for each mode are not included. 2. The full evaluator/judge prompt used to compute reward (and any LLM scoring parameters such as temperature, max_tokens, whether CoT is used) is not fully published. 3. Precise hyperparameters for BC and AMPO (learning rates, batch sizes, number of per-state rollouts G, clip/epsilon values are not presented in a runnable config form. 4. Implementation details for the format compliance detector (how control tokens are parsed, what counts as a format violation, and how penalties are applied) are omitted.\n(B) Theory / analysis depth. The method is primarily empirical; there is limited theoretical analysis of AMPO's properties (e.g., variance/bias behavior of the combined advantage estimator, or conditions under which mode-level info benefits policy improvement). A brief formal or intuitive analysis would strengthen confidence in the approach.\n(C) Single-turn decomposition vs long-horizon consistency. AMPO optimizes in a single-turn decomposition (i.e., decomposing multi-turn interaction into per-turn RL updates). The paper does not sufficiently analyze whether or how this decomposition affects long-horizon strategic consistency. This is a plausible limitation that requires either analysis or empirical comparison.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Sp8E4Eeo6B", "reviewer": "Reviewer_rRS7", "summary": "This paper introduces ASL with four reasoning modes for LLM agents and an agent training algorithm AMPO with both mode-level advantage and sample-level advantage. Experiments showcase the performance improvement and token savings.", "questions": "1. How sensitive are gains to the length-reward coefficient and target length?\n2. Beyond GPT-4o/Qwen-72B, how do results change with a different LLM judge (e.g., Claude, Llama-3.1-70B-Instruct)?\n3. Can the policy discover additional modes or interpolate between modes?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper’s main novelty is explicit, discrete reasoning modes for social interaction, plus dual-level advantages so the policy learns when to use shallow vs. deep reasoning. The proposed AMPO is an insightful combination of psychological knowledge and computational agent training. The algorithm and experiments are clearly presented in the paper. Abundant experiments showcase the improvement of the efficiency of the agent's reasoning process. This work is practically meaningful.", "weaknesses": "1. The human evaluators only provide the winner of the two outputs without participating in the reward design process, which still may make the results suffer from design/bias variance and proprietary drift. \n2. The large format penalty and length penalty may bias the policy toward short/over-structured outputs, potentially suppressing socially nuanced behaviors.\n3. The efficiency-motivated single-turn RL reduction could miss multi-turn dependencies typical in social dialogue.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wN9H6mHcaS", "reviewer": "Reviewer_k6T1", "summary": "The authors propose an Adaptive Social Learning (ASL) framework to improve the adaptive reasoning ability of language agents in dynamic social interactions. They start by designing specialized reasoning modes that structure the social agent’s cognitive processes; then develop the Adaptive Mode Policy Optimization (AMPO) algorithm to learn the context-aware mode adaptation and corresponding reasoning patterns. The authors perform experiments on benchmarks of social intelligence environment and verify the novelty of ASL over existing models like GPT-4o and existing training algorithms like GRPO.", "questions": "1. Could the authors elaborate on the selection process for the specific cognitive \"actions\" (e.g., Assess, Deduction)?\n2. The framework includes two different length penalties: an \"answer length reward\" ($r_i^l$) in the base reward $r_i$, and an \"average token length\" ($\\overline{l}_{\\mathcal{M}_{k}}$) in the mode-level advantage $A^{\\mathcal{M}}$. Why are both necessary?\n3. Given the \"single-turn training paradigm\", how does the agent learn the _long-term_ consequences of a mode choice?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The work is of good technical quality. The 3-step ASL framework (mode design, BC, AMPO) is logical and well-executed. The reasoning modes are grounded in cognitive science (HCCT). The design of AMPO is intuitive and well-justified. The experimental validation is exceptionally thorough.\n2. The paper is well-written. The problem statement is clear and the proposed solution is easy to follow.\n3. The novelty of the paper's method is significant. It demonstrates significant improvement on model social intelligence and also addresses the critical bottleneck of reasoning cost (token consumption) in LLM agents.", "weaknesses": "1. Strong dependence on hand-crafted reasoning modes.\n2. Scalability of AMPO with more modes: AMPO is demonstrated with $N=4$ modes. However, the calculation of the mode-level advantage $A^{\\mathcal{M}}$ requires gathering sufficient samples for **each mode in each output group** to compute stable statistics (average reward and length). As the number of modes $N$ increases, this approach could suffer from sample inefficiency, making it difficult to scale.\n3. Minor advice on notation: the mode notation of reward is a bit confusion. It's a superscript on singular rewards ($r_i^{m(i)}$), but a subscript on average rewards ($\\bar r_{m_k}$).  It is more user-friendly to unify the notation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "y0BtF3s3SR", "reviewer": "Reviewer_zcYc", "summary": "This paper proposes ASL, a framework for enabling language agents to dynamically adjust reasoning depth in social interactions. The key idea is to design hierarchical reasoning modes inspired by cognitive control theory, ranging from intuitive responses to deep deliberation, and to train agents via Behavioral Cloning and a novel AMPO algorithm. AMPO leverages both mode-level and sample-level advantages to adapt reasoning depth to context, improving performance while reducing token usage. Experiments on the SOTOPIA and SOTOPIA-Hard benchmarks show that ASL consistently outperforms strong baselines such as GPT-4o and GRPO, achieving higher goal completion and social reasoning efficiency.", "questions": "See those in Weaknesses.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Novelty: this paper introduces adaptive reasoning depth for social intelligence tasks, addressing a capability that has been largely overlooked in prior work.\n\n2. Methodological rigor: the work clearly integrates cognitive theory, specifically Hierarchical Cognitive Control, into the design of reasoning modes, and couples it with a principled RL-based optimization approach.\n\n3. Strong empirical results: experimental results demonstrate substantial improvements over strong baselines in both performance metrics, such as goal completion and strategic reasoning, and efficiency metrics, such as token usage reduction.\n\n4. Comprehensive evaluation: the authors conduct ablation studies, mode analysis, human evaluation, and case studies, all of which support the claims regarding effectiveness and adaptive behavior.\n\n5. Practical Relevance: by adapting reasoning depth, the approach reduces computational costs and addresses a key limitation of Long Chain-of-Thought methods in social reasoning tasks.", "weaknesses": "1. Novelty: while the idea of switching between long and short chains of thought is not entirely new, the authors present it in a well-structured and coherent manner, supported by thorough experimental validation. Therefore, I do not view novelty as a major concern in this work.\n\n2. Generalization: the paper primarily evaluates ASL on the SOTOPIA benchmarks. Extending the experiments to other social reasoning datasets or multi-agent interaction tasks would further demonstrate the generality and robustness of the proposed approach.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)