Temperature as a Meta-Policy: Adaptive Temperature in LLM Reinforcement Learning
Authors: Haoran Dang
Cuiling Lan
Hai Wan
Xibin Zhao
Yan Lu
Venue: ICLR 2026 Poster (2026)
Keywords: Large Language Models; Reinforcement Learning;Adaptive Sampling Temperature;Meta-Optimization;GRPO;
Primary area: foundation or frontier models, including LLMs

Abstract: Temperature is a crucial hyperparameter in large language models (LLMs), controlling the trade-off between exploration and exploitation during text generation. High temperatures encourage diverse but noisy outputs, while low temperatures produce focused outputs but may cause premature convergence. Yet static or heuristic temperature schedules fail to adapt to the dynamic demands of reinforcement learning (RL) throughout training, often limiting policy improvement. We propose Temperature Adaptive Meta Policy Optimization (TAMPO), a new framework that recasts temperature control as a learnable meta-policy. TAMPO operates through a hierarchical two-loop process. In the inner loop, the LLM policy is updated (e.g., using GRPO) with trajectories sampled at the temperature selected by the meta-policy. 
In the outer loop, meta-policy updates the distribution over candidate temperatures by rewarding those that maximize the likelihood of high-advantage trajectories. This trajectory-guided, reward-driven mechanism enables online adaptation without additional rollouts, directly aligning exploration with policy improvement. On five mathematical reasoning benchmarks, TAMPO outperforms baselines using fixed or heuristic temperatures, establishing temperature as an effective learnable meta-policy for adaptive exploration in LLM reinforcement learning.

TEMPERATURE AS A META-POLICY: ADAPTIVE TEM- PERATURE IN LLM REINFORCEMENT LEARNING
Haoran Dang'
Cuiling Lan Hai Wan'
Xibin Zhao
Yan Lu
'Tsinghua University MMicrosoft Research Asia
canghr23emal is tsinghua. edu. en, {eulan, www.aaammirrooottttoooooo (wanhai, wxbletsinghua .ww....euuu.............a..
ABSTRACT
Temperature is a crucial hyperparameter in large language models (LLMs), con- trolling the trade-off between exploration and exploitation during text generation. High temperatures encourage diverse but noisy outputs, while low temperatures produce focused outputs but may cause premature convergence. Yet static heuristic temperature schedules fail to adapt to the dynamic demands of reinforce- ment learning (RL) throughout training, often limiting policy improvement. We propose l'emperature Adaptive Meta Policy Optimization (TAMPO), a new frame- work that recasts temperature control as a learnable meta-policy. TAMPO oper- ates through a hierarchical two-loop process. In the inner loop, the LLM policy is updated (e.g.. using GRPO) with trajectories sampled the temperature selected by the meta- meta-policy. In the outer loop, meta-policy updates the distribution over candidate temperatures by rewarding those that maximize the likelihood of high- advantage trajectories. This trajectory-guided, reward-driven mechanism enables online adaptation without additional rollouts, directly aligning exploration with policy improvement. On five mathematical reasoning benchmarks, TAMPO out- performs baselines using fixed or heuristic temperatures, establishing temperature effective learnable meta-policy for adaptive exploration in LLM reinforce- ment learning.
INTRODUCTION
Reinforcement learning (RL) has become promising paradigm for aligning large language models (LLMs) with human preferences and task-specific objectives (Ziegler et al. 2019 Ouyang et al.
2022 Bai al. 2022; Chen al., 2025). Traditional RLHF approaches often rely PPO-based
RL-based post-training, which requires a learned value network and incurs significant computational
overhead. Recent critic- free algorithms, such GRPO (Shao al.] 2024 Guo et al. 2025) and
REINFORCE+ (Hu et al. 2025), demonstrate that large-scale LLM reinforcement learning can be both scalable and stable, bypassing the need for value networks while maintaining performance.
One of central challenges in RI. remains the exploration-exploiaee Kaelbling et al.) 1996). For LLMs, sampling temperature serves as a direct and interpretable con- trol knob: higher temperature produces more uniform (random) distribution, encouraging diverse but potentially noisy generations, while lower temperature concentrates probability mass, favoring precision but at the risk of missing promising alternatives. Existing approaches, however, treat tem- perature as fixed or manually tuned, ignoring feedback from the learning process. Popular critic-free RL algorithms (e.g.. GRPO) (Shao al. 2024 Guo et al. 2025 Hu et al. 2025 eu (025) gen- erate multiple rollouts at given temperature to estimate trajectory advantages and policy gradients, but never adapt temperature based on trajectory outcomes.
trade-off (Sutton et al. 1998
We argue that temperature should be treated as a decision variable, not fixed hyperparameter for LLM RL. Unlike entropy regularization coefficients KL penalties, which influence explo- ration (Guo et al., 2025; Shen (025), temperature directly modulates the sampling distribution over text outputs in simple, transparent manner. This motivates our work on principled, trajectory- guided temperature adaptation for effective LLM policy learning.
"Corresponding author.

---

Outer loop (learn temperature)
Inner loop (critic-free LLM RL)
KZ
Reference model
Compotation
Camg
A
Meta- policy
T
Policy model
T2
Reward model
A
:
Aa
A.
A.
A. 1
EMA TSA
Calculate
likelihoods
A
A
A
2 (r)
(t)
c (r)
A
A
A
2 (t.)
2gg (f.)
2 (r.)
A
A
2 (r,)
2g (5a)
(F.)
Temperatuure-specffic advantages (TSA)
Normalized likelihoods (at virtual temperatures T..Tg....T.)
Figure I: Overview of Temperature Adaptive Meta Policy Optimization (TAMPO). The framework operates through a hierarchical two-loop process. In the inner loop, the LLM policy is optimized with critic-free RL (e.g., GRPO) using rollouts sampled the temperature chosen by the meta- policy. In the outer loop, the meta-policy is updated by evaluating trajectory likelihoods under virtual temperatures, deriving lemperature-specifie advantages (A4,T*) é,, (T.)-A; for trajectory тı w.r.t. virtual temperature Tk), and reinforcing those that yield high-advantage rollouts (see $3). This design establishes temperature learnable meta-policy, enabling online adaptation and effective optimization of LLM policy without extra rollouts.
In this work, we introduce Temperature Adaptive Meta Policy Optimization (TAMPO), a meta- learning framework that jointly optimizes the LLM policy, and a meta-policy over temperatures. Figure | shows the overall framework, which operates through a hierarchical two-loop process. In the inner loop, the LLM policy model πe,r is optimized using critic-free RL (e.g., GRPO) at a sampled temperature T. In the outer loop, a meta-policy æ obtained based on temperature- specific advantages by reusing the inner loop rollouts, reinforcing temperatures that more likely to generate trajectories with high advantages.
Intuitively, the temperatures that facilitate discovering rewarding outputs reinforced, while the ineffective ones are suppressed, allowing the learned temperature to dynamically align with policy improvement.
We summarize our main contributions as follows:
We formulate temperature as a learnable meta-policy in LLM RL, reframing temperature selection policy optimization problem to enhance adaptive, reward-driven exploration.
We propose TAMPO, hierarchical framework that jointly updates the LLM policy and meta- policy over temperatures, rewarding those that prompt high-advantage rollouts.
On five challenging mathematical reasoning benchmarks, TAMPO achieves better performance than baselines using fixed or heuristic temperatures, demonstrating the effectiveness of our trajectory-guiie guided adaptive temperature control in LLM reinforcement learning.
TAMPO provides a principled, end-to-end, feedback- drrven mechanism to dynamically balance ploration and exploitation, climinating manual temperature sweeps, and improving the effectiveness of RL-based post-training.
2

---

2 RELATED WORK
Critic-Free RL Methods. Critic-free RL algorithms, such as REINFORCE Leave-One-One (RLOO) (Kool et al. 2 GRPO (Shao al. 2024 Guo al. 2025), DAPO Yu et al. 2025). and REINFORCE++ (Hu et al. 2025) eliminate the need for learning value networks (critics), making them more efficient and scalable for LLM RL-based post-training. However, these approaches still mainly rely on fixed sampling temperatures, which may lead to under-exploration when the train- ing temperature is too low, or wasted computation and noisy samples when too high. Our method complements these algorithms by learning meta-policy over temperatures, which adapts sampling based on trajectory outcomes to better guide LLM policy optimization.
in RL. The iiloration-exploiiationn dilemma is fundamental chal- lenge in RL, where agents must balance between exploring new actions to discover potentially bet- ter strategies and exploiting known actions maximize immediate rewards (Wang et al. 2025
ipploration-Explottaeii
Traditional methods, such as e-greedy, temperature annealing, and upper confidence bounds (UCB). employ fixed or heuristic schedules to manage this balance (Wikipedia contributors 2025 Sutton let al. 1998). Typically, <-greedy linearly or exponentially decays e over time, temperature anneal- ing gradually lowers the sampling temperature to reduce exploration, and UCB adaptively selects actions based on upper confidence bounds that balance estimated value and uncertainty. In RL for LLMs, maintaining exploration is crucial avoid premature convergence to suboptimal reason- ing paths. Nucleus sampling (Holtzman et al., 2019) provides one practical strategy by restricting sampling to the smallest set tokens whose cumulative probability exceeds a threshold p. thereby adaptively balancing diversity and reliability in generation, where is in general fixed (e.g.. 0.95). Entropy regularization is widely adopted to promote diverse outputs by encouraging higher-entropy policies (Guo et al. 2025 Shen 2025). Sethi er al.(Sethi et al. (025) view policy optimization as continuous-tin time dynamical system and gradually decay the entropy regularization weight (typically constrains entropy within a pre-defined range. However, the optimal entropy level during training remains unclear.
as 1/t). Shen (Shen 2025
Beyond entropy, sampling temperature offers a direct and interpretable control knob for balancing exploration and exploitation in LLMs. Du et al. (Du et al. (2025) propose an adaptive inference-time method that selects optimal temperatures using multiple sampled generations. While effective for inference, it does not address how temperature can be dynamically optimized during RL training. Current LLM RL approaches either fix the temperature (Guo et al., 2025 Chen al. 2025 or manually tune it (An et al., 2025 Liu et al.] (2255). ), without incorporating trajectory feedback. Our work frames temperature as a learnable meta meta-policy that adapts exploration online, guided directly by trajectory advantages and likelihood to align LLM policy optimization.
Meta-Policy in Reinforcement Learning. Meta-policies have been studied in conventional RL. In hierarchical RL., the meta-policy acts as a high-level controller over low-level skills or options (Vezh- nevets et al. 2017 |acoo et al. 2017| |rans et al., 2017). MISH (Frans et al.] (017) learns a set of reusable sub-policies, which are shared across tasks, while task-specific master policy then com- poses these sub- policies to solve new tasks. Another line of work leverages meta-gradient methods to treat hyperparameters- -sushssuchhha as discount factors, bootstrapping parameter A-as differentiable variables updated through meta-gradients (Xu et al.) 2018 Wang & Ni. 2020). Meta-SAC learns the optimal entropy regularization parameter in Soft Actor-Critic (Wang & Nil 2020)).
While these works highlight the potential of meta-policies in conventional RL, they have not been explored in the context of L.M RI. To the best of our knowledge, our approach is the first treat sampling temperature as a meta-policy, directly addressing the exploration-exploitaiii during LLM RL. Note that the hyperparameters studied in (Xu al., 2018 Wang & Ni 2020) are differentiable, enabling direct optimization. In contrast, the sampling temperature in the typical LLM RL frameworks is non-differentiable, rendering these methods infeasible. We propose the practical TAMPO approach, which efficiently adapts the sampling temperature by reusing existing rollouts without requiring gradient- optimization.
trade-off
3 TEMPERATURE ADAPTIVE META POLICY OPTIMIZATION (TAMPO)
We aim to treat temperature as a learnable meta-policy that dynamically balances exploration and exploitation during L.M reinforcement learning. Given discrete set of candidate temperatures
3

---

(T,...... define meta-policy (T) that outputs probability distribution over tem- peratures. During training. each rollout sampled at a temperature determined based on x (T"). To achieve this, we introduce Temperature Adaptive Meta- Meta-Policy Optimization (TAMPO), hier- archical framework that shifts probability mass toward temperatures that produce high-advantage trajectories while suppressing ineffective ones.
In this section, we first review background in $31 then formalize the problem in $3. and finally
present TAMPO $.3
3.1 BACKGROUND
We consider an LLM parameterized by 8, trained with reinforcement learning. For a given prompt q. the model generates trajectory T; Io,, Oi.n) of n tokens, where each token Oi, is generated based on the state s (g, oi, Reward is provided at the sequence level as
Critic-Free RL with GRPO. In LLM RL, critic-free algorithms have become widely adopted due to their scalability and simplicity. One representative method is Group Relative Policy Optimization (GRPO) (Shao et al. 2024, Guo et al., (2025) which provides a simple way to compute trajectory- level credit/advantage without an explicit critic. For a given prompt. it samples group of G rollouts (TI...., To} at a given temperature T. then computes trajectory-lever advantages A, by normal- izing rewards r7 within the group:
A 10-mean ( rct)
std((rr....,ra))
GRPO updates the policy nø,r by maximizing the expected advantage while including a KL regu- larization term with respect reference policy (see Appendix [A for the full objective).
Temperature Rollout Generation. During rollout generation, the sampling distribution con- trolled by temperature parameter T which scales the token logits 2(0,,t si,r) as:
exp(2(o1, (1,))T) ( st,)) Ed, exp(=(o,,, sT,)/T)
The temperature controls the exploration-explooiatinn trade-off: too low results in over-exploitation, too high introduces excessive randomness, reducing useful exploration. Despite its importance, T is typically fixed, limiting policy optimization.
3.2 PROBLEM FORMULATION
We formalize temperature adaptation bilevel meta-optimization problem. The inner loop op- timizes the LLM policy Ty under temperatures sampled from the meta-policy Tæ. while the outer loop optimizes Fa to maximize the performance of the LLM policy.
Let T q: T) denote the model policy under temperature 7 (which we also denote as m..r( ( 4)). where q D is prompt. The meta-policy H5 specifies distribution over candidate temperatures.
The inner objective is
(x) = arg max EqED ETwRy Ensns(tar)(((r,,
9
while the outer objective seeks
arg max Jmeta (0' (d)):
where Jmesa denotes the evaluation metric of interest (e.g., expected reward). This bilevel formula- tion highlights the adaptive role of temperature, but not directly tractable in practice. Next, we describe our TAMPO.
3.3 TAMPO: MECHANISM AND IMPLEMENTATION
Tractability Challenge. In many LLM RL pipelines, rollout generation and policy optimization typically decoupled: rollouts generated using lower-precision models for efficiency, which
4

---

prevents backpropagation of gradients through the sampling process. This makes direct gradient- based optimization of the temperature meta-policy infeasible. Moreover, a naive trial-and-erre strategy-generatiing rollouts for each candidate temperature- computationally prohibitive. We thus seek a solution that works with the existing decoupled pipeline and requires no extra rollouts.
Key Observation. Every trajectory inherently encode its "prefereed" temperature, i.e., the temper- ature under which most likely to be generated (see Figure 2 Intuitively, for a high-reward (positive advantage) trajectory, should reinforce temperatures that increase its likelihood; for low-reward trajectory (negative advantage), we should down-weight such temperatures. This insight provides a tractable signal for adapting temperature.
Building on this idea, TAMPO treats temperature learnable meta-policy updated directly from trajectory-level signals. The meta-policy shifts probability mass toward temperatures associated with advantageous rollouts and suppresses ineffective ones, enabling adaptive exploration aligned with policy improvement.
3.3.1 TEMPERATUURE-DEEEDDDE TRAJECTORY LIKELIHOOD
The likelihood (i.e., probability) of trajectory at temperature T from policy 0
|
Average loo logilllllloodd
(o.
II No.r (oculs)))))
(5)
To remove dependence on trajectory length, we
the average log-likelihood (likelihood for short hereafter):
Parbat
To Rolbut2
Rolbut
Rolbut
(T:)))
log (T.)(Tt)
03
0.5
33
Tempersture
35
23
[r,
Figure 2: Example of trajectory likelihood under different temperatures for 8 rollouts prompt.
log ( s1,)).
(6)
We can probe the likelihoods of trajectory under set of virtual temperatures T3 T via (6) The trajectory likelihood is unimodal w.rt. T (see the proof in Appendix B). and there exists likelihood-optimal temperature T that maximizes the likelihood:
T = arg max Cy, (Tt). T.ET
Figure visualizes the trajectory likelihoods under different temperatures for 8 rollouts of a prompt, illustrating the unimodal nature. This implies that we can increase the likelihood trajectory by adjusting the temperature.
The empirical analysis in Appendix shows that positive advantage trajectories cluster to values T* distinct from those of negative advantage trajectories, indicating that sampling from suitable temperatures can preferentially increase the likelihood high advantage rollouts.
3.3.2 META-POLICY OPTIMIZATION WITH TEMPERATURE REWARDS
We formulate online temperature adaptation as a policy over temperature. key challenge is eval- uating the advantage of specific temperatures. Instead of sampling additional trajectories, we reuse the rollouts from inner-loop optimization. Particularly, for trajectory with positive advantage, we increase its likelihood by rewarding temperatures that lead to higher trajectory likelihoods. Con- versely, for trajectory with negative advantage, we down-weight its likelihood by punishing tem- peratures that lead higher trajectory likelihoods. This aligns temperature adaptation with policy optimization: reinforcing the likelihoods of high-reward trajectories, while suppressing likelihoods of low-reward trajectories.
Temperature-specific Advantage. Let T, denote a sampled trajectory and T .T....., Tx set of candidate temperatures. For each trajectory and virtual candidate temperature Tk. similar to
5

---

6 compute the trajectory's likelihood under the LLM policy 0 at temperature T3
lr, (T:) = log Pro (Ti)
log To, (OLa (1.)).
(8)
To capture the relative desirability of different temperatures, we normalize the likelihoods én, (Ti) across the K candidate temperatures using sparsemax (Martins & Astudillo 2016), yielding nor-
malized likelihood éx; (T;) with Σj= lx; (Ti) = 1. The trajectory's advantage A, (see (1 is then
scaled by these normalized likelihoods to produce the lemperature-specffic advantage:
A êy (T:) A4-
This can be interpretation below:
Positive-advvantgg trajectories (A; 0) reinforce the likelihood-optimal temperature most strongly, with neighboring temperatures receiving attenuated positive contributions.
Negative-advantage trajectories (A; < O) penalize the likelihood-optimal temperature most, while nearby temperatures inherit attenuated negative contributions.
Meta-policy Update. Using trajectory's iumeerauure-specffe advantages, we maintain a meta- policy x(T) over T. which characterizes probability distribution over T. For batch B of [B] samples, each with G generated trajectories, we aggregate the temperature-ppecffic advantages for each candidate temperature Tu:
B BBG
(Tv)
(10)
b,
b=1
where (Tt) denotes the temperature advantage of the i-th trajectory of sample b with respect to temperature Tre A(T)) serves the update target for the meta-policy, representing the batch-level temperature-specific advantage.
To stabilize updates, we maintain exponentially weighted moving average (EMA) of temperature- specific advantages:
A (1 a) +
+ (Tx)
(11)
where a E [0, 1) controls smoothing. denotes training step index. This EMA provides stabilized advantage estimate, reducing variance from individual batches while retaining responsiveness to new trajectory feedback.
Finally, the meta-policy Tx (T&) at step is computed simply via min-max normalization:
(T)) min, (T))
To (Ta) ΣØ (
A(TR)
K.
(12)
max, A min, A (T))
o (Tk) 1, favouring higher- This ensures TT forms valid distribution: ΣK
advantage temperatures while suppressing lower-advantage ones, and forming valid distribution: ΣK (Tk) 1.
3.4
OVERALL ALGORITHM
Algorithm summarizes the TAMPO, which operates as a hierarchical two-loop process:
Inner loop:
Optimizes the LLM policy using trajectories (e.g., via GRPO) sampled under the temperature selected with meta-policy.
Adaptively updates the temperature meta-policy. The same inner-loop trajectories are reused to update the meta-policy: trajectory advantages are modulated by their likelihoods under each candidate temperature as temperature-specific advantages (see Equations 0
Outer loop:
This trajectory-guided update enables principled online temperature adaptation, balancing explo- ration and exploitation without extra rollouts. TAMPO compatible with critic-fres RL algorithms and introduces negligible additional cost. We GRPO critic-free RL in experiments.

---

Algorithm Tempeeature Adaptive Meta Policy Optimization (TAMPO)
Require: LLM policy ka, reference policy Tet training step number S, temperature candidates T {Ti,.... TK). temperature meta-policy n, EMA decay a.
1: Initialize Ao (Tk) 0, (Tke 1/K for all Tj E T.
2: for to do
3: Outer Loop: Sample Temperature
4: Sample temperature T, ~ (T))
5:
6:
Inner Loop:
Critic-free Policy Optimization Sample a batch of prompts for each prompt E B do Generate G trajectories (TI....,Ta) using we at temperature T.. Compute reward r(Taa) and advantage Aq, (e.g., via GRPO) for each trajectory Tq. Update LLM policy 0 using advantages Ag.i-
7:
8:
9:
10:
11:
12:
end for
13:
14:
Outer Loop: Update Meta-Policy
Calculate temperature-specifie advantages A for all Ta (see (10 for each temperature T3 E T do
15:
16:
Update EMA: A(Th) t (1 a) ( a A (Tk)
18:
end for
Update meta-policy n. (Tx) based on the A and 12 for all T3 E
19:
20:
end for
EXPERIMENTS
4.1 SETUP OF MAIN EXPERIMENTS
Training Datasets and Benchmarks. We utilize public math reasoning dataset open-sl (Dang & Ngo 2025) for training. For evaluation, we use five distinct mathematiis-foousel bench- ATME24, MATH-500, AMC23, Minerva, and OlympiadBench- trum of difficulty levels and problem styles to comprehensively assess reasoning ability and gener- alization performance.
marks
broad spec-
Implementation Details. We use DeepSeek-Rl-Disse -i--issill---een--- 1.5B (DS-Qwen-1.5B for short) (Guo et al., (225) our base model to train both baselines and our models. We set our candidate temperatures in the range 0.6-1.5 with the interval 0.1, and K = 10, and the EMA coefficient a = 0.05 by default. In the warmup phase (i.e., the first 10% of training steps), we use fixed sampling temperature of 1.0 for LLM policy, while the meta-policy is updated. After the warmup phase, the samping temperature is determined dynamically by the online learned meta-policy. The maximum response length is set 6k tokens.
All experiments are conducted on NVIDIA 8x V100 GPUs. We train both baseline models and our models for 200 steps, using initial learning rate of x 10 warmup ratio of 0. followed by a cosine schedule. The training batch size is set to 32, with 8 rollouts per question.
Evaluation Protocol. We evaluate using Pass@1 and Pass@8 in order measure single-shot accuracy and performance under multiple sampled attempts, showing model's exploration potential solve questions. All evaluations are performed with maximum response length of 6k tokens and fixed decoding temperature of 1.0.
4.2
MAIN RESULTS
Comparison with Baselines. We adopt GRPO as the RL algorithm for both baselines and our scheme. 1wen-weenl5. refers to Di-icck-RR--istil-----eeeee model for all 1.5B model experiments. We construct baselines using a fixed sampling temperature 0.9, 1.2, and 1.5 during training, denoted by GRPO (T, 0.9), GRPO (T, 0.9), and GRPO (T,
which serves as the starting

---

Averuge
AIME24
MATHL-500
AMC23
Minerva
OlympiadBench
Method
Pass 6
Pass
Pass 1
Pass #
Pass 6P
@
Pass 6
Pass 8
Pass 6P
8
Pass
Pass
DS-Qwen-1.5B
39.1
57.8
13.3
26.7
76.2
89.2
45.0
72.5
22.8
41.5
38.4
59.0
GRPO ( (0.9)
42.0
60.8
20.0
30.0
75.2
91.0
50.0
80.0
26.1
43.4
38.7
59.4
(T, 1.2))
33.3
77.4
77.5
60.5
GRPO (T, 1.5)
42.6
62.1
23.3
36.7
75.4
90.8
52.5
77.5
22.8
44.5
39.0
61.2
(7, :0.9 1.5)
42.8
77.5
60.4
TAMPO (Ours)
60.7
Table 1: Comparison of TAMPO with baselines on math reasoning using 1.5B models, evaluated with Pass @ and Pass@8. 8. DS- DS-Qwen-1L.e Q05e-11.5B denotes DeepSeek-RI --bissk--R--sssssii--eeeee 1.5B (Guo et al. 2025), which serves as the base model for all training on the open-sl dataset. GRPO (T, 0.9) indicates a baseline trained with GRPO at fixed sampling temperature of 0.9. The maximum response length set to 6k tokens. Best results are in bold, and second-best results are underlined.
0.9), respectively. Additionally, we include a baseline with linearly increasing training temperature from 0.9 to 1.5, denoted as GRPO (T, 0.9 - 1.5).
Table reports Pass@1 and Pass@8 accuracy across the five evaluation benchmarks. On average, TAMPO outperforms the best fixed-temperature baseline, achieving 1.9% and 1.7% improve- ments Pass@1 and @ass@8, respectively. Across all datasets, TAMPO either achieves the best performance or the second best, highlighting the effectiveness of treating temperature as a learnable meta-policy.
Complexity. Our scheme has nearly the same computational complexity compared to the bascline schemes. (i) The meta-policy model extremely lightweight, as it only maintains list of temper- ature advantages, introducing negligible overhead during training and being discarded inference. (ii) TAMPO reuses rollouts from the inner loop, eliminating the need for generating additional tra- jectories. For the same number of training steps, both the baseline schemes using GRPO and our TAMPO use approximately the same training time (9 hours 54 minutes on an 8x V100 GPU machine for 200 steps).
4.3 ABLATION STUDY
Influence of EMA Coefficient a. As described in (II). a controls the exponential moving average smoothing, balancing the contribution of current feedback and historical accumulation. Table 2 reports the results for a {0.01, 0.05, 0.10). The increase a from 0.01 0.05 increases the average score from 41.6 to 44.5 and all benchmarks showing improvement. Further increasing a to 0.10 produces mixed effects. a 0.05 provides good trade-off between reducing variance and retaining responsiveness to new feedback.
Average
AIME24
MATH-500
AMC23
Minerva
OlympiadBench
0.01
41.6
20.0
75.2
50.0
25.4
37.5
0,05
44.5
23.3
76.8
55.0
27.9
39.6
0.10
43.6
23.3
75.4
57.5
23.2
38.8
Table Influence of the EMA coefficient a for our TAMPO. We report the performance on Pass
a distribution π over candidate temperatures. At each training step, similar to token sampling in LLM, we sample temperatures from и via nucleus sampling (i.e., top-p sampling), which selects actions from the smallest set whose cumulative probability exceeds threshold p. focusing the "nucleus" of likely outcomes while filtering out low-probability options. p controls the trade-off between exploration and exploitation in the meta-policy.
Influence of Sampling Strategy on Meta-policy. We model the meta-police
O. When p = O, this corresponds to greedy sampling, selecting the temperature with the highest probability (equivalent top-k sam- pling with 1). We can see that a moderate threshold p = 0.7 achieves a good balance between exploration and exploration temperature. We set p 0.7 by default. Too high threshold lim-
Table 3 shows results for top-p sampling with p = 0.9,
8

---

Sampling strategy
Average
AIME24
MATH-500
AMC23
Minerva
OlympiadBench
Top-p (p: 0.9)
43.0
20.0
76.6
52.5
26.1
39.7
Top-p (p: 0.7)
44.5
23.3
76.8
55.0
27.9
39.6
Top-p (p: 0.5)
42.2
23.3
75.4
50.0
24.3
38.1
Top-p (p: 0, greedy)
40.9
20.0
73.8
50.0
23.5
37.2
Table 3: Influence of sampling strategy on meta-policy. We report the performance on Pass @
1.4
1.3
Temperatur
1.0
0.9
0.8
80
100
120
140
160
180
200
Training step
Figure 3: Sampling temperatures from the meta-policy using nucleus sampling.
its exploitation of the learned meta-policy, sacrificing certain opportunities to exploit the benefit of learned meta-policy. Too low threshold (e.g., p 0.5) greedy sampling (i.e., p 0) leads to under-expia under-exploration, reducing effectiveness. Since the greedy sampling always sample temperatures that having the highest advantages, even though the second-best temperature may have similar ad- vantages, such under-exploration of temperatures leads to LLM policy losing opportunity generate rollouts under those temperatures and reducing the experience diversity.
4.4 ANALYSIS
Sampling Temperature from Meta-policy. Jsows the temperatures sampled by the meta- policy using nucleus sampling (p=0.7), reporting the mean and standard deviation over a sliding window of 25 training steps. After 20 warm-up steps, the meta-policy favors high temperature (around 1.3), encouraging exploration in LLM policy generation. As training proceeds, the mean temperature gradually decreases, shifting toward exploitation while maintaining diversity. The large variance arises from nucleus sampling, which balances exploitation the learned meta-policy with exploration over diverse temperatures.
4.5
MORE EXPERIMENTS RESULTS
To evaluate TAMPO's effectiveness across different base models and domains, conducted ad- ditional experiments. Specifically, we implemented TAMPO on Dwen2.5--B-Inntruus (Yang et al.) 2024) base model and evaluated on ECQA (Aggarwal al.), commonsense reasoning benchmark. The results in Table 4 show consistent improvements, demonstrating that TAMPO generalizes be- yond mathematical reasoning.
Method
Pass @
Pass@8
Owen2.553B--nnsiiie IBssruttttt (no RI.)
73.06%
77.76%.
GRPO
75.07%
78.94%
TAMPO
76.12%
79.67 %
Table 4: ECQA results evaluated with Pass@ and @a

---

5 CONCLUSION
We presented TAMPO, a hierarchical framework that treats temperature as learnable meta-policy LLM reinforcement learning. By reusing trajectories to adaptively update a temperature meta- policy, TAMPO enables exploration without extra rollouts and achieves consistent improvements on mathematical reasoning benchmarks. These results demonstrate that principled temperature adapta- tion is a practical and effective tool for advancing LLM reinforcement learning.
LLM USAGE
We used large language models (LLMs) to assist with refining the writing and presentation of this paper. LLMs were employed for improving clarity, conciseness, and formatting, while all ideas, methods, and experiments were conceived and executed by the authors.
REPRODUCIBILITY STATEMENT
We have made extensive efforts to ensure the reproducibility of our work. Our temperature adap- tation method and algorithm are described in $ 3.3 The training details, including base models, hyperparameters, and optimization settings, uation benchmarks used in this study are publicly available. We will release the source code upon acceptance further facilitate the verification and reproduction of our results.
described in 4.1 The training datasets and eval-
REFERENCES
Shourya Aggarwal, Divyanshu Mandowara, Vishwajeet Agrawal, Dinesh Khandelwal, Parag Singla, and Dinesh Garg. Explanations for CommonsenseeA:: New Dataset and Models.
Chenxin An, Zhihui Xie, Xiaonan Li, Lei Li, Jun Zhang, Shansan Gong, Ming Zhong, Jingjing Xu, Xipeng Qiu, Mingxuan Wang, and Lingpeng Kong. Polaris: post-training recipe for scal- ing reinforcement learning advanced reasoning models, 2025. URL https: //ku//pp github. httbb/o//002////0aaaa
Pierre-Lue Bacon, Jean Harb, and Doina Precup. The option-critic architecture. In Proceedings of the AAAI conference on artificial intelligence, volume 31, 2017.
Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. Constitutional ai: Harm- lessness from ai feedback. arXiv preprint arXiv:22/2.08073. 2022.
Qiguang Chen, Libo Qin, Jinhao Liu, Dengyun Peng. Jiannan Guan, Peng Wang, Mengkang Hu, Yuhang Zhou, Te Gao, and Wanxiang Che. Towards reasoning era: survey of long chain-of- thought for reasoning large language models. arXiv preprint arXiv:2503.0950 2025.
Quy-Anh Dang and Chris Ngo. Reinforcement learning for reasoning small 1ms:: What works and what doesn't. arXiv preprint urXiv:2503.16211 2025.
Weihua Du Yiming Yang, and Sean Welleck. Optimizing temperature for language models with multi-sample inference. arXiv preprint arXiv:2502.00234, 2025.
Kevin Frans, Jonathan Ho, Xi Chen, Pieter Abbeel, and John Schulman. Meta learning shared hierarchies. arXiv preprint 1rXiv:1710.09777. 2017.
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang. Xiao Bi, et al. Deepseek-RI Incentivizing reasoning capability in 1lms via reinforcement learning. arXiv preprint arXiv:2501. 2025.
Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The curious case neural text degeneration. arXiv preprint 19Xiv 2019.
10

---

Jian Hu, Jason Klein Liu, Haotian Xu, and Wei Shen. Reinforce++: An efficient rlhf algorithm with robustness to both prompt and reward models. arXiv preprint arXiv:2501.03262. 2025.
Leslie Pack Kaelbling, Michael L Littman, and Andrew W Moore. Reinforcement learning: A survey. Journal of artificial intelligence research, 4:237-285, 1996.
Wouter Kool, Herke van Hoof, and Max Welling. Buy 4 reinforce samples, get baseline for free! In International Conference on Learning Representations (ICLR) Workshop, 2019.
Zihan Liu. Zhuolin Yang, Yang Chen, Chankyu Lee, Mohammad Shoeybi, Bryan Catanzaro, and
Wei Ping. Accreason-nemootron 1.1: Advancing math and code reasoning through sft and synergy. arXiv preprint arXiv:2506. 13284, 2025.
Andre Martins and Ramon Astudillo. From softmax to sparsemax: A sparse model of attention and multi-label classification. In International conference on machine learning, pp. 1614-1623. PMLR. 2016.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to fol- low instructions with human feedback. Advances neural information processing systems, 35: 27730-27744, 2022.
Deven Sethi, David Šiška, and Yufei Zhang. Entropy annealing for policy mirror descent in contin- uous time and space. SIAM Journal on Control and Optimization, 63(4):3006-304) 2025.
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al. Deepseekmath: Pushing the limits of mathemati- cal reasoning in open language models. arXiv preprint arXiv:2440..33300, 2024.
Han Shen. On entropy control in Ilm-rl algorithms. arXiv preprint
2025.
Richard S Sutton, Andrew G Barto, et al. Reinforcement learning: An introduction, volume 1. MIT press Cambridge, 1998.
Alexander Sasha Vezhnevets, Simon Osindero, Tom Schaul, Nicolas Heess, Max Jaderberg, David Silver, and Koray Kavukcuoglu. Feudal networks for hierarchical reinforcement learning. In International conference on machine learning, pp. 3540-3549, PMLR, 2017.
Da Wang. Wei Wei, Lin Li, Xin Wang, and Jiye Liang. Rethinking exploration-exploitaiee trade-off in reinforcement learning via cognitive consistency. Neural Networks, 187, 2025.
Yufei Wang and Tianwei Ni. Meta-sac: Auto-tune the entropy temperature of soft actor-critic via metagradient. arXiv preprint 2:X0::2007 2020.
Wikipedia contributors. Exploration-exploiiatioon dilemma, 2025. URL https: //en. di lemma
wikipedia h ottp::/kw/Appooaaaaaoo//222/2aaa cessed: 2025-08-18 18.
Ac-
Zhongwen Xu, Hado P van Hasselt, and David Silver. Meta-gradient reinforcement learning. Ad- vances in neural information processing systems, 31, 2018.
An Yang, Baosong Yang. Binyuan Hui, al. Qwen2 technical report. arXiv preprint
arXiv:2407. 10671, 2024.
Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu. et al. DAPO: An open-source Ilm reinforcement learning system at scale. arXiv preprint arXiv:2503.14476 2025.
Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. arXiv preprint ar xrv::
2019.
11

---

A GRPO OPTIMIZATION OBJECTIVE
GRPO updates the policy Te,r by maximizing the expected advantage while including a KL regu- larization term with respect to reference policy:
JGRPO (0) (r.
mnd
Σ (pL.(0.TTA,,clp((ood0..... -00,1111+++pp c)a.) - (Dxx(To. /meet))
G
(13)
where PT)
(o. T (or,e (ior,ce))
6 and & are hyperparameters.
B PROOF OF THE UNIMODAL OF TRAJECTORY LIKELIHOOD W.R.T. T
(a1,..., an) of tokens, each token a, generated based on the state = (g. aee)) with policy model parameterized by 8. Here denotes the prompt/question. Un- der temperature scaling of > the conditional distribution can obtained based on the model output logits =(a 8t) as
For a trajectory
cxp (2(a st)/T)
Tarr(a st) = Ev exp (=(a' st)/T)
(14)
The ligeliill of the trajectory (a1, 02 ................................ an) is
(T(T) = log (T) where Pe,r(T) = II
ne Fe,r(a sc).
(15)
We define baseline distribution at T=1 under condition s
(a(())))
(a) Σn c=(u]ss)
(16)
Then,
log qi (a) = =(a st) - log (u(()))
(17)
Let 1/T which denotes the inverse temperature. We have
ge(a)
<a 8t) Σ. 9e (u)
(18)
Let us define S(7) = Σ._ log ge (at). A((B) = logE,, (u) and A(3) Σ Ar(3). We
obtain the standard exponential family form:
log (o.a(T) S(T) A(3).
(19)
Second derivative w.r.t the inverse temperature 3 (unimodality): Based on standard properties,
A₂(3) = ( (o
A((33) Var (-lee) (or 2
(20)
Then,
&
dza log (a.
S 0.
(21)
Therefore, log Po.a(T) is strictly concave in 3. Since 3 = L/T is monotone reparameteriza- tion. Pe) unimodal in T, with a one-to-one correspondence between maximizers: 8 (T) (T)*
Wee denote g (acl))) by (0<) for conciseness.
12

---

First derivative w.r.t. temperature and endpoint limits (typical "rise-then-fall"): We have
=(a 8t)
log To st)
T log Σ exp(2(u st)/T).
(22)
Let us define (8)) = B e~ o (+).)))/ 8x)] A direct differentiation yields
log ma 88) - 2(a
88) - pe p
(23)
d7
Therefore,
(T( dl
d log Po Pa. =
" Σ
>(( st) - p
(24)
As - 0+, each step tends to select the action with maximum logit. Therefore, ( max, z(a sz). When the trajecotry is not composed by greedy sampling of actions/tokens, z(a] < pT (8x) and then (T) > 0, i.e,, a small increase of T increases log Po,r(T).
As T + xc, T approaches uniform distribution and pr((s)) + E(sr) = - z(a st). The LLM sampled tokens in general have ={a(s;) > 7(s)) (except for edge case), making the sum positive and (T) < 0, i.., a increase of T reduces log (T.
Conclusion: unimodality and "rise-then-fall. Combining the striet concavity w.r.t 3 (Eq. 21) and the opposite signs of (T) at the low- and high-temperature limits (Eq, (24. 24). we conclude that, for trajectories except edge cases, there exists unique T (T) (0, oo) such that
(T) > 0 for < T", (T*) = O, and (T) for T>
(25)
Log-likehood log (T) is typically unimodal w.r.t I" and exhibits a "rise-then-fall" shape beside edge cases.
Edge cases. First, if trajectory is composed of greedy tokens, that is, every step LLM policy picks the most probable token, then its likelihood decreases monotonically with T, approaching T - 0+. Second, if trajectory has many chosen tokens having probability below |V]-1, i.e. which rarely occur under the decoding strategy, its likelihood will slowly increase when T increases.
The unimodal property of the trajectory log-likelihood for general cases, and the monotonicity for edge cases, all assure that we can use discrete temperatures (as virtual temperatures) to well estimate the likelihood of trajectories under different temperatures.
C CONNECTION BETWEEN TEMPERATURE AND ADVANTAGE
To gain more intuitive understanding of the correlation between trajectory advantage and likelihood-optimal temperature, we conducted training under fixed sampling temperatures (0.9, 1.2, and 1.5) using GRPO and computed the likelihood-optimal temperature (*(r) for each trajectory 7. Figure 4 shows the resulting distributions for trajectories with positive advantages (green) and negative advantages (red) along training, with mean and variance computed for rollouts of every 5 training steps.
Three patterns can be observed. (i) A fixed sampling temperature generally does not align with the likelihood-optimall temperatures observed during training- (ii) Positive-advantage trajectories cluster around distinct T* values compared to negative-advantage trajectories, indicating that high-rewart rollouts are intrinsically associated with specific temperature regimes. (iii) At extreme sampling temperatures, trajectories tend to revert toward moderate T* values: when T 1.5, most T are well below 1.5, while at T = 0.9, most T are well above 0.9. This demonstrates that both excessive randomness and excessive determinism reduce the likelihood of producing advantageous rollouts.
Overall, these results suggest that there exist more optimal temperatures that increase the proba- bility of high-advantage trajectories, highlighting substantial room for improving the likelihood of sampling advantageous rollouts.
D SYSTEM PROMPT
Figure shows the system prompt used during training.
13

---

1.5
Sampling ing tamperatune
Unafnood-gge prooddappiiaa bempe pemperater AorA
Indffrodioptinall Homperaturs 0
Temperature
1.0
0.9
0.8
180
Training step
(a) Fixed T, 0.9
1.2
Temperature
1.1-
1.0
0.9
Campl thg temperatunet.c. 0
Licelhood--pe coc---ptimaa temperature S-
Litelhood- o20 temperature for c
Training step
(b) Fixed T, 1.2
1.5-
1.4
Temperature
1.2
1.1-
1.0
Gampl ing temperatunet--pg
Licelhord-zz o20 temperattre a
Litalhood-aze ap0 bamperr temperature for x
160
Training step
(c) Fixed T. 1.5
Figure 4: Distribution of trajectory likelihood-optimal temperatures under three fixed training tem- peratures, respectively. Green curve corresponds to the tikelihood-optimal temperatures of positive- advantage trajectories (AA >> 0), red curve to negative-advantage trajectories (A4 < 0), and blue curve the sampled fixed temperature.
conversation between User and Assistant. The user asks questien, and the Assistant solves it. The assistant first thinks about the reasoning process in the mind and then provides the user with the anawer. The reasoning process and answer are enclosed within <think> </think> and <answer> </answer> tags, respectively, i.e., <think> IREASONING PROCESS HEREI </think> Therefore, the final answer is: <answer> $ ( (ANSKER CONTENT HERE </answer>
Figure 5: System prompt for the policy model.
14


Reviews:
[{"review_id": "tCPRisDriJ", "reviewer": "Reviewer_c2ti", "summary": "The paper proposes Temperature Adaptive Meta Policy Optimization (TAMPO), a framework to dynamically adjust the sampling temperature of a LLM during reinforcement learning. Instead of treating the temperature as a fixed hyperparameter, TAMPO treats it as a learnable meta-policy that is optimized alongside the main policy. The approach operates in a hierarchical two-loop process: an inner loop updates the LLM’s policy (using a critic-free RL algorithm like GRPO) with trajectories sampled at a temperature chosen by the meta-policy, while an outer loop updates the meta-policy by rewarding temperatures that yield high-advantage trajectories. The authors evaluate TAMPO on five mathematical reasoning benchmarks (e.g., AIME, MATH, AMC, Minerva, Olympiad datasets), comparing against fixed and heuristically scheduled temperatures", "questions": "1. Which prior approaches to adaptive exploration or hyperparameter tuning in RL are most closely related, and how does TAMPO differ from them? The submission would benefit from discussing additional related works that were not cited. For example, methods that dynamically adjust exploration parameters (entropy, $\\epsilon$-greedy schedules, or temperature annealing strategies in RL) and meta-learning approaches for RL hyperparameters deserve mention\n2. What is the effect of TAMPO’s design decisions such as using a discrete candidate set of temperatures and the chosen range [0.6, 1.5]? The method currently relies on a fixed set $\\mathcal{T}$ of 10 temperatures and updates a categorical distribution over them. This discretization might limit the precision of the optimal temperature found.\n3. Can the authors provide more intuition or evidence about the underlying assumption that trajectory likelihood vs. temperature is unimodal?\n4. The paper's outer-loop update contains two confusing choices.a) Why use sparsemax to normalize the likelihoods? Sparsemax is designed to create sparsity, which seems to contradict the stated goal of giving attenuated positive contributions to neighboring temperatures. Would a standard softmax not be more appropriate?b) Why use min-max normalization (Equation 12) for the final policy update? This is a simple heuristic. \n5. Given the citation to meta-gradient methods, why not use a proper policy gradient update on the meta-policy $\\pi(T)$, using the aggregated advantage as the advantage signal for the action $T_k$?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. The technical approach is sound and well-motivated. The paper identifies a clear limitation of existing LLM RL methods – the use of fixed or manually tuned exploration temperature – and offers a principled solution.\n2. Significant practical strength of TAMPO is its decoupled outer-loop update mechanism, which enables online adaptation without additional rollouts。", "weaknesses": "1. Failure to Contextualize within the Meta-Gradient Literature: This omission is compounded by the paper's flawed positioning within the meta-learning literature it does cite.The paper does cite \"meta-gradient methods\" (Xu et al., 2018) for learning other hyperparameters like $\\gamma$ and $\\lambda$. It fails to cite the direct combination of these two concepts: \"Meta-SAC: Auto-tune the entropy temperature of soft actor-critic via metagradient\" (Wang & Ni, 2020), which applies a meta-gradient approach to this exact problem.\n2. The paper asserts: \"The trajectory likelihood is unimodal w.r.t. $T$\". This unimodality is what justifies searching for a single likelihood-optimal temperature $T^{\\star}$ (Equation 7). No direct evidence found in the manuscript. Figure 2 provides only 8 illustrative examples, which is not a proof.If this claim is false (e.g., if the likelihood function $\\ell_T(\\tau_i)$ is multi-modal), the entire basis for the update mechanism is flawed. The method would reinforce a local likelihood-optimal temperature, and the neighboring contributions (§3.3.2) would be meaningless. This is a critical gap in the paper's technical soundness.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mFOvw7eVwQ", "reviewer": "Reviewer_Eawz", "summary": "This paper proposes Temperature Adaptive Meta Policy Optimization (TAMPO) that can dynamically adjust the temperature of large language models (LLMs) for reinforcement learning (RL)-based post-training of LLMs. More specifically, TAMPO consists of a hierarchical two-loop process: inner loop and outer loop. The inner loop optimizes the LLM policy, and the outer loop learns the temperature meta-policy. This paper evaluates TAMPO on five math reasoning benchmarks including AIME24, MATH-500, AMC23, Minerva, and OlympiaBench. The experiment results show that TAMPO can achieve higher scores than a base model (i.e., DeepSeek-R1-Distill-Qwen-1.5B) and GRPO-based post-trained models.", "questions": "- Q1. [Overhead] Compared to the baseline method (e.g., GRPO-based post-training), how much computational complexity does the TAMPO have?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "- S1. [Presentation] First of all, this paper is well written and organized.\n\n- S2. [Novelty] The basic idea of learning LLM temperature for RL-based LLM post-training (i.e., meta-policy learning) seems novel.", "weaknesses": "- W1. [Performance] For meta-policy learning, TAMPO additionally calculates likelihoods at virtual temperatures when training the policy model. This increases the computational complexity of the GRPO-based LLM post-training. However, compared to the basic GRPO-based post-training (pass@1: 42.6%), TAMPO provides a slightly higher average accuracy (pass@1: 44.5%). \n\n- W2. [Hyper-parameters] Even though this paper proposes to learn the temperature meta-policy, this may introduce additional hyper-parameters to search (e.g., EMA coefficient and sampling strategy).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dd9mlTeOMX", "reviewer": "Reviewer_a4qg", "summary": "This paper proposes Temperature Adaptive Meta Policy Optimization (TAMPO), a new framework that recasts temperature control in large language model (LLM) reinforcement learning (RL) as a learnable meta-policy rather than a fixed hyperparameter.\nIn typical LLM RL (e.g., GRPO or PPO-based RLHF), temperature controls the exploration–exploitation balance during text generation.\nHowever, most existing methods use static or heuristic temperature schedules, which fail to adapt to the evolving training dynamics.\nTAMPO formulates temperature control as a two-level learning process: \nIn the inner loop, the LLM policy is optimized using critic-free RL (e.g., GRPO) at the temperature chosen by the meta-policy. \nIn the outer loop, the meta-policy updates its temperature based on temperature-specific advantages by reusing the inner loop rollouts, reinforcing those that are more likely to generate trajectories with high advantage trajectories, without requiring additional rollouts.\n\nExperiments on five distinct mathematics-focused benchmarks (AIME24, MATH-500, AMC23, Minerva, and OlympiadBench) show that TAMPO outperforms fixed or heuristic temperature baselines, in terms of Pass@1 and Pass@8 accuracy, by approximately 1.9% and 1.7%, respectively.", "questions": "- It would be helpful to know whether the authors have conducted any ablation studies to assess the generality of TAMPO: \n  1. Using base models other than DeepSeek-R1-Distill-Qwen-1.5B, \n  2. Across different domains such as dialogue or code generation, and\n  3. With alternative critic-free reinforcement learning algorithms.\n- Could the authors clarify what the optimal solution of the iterative optimization represents, both from a theoretical and an intuitive perspective?\n- Could the authors elaborate on whether they plan to extend TAMPO to continuous temperature optimization rather than relying on a fixed discrete set of candidates?\n- Why does adaptive temperature lead to improved reasoning quality? In addition, are there any potential negative effects of adaptive temperature, such as reduced exploration or excessive exploration?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- This work provides a interesting way to control exploration in LLM RL by recasting temperature as a meta-policy variable.\n- TAMPO efficiently reuses existing rollouts for meta-policy updates, introducing negligible additional cost.\n- TAMPO demonstrates consistent improvements across multiple reasoning benchmarks using DeepSeek-R1-Distill-Qwen-1.5B.", "weaknesses": "- Experiments are restricted to mathematical reasoning tasks $\\to$ generalization to other domains (dialogue, code generation, ...) remains untested.\n- It’s unclear how TAMPO can be applied to critic-based or hybrid RLHF methods.\n- The approach uses a fixed discrete set of temperatures {0.6, 0.7, ..., 1.4, 1.5}. Continuous temperature optimization might yield smoother adaptation.\n- The paper does not include an ablation study for different base models.\n- While results are strong, interpretability and deeper insight into how adaptive temperature affects reasoning quality are somewhat limited.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BvmDzvIsyh", "reviewer": "Reviewer_aLbY", "summary": "This paper proposes an intuitive method TAMPO, which replacing the conventional static or heuristic temperature to a meta-policy and learning it online, thereby achieving trajectory-driven exploration–exploitation adaptation in LLM reinforcement learning. In detail, the framework consists two loop process: inner loop generates trajectories under temperatures determined by meta-policy and optimized (using GRPO) LLM, and outer loop optimize meta-policy based on temperature-specific advantage, which can be calculated by reused trajectories likelihood from inner loop under certain temperature. Experiments on mathematical reasoning tasks show robust improvements over several fixed or linearly scheduled baselines, along with ablations and analyses (e.g., temperature trajectories, T* distributions), demonstrating the potential and interpretability of the proposed method.", "questions": "Please refer to Weakness.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "S1: Treating temperature as a meta-policy is a fresh and practical perspective in the context of LLM RL. It addresses a real limitation of current methods that use fixed or heuristically scheduled temperatures.\nS2: TAMPO can be seamlessly combined with existing critic-free RL methods (such as GRPO and REINFORCE++) without adding additional sampling costs, and has good practicality and promotion potential.\nS3: The paper is well-written and easy-to-follow.", "weaknesses": "W1: The current meta-policy employs a shared temperature distribution across the entire dataset, without explicit conditioning on the prompt, subtask, or training stage. This may lead to inefficiency in multi-task settings or under highly heterogeneous prompts.\nW2: As the reinforce learning is known by its high variance, report the mean +- standard deviation along with significance tests under at least 3-5 independent runs is preferred to demonstrate the effectiveness of the methods.\nW3: All experiments are conducted on a 1.5B parameter model and focus solely on mathematical reasoning tasks. It is unclear how well the method generalizes to larger models or other types of tasks (e.g., code generation, dialogue, summarization).\nW4: Typo: Line 356 and 357, “using initial learning rate 1x10e-6” but “minimum learning rate of 0.1” is confusing and counterfactual.\nW5: The range (0.6-1.5) and interval (0.1) of the candidate temperature set are manually preset without an adaptive adjustment mechanism. Moreover, the applicability of this fixed candidate set to different models or task scenarios is not verified, which may lead to missing better exploration intervals due to the limitations of the temperature search space.\nW6: The trajectory likelihood relied on for meta-policy updates is strongly coupled with the dynamically iterated LLM policy in the inner loop. No mitigation measures are designed for the likelihood estimation bias caused by policy updates, which may result in the evaluation of temperature-specific advantages being affected by the current policy bias, reducing the stability of temperature adaptation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)