Learning What Matters Now: Dynamic Preference Inference under Contextual Shifts
Authors: Xianwei Cao
Dou Quan
Zhenliang Zhang
Shuang Wang
Venue: ICLR 2026 Poster (2026)
Keywords: Dynamic Value Preference Inference
Cognitive Modeling
Adaptive Decision-Making
Primary area: applications to robotics, autonomy, planning

Abstract: Humans often juggle multiple, sometimes conflicting objectives and shift their priorities as circumstances change, rather than following a fixed objective function. 
In contrast, most computational decision-making and multi-objective RL methods assume static preference weights or a known scalar reward. 
In this work, we study sequential decision-making problem when these preference weights are unobserved latent variables that drift with context. 
Specifically, we propose Dynamic Preference Inference (DPI), a cognitively inspired framework in which an agent maintains a probabilistic belief over preference weights, updates this belief from recent interaction, and conditions its policy on inferred preferences. 
We instantiate DPI as a variational preference inference module trained jointly with a preference-conditioned actor–critic, using vector-valued returns as evidence about latent trade-offs. 
In queueing, gridworld maze, and multi-objective continuous-control environments with event-driven changes in objectives, DPI adapts its inferred preferences to new regimes and achieves higher post-shift performance than fixed-weight and heuristic envelope baselines.

Published as a conference paper at ICLR 2026
LEARNING WHAT MATTERS Now: DYNAMIC PREFER- ENCE INFERENCE UNDER CONTEXTUAL SHIFTS
Xianwei Cao12 Dou Quan'
Zhenliang Zhang** Shuang Wang
SShooo of Artificial Intelligence, Xidian University. Xi'an an. 710071, China State Key Laboratory General Artificial Intelligence, BIGAI, Beijing, China
21 thzangebigai. ai, shwang@mail xidiann aediian.edu.com en
ABSTRACT
Humans often juggle multiple, sometimes conflicting objectives and shift their priorities as circumstances change, rather than following a fixed objective func- tion. In contrast, most computational recision-making and multi-objective RL methods assume static preference weights or a known scalar reward. In this work, we study sequential decision-making problem when these preference weights are unobserved latent variables that drift with context. Specifically, we propose Dy- namic Preference Inference (DPI), cognitively inspired framework in which an agent maintains a probabilistic belief over preference weights, updates this belief from recent interaction, and conditions its policy on inferred preferences. We in- stantiate DPI variational preference inference module trained jointly with preference-coneeee conditioned actor-critic, using vector-valued returns as evidence about latent trade-offs. In queueing, gridworld maze, and multi-objective continuous- control environments with event-driven changes in objectives, DPI adapts its in- ferred preferences regimes and achieves higher post-shift performance than -ixiht and heuristic envelope baselines
INTRODUCTION
Human behavior is widely modeled goal-directed and value-driven, but people typically juggle multiple goals and adjust their priorities as circumstances change, rather than acting on a single fixed priority ordering (Simon 1955 Payne et al.) 1993 Carver & Scheier, 2001] Wrosch et al.) 2003). We reweight priorities, abandon infeasible objectives, and reorient toward what remains achievable. For instance, illustrated in Fig. [ a person waiting in line may initially value fair- ness and patience, but as hunger escalates and time runs out, they may rationalize cutting ahead. Work on self-regulation, multiple-goal pursuit, and constructed preferences formalizes such behav- ior as feedback-
goal control and context-dependent weighting of attributes (Payne et al., 1993] Slovic, [195] [ichtenstein & Slovic (2006). Yet. despite its importance, computational modeling of dynamic value preference adaptation remains underexplored in artificial intelligence and multi-
objective decision-making (Roijers et al., 2013 Yang al., 2019 Agarwal al., 2022 Basaklar
et al. 2023
Liu et al. 20255.
large literature in cognitive psychology and cognitive science has examined how people regu- late goals and reconcile competing motives. Theories of self-regulation and multiple-goal pursuit emphasize feedback-bast
adjustment of goal importance and effort allocation (Carver & Scheier, 2001 2004; Wrosch et al., (2003), while multi-attribute and context-dependent choice models show that attribute weights and even preferences themselves can shift with task demands and elicitation formats (Payne et al., 1993] Slovic, 1995, Lichtenstein & Slovic 2006). Abstracting from these literatures, we conceptually separate two tightly coupled processes: (1) value appraisal- forming an internal judgment about what matters most right now in a given situation; and (2) action selec- tion tion-choosing behavior conditional on the current value preference. In this paper, we use these
"Corresponding authors.
'Code is available httppps////iihhhb.ccc///

---

Published as a conference paper at ICLR 2026
Time step
Time step
Time stop
eady.
Hold atmout my fum....
ronning out. need get afread loe.
Ageni
Agont
Agust
Value weight
Morality
Energy
0.3
0.7
0.5 0.5
0.8 0.2
Figure I: Adaptive value preference adjustment in a queueing scenario. At early stages (t1), the agent prioritizes morality and chooses to wait. As the deadline approaches (t2), preferences between morality (M) and energy (E) become balanced. When time is nearly exhausted (t3), energy becomes dominant and the agent rationalizes cutting in line, illustrating dynamic reweighting of values under changing pressures.
terms purely as labels for components of our computational framework: the second has been exten- sively studied as policy optimization under given reward utility function, whereas the first-how an agent constructs and updates its value function from experience and context-remains compar- atively underexplored in computational models. From a modeling perspective, many psychological and Mhoice-theorete accounts provide rich descriptions of how humans adjust goals and preferences, but they are not directly formulated as algorithms that can be deployed in high-dimensional, partially observable, and non-stationary control problems. In contrast, artificial agents in safety-critical do- mains must make a stream of sequential decisions from raw observations, under changing resource, time, or risk constraints. This motivates the computational question at the core of our work: given only vector-valued rewards and partial observations can an agent infer and adapt its current trade-off over objectives online in way that is both effective and interpretable? We address this question in the language of multi-objective reinforce- ment learning, using dynamic preference inference bridge between cognitive theories of goal regulation and practical control algorithms.
a non-stationary environment, how
Many computational models of sequential and modern reinforcement learning agents-assune a fixed and externally specified utility or re- ward function (Bellman & Kalaba 1957 Russell & Norvig 1995). While this assumption simpli- nature of value trade-offs
Markov decision processes
fies learning and planning, it fails to capture the fluid, context-dependee observed in realistic settings. In mblti-objecctive environments (e.g., efficiency vs. safety, energy vs. morality), agents inevitably face shifting constraints: some goals become infeasible, temporarily irrelevant, or disproportionately important as resources, time, or external conditions change (Roijers) et al. 2013: Vamplew et al.) 201T). Without the capacity for dynamic value reconfiguration, such agents risk either pursuing unattainable goals or neglecting emergent -eaddnng to subop- timal or even catastrophic outcomes in domains such as autonomous driving or healthcare (Amodei ct al. 2016 Dulac-Arnoke et al. 2021).
To mitigate this limitation, recent work in multi-objective decision-makinn has developed preference-conditione policies, where an agent is trained to optimize under different trade-offs specified by a preference vector (Van Moffaert & Nowé, 2014 Yang et al., 2019) Basaklar al., 2023 Liu et al., 2025). Such approaches enable generalization across static preferences and com- plement earlier scalarization and envelope methods (Vamplew et al.) [011] Mossalam et al. 2016). However, they typically assume that the preference vector is given. In realistic scenarios, prefer- ences are rarely observed directly: the agent must infer its priorities from incomplete, noisy, and evolving perceptual data. This calls for an online preference inference mechanism that is sensitive to environmental cues yet robust to transient noise capability that is both cognitively plausible and computationally underexplored.
In this work, we propose a Dynamic Preference Inference (DPI) framework that makes this gap explicit. The key idea is to treat the preference vector encoding the relative importance of multiple objectives as a latent state that must be inferred online rather than fixed a priori. The agent maintains posterior distribution go go( st-H+1:)) over current preference vector w., parameterized by recurrent encoder over recent history. Sampling from this posterior captures epistemic uncertainty and lets the agent explore alternative value configurations before acting. A preference-condiiionee actor critrccritic then conditions its policy on w, to adapt behavior as task demands shift. To keep updates stable and interpretable, we regularize da with Gaussian prior and directional alignment term
2

---

Published as a conference paper at ICLR 2026
that encourages preference changes consistent with observed return vectors. DPI thus provides compact variational Bayesian formulation of value appraisal, in which the agent maintains a belief over "what matters now" and revises it only when recent outcomes provide sufficient evidence.
The main contributions of this work are:
We formalize setting where preference weights in environments are unknown and dy- namically varying, and highlight the challenge of enabling agents to infer and adapt their priorities online from experience.
We introduce the Dynamic Preference Inference (DPI) framework, a computational ar- chitecture that jointly learns (i)a probabilistic preference inference model from perceptual history and (ii)a preference-conditiones policy, regularized for stability and interpretability.
We empirically evaluate DPI in dynamic Queue environment and dynamic Maze envi- ronment, showing consistent gains over fixe--peference and static-inference baselines in adaptability, robustness, and cumulative performance, and illustrating more interpretable, event-aligned adaptive strategies.
RELATED WORK
2.1
MUL.TI-OBJECTIVE DECISION-MAKING COGNITIVE SCIENCE
Human decision-making rarely involves optimizing single objective in isolation. Instead, individ- uals continuously negotiate multiple, sometimes conflicting goals such as efficiency, fairness, en- ergy preservation, and social norms. Classical theories of bounded rationality (Simon 1955) argue that humans rely on satisficing heuristics rather than globally optimal strategies. Dynamic models such Decision Field Theory (Busemeyer & Townsend 1993) and Prospect Theory (Kahneman & Tversky (2013) emphasize that preferences evolve with time pressure, risk, and context. Research on multiattribute decision-making (Payne al.] 1993 Zanakis et al. 1998) shows that humans flexibly reweight attributes depending on contextual demands example, prioritizing efficiency under time pressure or fairness cooperative settings. Theories of self-regulation and control fur- ther highlight that goal pursuit is adaptive, context-sensiivve and autonomy-driven (Shenhav et al.) 2013 Deci Ryan, [985] 2012). These insights motivate computational frameworks that treat preferences not fixed constants, but latent variables dynamically inferred from context.
2.2 COMPUTATIONAL APPROACHES TO MULTI-OBJECTIVE DECISION-MAKING
Multi-ohjective reinforcement learning (MORL) provides principled framework for sequential Decision-making with vector-vale valued rewards. Classical approaches rely scalarization (Vamplew ctt al. 2011 Roijers et al. 2013 Agarwal et al. 2022). collapsing the reward vector into a scalar us- ing lixed preference vector. While effective for static settings, these approaches f'il under shifting priorities. Pareto-based methods, such as Envelope Q-learning and Pareto-condition
policy opti-
mization (Van Moffaert & Nowé 2014 Yang et al. 2019 Basaklar et al.) 2023] Liu et al.] 2025), approximate the Pareto front of optimal policies, enabling post-hoc preference selection. However, they still assume externally specified preferences and lack online adaptation. Recent work explores nonlinear and dynamic scalarization (Mossalam et al.] |2066] Abels et al. (019), meta-learning for preference adaptation, and learning from human feedback (Christiano al. |2017] Ibarz al. [018]] Ramachandran & Amir, 2007; Fu et al. 2018). Nevertheless, few methods explicitly treat pref- erence weights as latent states to be inferred online. Our approach addresses this gap by framing preference adaptation as variational inference problem, enabling agents to maintain a belief over preferences and dynamically reweight objectives in response to environmental changes, aligning with psychological theories of adaptive goal regulation.
3 METHODOLOGY
We ground our study on the assumption of boundedly rational agent, capable of reappraising what matters in response situational changes (Simon 1955) Carver & Scheter 2001 Kruglanski al. 2018 Zhang et al.] 2024). Human decision-making in dynamic, multi-objective settings rarely fol- lows single, fixed plan. Instead, we continuously reappraise goals in light of the currett
3

---

Published as a conference paper at ICLR 2026
Moralb
State embeddinng
History memory
67"
angmaa /
Erergy
encoder
Preference
posterior
ii (org (5xc)
Pareto
0.8
HistoryPuuu
State
agent
Sample
Osservation
candidates
Input
Perception
Value Appraisal
Envelope Operator
Time is running out. to iut line!
Conditioned actor x(a,ls,.6d))
i should keep queuing.
Conditioned critic V(s,,
c I'm almost late. need to cut line.
Environment Execution Action Selection
Figure 2: Two-stage oognitive-inspieee decision framework. History states transformed into latent preferences via Value Appraisal, which in turn guide the Action Selection. The resulting policy drives environment execution, forming a dynamic decision pipeline analogous human ap- praisal-action coupling.
situation decidding not just how to act, but also what matters most right now. This dual process is reflected in cognitive models such appraisal theory in emotion research (Lazarus) 1991 Scherer
1999
Frijda 1993) and dual-process frameworks psychology (Kahneman) [2011] Stanovich al,,
2000
Kahneman 2012), where value assessment and action selection are distinct yet tightly
pled.
In this work, operationalize these principles through two-module computational agent (Fig.
Value Appraisal Module (history preferences): From short history-memory window
of states, recurrent encoder infers a distribution over the latent preference vector wi E A d-1 (a point on the simplex), from which a sample W; is drawn for control.
Action Selection Module (state, preferences action): An actor-critic conditions jointly on the current state and inferred preferences to produce vector value estimates and policy. At decision time, an on-policy envelope operator evaluates K preference samples and selects the W, that maximizes the predicted scalarized value.
Training is regularized by a variational evidence lower bound (ELBO) with simple prior, directional alignment between inferred preferences and vec- tor returns, and a self-consistency term anchoring posterior predictions to the envelope- selected w,, Together, these stabilize preference inference and improve interpretability.
Stability & Alignment:
This division is consistent with cognitive models of dual-process decision -aakion:n-aainn::: a working mem- ory-based appraisal system that updates value priorities, and a controller that acts accordingly.
3.1 VALUE APPRATSAL AS VARIATIONAL. INFERENCE OVER DYNAMIC PREFERENCES
Cognitive & statistical generative assumption. Let w E d denote the (unobserved) prefer- ence weights over objectives at time f. We assume non-stationary latent dynamics w ~ p(w,, wi-1, Ee), driven by unmodeled situational factors & (e.g.. urgency, risk, energy scarcity). In prac- tice, & is not explicitly modeled but absorbed into the stochastic posterior updates. The agent does not observe w/ but receives a state stream 8t- #+i (which may contain observation +t- and self-states such as energy and deadline time) over finite working-memory horizon H 1. Follow- ing the Bayesian brain view (Knill & Pouget 2004 Colombo & Series 2012 Bottemanne,, 2025) and updates them as new evidence arrives. Note that this generative factorization is an internal perceptual model used by the agent to infer preferences; the actual environment transition kernel Peny (8++1 8t, (t,00) in all our experiments is independent of and follows the standard MDP assumption (see Appendix(A)
maintains beliefs p(w: s;-H+1:1)
p( w.) w st-H+1:-11
Latent Representation and Posterior Approximation. Instead of treating preferences as fixed inputs, we approximate them as distributional latent states that capture both epistemic uncertainty and exploratory variation in preference space. Concretely, we introduce an unconstrained latent vector 2, Rd with (z, $t-H+1:) = N diag( (f)),, and w; softmax(z,) This design provides: (i) uncertainty over value preference weight: ambiguous situations produce broader poste- riors, explicitly capturing the agent's uncertainty. (ii) exploration in value preference space: instead
4

---

Published as a conference paper at ICLR 2026
of committing to single trade-off, the agent samples from the posterior, which slightly perturbs wi and encourages trying nearby preferences before acting. A unit Gaussian prior po(z) = /) regularizes the posterior, anchoring values unless evidence strongly suggests change. Detailed im- plementations are provided Appendix C
Learning objective. Since the latent preference state 2₁ cannot be directly observed, resort to variational inference and approximate its posterior by go (zz e2), where G == 8 -+ denotes the recent state history within finite working-memory window. We map 2, preference vector on the simplex via deterministic function W; - fo(=;) (e.g., a linear layer followed by softmax).
This requires specifying how the collected evidence e supports different preference configurations. Following bounded-rationa choice models (Luce et al.) 1959) and the free-energy formulation of Bayesian brain theories (Friston 2010), we treat the scalarized return under a preference vector as selizmann-ratioons likelihood of the evidence:
p(e: 21) exp (BUllw)))) B 0,
where Up(wrie)) ex) = (wi, (e.)) represents a scalar utility and G.(et) E IRd the vector return estimated by the actor-critic from the history Ez.
With an isotropic Gaussian prior po(z,) NN(,, I), Bayes' rule yields the (unnormalized) target posterior
p p= et) x po())) exp (BU((w)))))
(2)
We approximate p" with (z | ex) and optimize qa by minimizing KL l|p"), which is equiva- lent to maximizing the following evidence lower bound (ELBO):
CELBO B Ez, [Ui(w;;
(g. ex) (O..
This objective balances fit to the current evidence (the utility term) against regularization from the prior, yielding stable yet adaptive updates of preference beliefs. A complete derivation is provided in Appendix
3.2 ACTION SELECTION BASED ON PREFERENCE WEIGHTS
Humans rarely commit single immutable weighting of goals. When facing trade-off's (e.g., morality vs. survival), we may entertain several plausible configurations and act according to the one that seems most promising at the moment. Inspired by this, our agent does not rely on fixed preference, but rather evaluates small set of candidates and selects the one that yields the highest predicted utility. To achieve this goal, we employ preference-conditionn policy method, which can adopt policy based on the given preference weights. Specifically, both policy and value functions conditioned preferences: (a(a, 8t, wi), Vo(st, Rd Given preference weight vector, the scalarized value obtained by Vsealar (s,, w,) (w,, Vg(s,, w;))).
Envelope Operator, At each step, K preference candidates are sampled from the appraisal posterior (. 9t-H+1:t), and the one with the highest predicted scalarized value selected as Wi, e.g.,
(w)) (t) (e (o("))
W arg
max (L
(4)
This is exactly the envelope operator (Yang et al. 2019). but executed on-policy during action selec- tion, using a single refference-condiiionne actor-critic. In addition, we denote by w. wrrred preference prediction of the encoder. This worrdd is used for regularization terms below.
Vector-valued GAE. Given vector rewards F. we compute temporal differences for each dimension:
5, Fe + 7 VVo (.)) Vo wr).
(5)
and accumulate vector advantages A, using standard GAE recursion. This ensures temporal credit assignment is handled per-dimension.
Scalarized advantage for PPO. The policy update requires a scalar advantage. We project using the same on-policy we: A. At), A. normalize (At), calculating the usual clipped surrogate
(0) = E min (rx(0)Ar, clip(r,, 1 1
(6)
5

---

Published as a conference paper at ICLR 2026
with r. (0)
no fof la ( Toaalay (s,,)))) ensuring on-policy consistency in both action and preference space.
Dual critic loss. To stabilize learning process, we combine vector-level supervision with scalarized supervision:
Cente - - G12 + g) (Vocalarr (st, (Vocalat(ss,)) (00, G))
where G are vectorized returns from vector-GAE.
3.3 STABILITY AND ALIGNMENT OF PREFERENCES
Preference Alignment. Intuitively, simply maximizing Uz(wr) (wr, G3) risks degenerate so- lutions:
unstable oscillations of w, or opportunistic "gaming" of temporary fluctuations. Humans avoid this by adjusting preferences smoothly and in line with feasible opportunities. Instead of max- imizing utility directly, we introduce two cognitive-inspirg regularizers to stabilize the learning process. In detail, to encourage the predicted preference weights w aligned with true environment dynamics, we apply direction alignment loss:
(w pred G.)
Cdir
E
(8)
prod
lz |Grll2
which is applied only when ||G,]| > 0. This discourages caring about objectives that are unattainable the moment.
Self-consistency. In addition, to ensure the predicted preference matches the envelope-selected one:
Stabb
| pred - well
We aim to conduct posterior instance self-consistens constraint, e.g., if envelope consistently selects mode, the encoder should directly predict reducing policy-preferenc mismatch.
Model Optimization. Together with the ELBO term, the overall training objective of DPI is formu- lated as
L Lpro + Lenie - CELBO + AL + y Lstab:
(10)
where À and y are coefficient. The complete training procedure is summarized in Algorithm |
EXPERIMENTS
In this section, we evaluate our framework along three key aspects: (i) Q1: Effectiveness-DDe
dynamic preference inference improve task performances? (Sec. 4.4) (ii) Q2: Rationality-Doc
the agent adapt its preferences to environmental changes? (Sec. 4.5 (iii) Q3: Interpretability
the inferred preferences interpretable and semantically aligned? (Sec. 4.6
4.1 EXPERIMENT ENVIRONMENTS
(i) Queue. As is illustrated in Fig. 6a Queue is a simple but illustrative toy problem, where an agent must decide whether wait line or ahead obtain food, balancing two conflicting objectives: energy (survival) and morality (fairness). No fixed weighting suffices: always waiting risks starvation, always cutting erodes morality. Success requires dynamically rebalancing prefer- ences according to context. (ii) Maze. We mainly conduct our experiments on Maze environment is shown in Fig. (6b). which introduces a 2D navigation task with multiple objectives in pixel space: reaching the goal, meeting deadlines, avoiding hazards, and conserving energy. Random hazard storms and shifting costs introduce non-stationarity, making any static value composition fail. Dynamic value appraisal is essential for progress under varying conditions. (iii) Continuous control. We have modified a multi-objective variants of MuJoCo with widely used baselines. See Appendix D.3 for details. Despite differences in modality, these three environments share struc- tural property: without dynamic preference adjustment, it is diffucult for the agent to complete the task.

---

Published as a conference paper at ICLR 2026
4.2 BASELINES
We compare DPI against a diverse set of representative baselines: (1) Fixed-Preference MORL (FIXED): strong baseline where the preference vector is fixed emphasize task-completion objectives (e.g., progress and deadline) (Mossalam et al.] (016), simulating conventional single- objective RI. methods that optimize for a scalar reward with secondary penalties treated as regu- larizers. (2) Randomized Switching (RS): w is randomly resampled at runtime, testing whether naive stochastic preference variation can mimic adaptivity.
(3) Heuristic MORL (HEURISTIC); hand-crafted preference schedules are applied for different event types and then converted into static w settings, serving as a strong rule-based baseline. (4) Rule-based Envelope Q-learning (ENVE- policiess conditioned externally supplied w as in a rule-basee envelope Q-learning (Yang LOPE): policies
et al. 2019) with event-dependent preferences (5) Random Policy (RANDOM) a uniformly ran- dom agent that samples actions independently at each step, providing lower bound on performance. This serves as a sanity check to confirm that learned policies achieve meaningful improvement over chance-level behavior. (6) Dense Oracle: receives access the same event signals and addition- ally uses time energy information to continuously update preference weights at each timestep (see Appendix D.2,, which provide a practical upper bound under our settings.
4.3 EVALUATION METRICS
We report three complementary metrics to comprehensively evaluate overall performance, success rate and adaptation under distribution shifts, respectively, (i) Mean Episodic Return (MER). We compute the total return across N episode: R = N EN Ei=rt r( where is the step length
and ri (f) the scalar reward at step t. (ii) Success Rate (SR). We measure the fraction of episodes achieving task success: SR = x/2 >m
i== 1( = 1}. where & E {0, 1] a task-specific completion flag (e.g., successfully obtaining food in Queue reaching the goal in Maze). (iii) Post-Shift Performance (PS @ K). For an environment event occurring at the change point L", we measure the
average return in the first K steps following the change: PS@ K N Em -1= (1) which captures adaptation ability after environment contextual shifts. All reported results are averaged over N 200 evaluation episodes for each of 10 random seeds. We report all metrics with the mean and 95% confidence interval (CI) across N episodes: CT(.) = 1.96 a()
VN
4.4 EFFECTIVENESS - DOES DYNAMIC PREFERENCE INFERENCE IMPROVE TASK PERFORMANCE?
RANDOM
Quene
Maze
FIXED RANDOM CHOICE
MER
SR (%)
MER
SR (%)
RANDOM
0.55
17.25
10:12
223..55 2: 10.61
0.00 1: 0.01
10-1-
HEURISTIC
FIXED
2.55
10.05
4 3.24
16.15 1.62
1.12 1: 0.06
psak
-10-
ENVELOPE
-..0
11.43
4.01
23.86-+4.44
0:01
HEURISTIC
-..00
10.05
1: 8.12
-3.65 0.18
0.00 0.00
ENVELOPE
-..5
25.10 1: 6.01
10:36 0.18
0.01 1: 0.00
-10
w/ Q-Karnnn
2.30
29.09 +: 5.33
27. 85 1.25
42.94 3.72 3.72
PPO
10.34
39.95 1: 2.75
30.16
1.22
58:04 1: 0.10
2
8
Steps after event change (t=0
Table 1: Mean episodic return (MER) and Success rate (SR) across all baselines.
Figure 3: Post-shift performance
(PS@ K) on Queue environment.
Toy Environment Result. We begin with a symbolic Queue environment, a minimal toy setting where the agent must balance survival (energy) against fairness (morality) dynamically. Table shows that our DPI agent substantially improves MER and achieves 14.85% higher SR than the strongest baseline (ENVELOPE). This demonstrates that dynamically inferred preferences not only improve cumulative performance but also enable the agent to successfully complete the task, whereas fixed or heuristic preferences frequently fail. We additionally analyze the pre- and post- event Pareto fronts over efficiency and fairness, showing that any fixed scalarization necessarily sacrifices performance in at least one regime, whereas DPI adapts its inferred preferences remain close to the dynamic front (A (Appendix(D..) D.I.. Combined with the MER- and SR-PPO (Appendix[D.) this suggests that DPI's gains are not simply due to optimizing particular scalar metric.
7

---

Published as a conference paper at ICLR 2026
00.03
-0.15
10.16
[ANDOM CHOME
-0.42
-0.43
HEURISTIC
A A
-0.57
ENVELOPE
-0.58
0.20
0.23
DP (Curs)
Envirommntt steps
(a) Deadline shock
(b) Hazard surge
(c) Energy drought
(d) Alignment analysis.
Figure 4: Event-aligned trajectories in Maze environment. After each event. DPI updates its preferences and modifies its behavior in contextually appropriate way: (a) prioritizes shorter routes under deadl ine shock. (b) exhibits increased avoidance under hazard surge. (c) prefers waiting and selecting minimal-cost routes under energy drought. Arrows indicate agent mo- tion; shaded regions mark environmental hazards or costs. (d) Alignment between inferred prefer- ences and reward vectors. DPI maintains positive cosine similarity and sharply increases alignment after event onsets, whereas baselines remain near zero or negative, indicating that only DPI learns a value representation that tracks task semantics.
Main result. Table I reports mean episodic return (MER) and success (SR) in the Maze envi- ronment. RS fails catastrophically, yielding extremely low and highly variable returns. HEURISTIC also performs poorly in this non-stationary setting, indicating that simple hand-designed preference schedules cannot handle the combinatorial diversity of event configurations. ENVELOPE, which given deeendee preferences but does not infer them, achieves substantially higher MER, highlighting the importance of conditioning w at deployment. Our Dynamic Preference Inference (DPI) agent achieves the highest overall MER, outperforming ENVELOPE by + 191.1 11,, confirming that online inference over preferences enables more robust long-term behavior under distributional shifts. Classical baselines such as FIXED, HEURISTIC, and ENVELOPE all achieve near-7ero SR, when FIXED attains relatively high MER, showing that they cannot complete the task reliably under all event configurations. DPI attains the highest SR (59.0%), significantly outperforming all ablations and demonstrating its ability to consistently adapt and complete the task under dynamically changing conditions. For completeness, Appendix [D.2] further compares DPI against scalarized RL. baselines that directly optimize MER SR (MER-PPO and SR-PPO), well as a Dense Oracle with privileged access to event signals; DPI still yields superior post-shift performance and success rate these settings. We further confirm these trends in the modified multi-objective continuous- control setting, where DPI again matches or surpasses strong fixed- seed-preference and oracle baselines (see Appendix D.3).
4.5 RATIONALITY-DON DOES THE AGENT ADAPT ITS PREFERENCES TO ENVIRONMENTAL CHANGES?
To verify that DPI performs meaningful preference adaptation rather than merely exploiting reward structure, we report Post-Shift Performance (PS@K) K) for K 1,...,& 8 in Fig. When events are triggered, both HEURISTIC and the ENVELOPE baselines exhibit persistently low post-shift performance, indicating their inability adapt to non-stationary dynamics. Interestingly. FIXED attains moderately good PS @ values by greedily pursuing the highest scalarized reward under statie weighting. However, as confirmed Table this strategy fails to reliably complete the task, demonstrating that no single fixed preference vector is sufficient to handle all event configurations. In contrast, DPI shows rapid recovery after each change point, maintaining high post-shift reward and significantly outperforming all baselines. Together with its 59.04% success rate, these results indicate that DPI is not merely memorizing action sequences, but aligns its internal value preferences with changing environmental demands.
4.6 INTERPRETABILITY-ARE THE INFERRED PREFERENCES INTERPRETABLE AND SEMANTICALLY ALIGNED?
Qualitative evidence: event > preference + behavior. Fig. 4b 4c shows three representative Maze episodes with distinct event types: deadline shock, hazard surge, and energy drought. After each change-point, DPI reweights its preferences in a context-consisentt manner and correspondingly switches its
when deadlines shrink, rerouting when
8

---

Published as a conference paper at ICLR 2026
0.35
1.0
Mutt Re2 avers 88
0.30
30-
0.8
IDD STa
28-
with elbo
episddc
0.66
Q-learning
0.20
26
0.4
0.15
24-
0.10
22-
0.05
Steps after (t=0 event)
0.00
Method
5 11 17
History window
(a) PS@K for each ablated variant.
(b) Multistep average reward. (c) Different history window size.
Figure 5: Ablation study results. (a) Post-Shift Performance (PS@K) K) curves over the first K 8 steps after each event. (b) Multi-step average @ summarizing short-term recovery into a single metric for each method. (c) Mean episodic return (MER) function of history window size H. Across all plots, our full DPI agent consistently outperforms ablations, confirming the necessity of KL regularization, directional alignment, and self-consistency, and showing that performance is robust to H beyond a small temporal context.
hazards intensify, and conserving movement under energy scarcity. This event ceat--ppeeeeeee--acttine chain indicates that DPI is dynamically revising what matters now instead of replaying fixed plan.
Quantitative alignment with reward structure. To verify that the inferred preferences track task objectives, we compute the cosine similarity between the inferred preference W, and the instanta- neous reward vector Fr: Align(!) = | lalf
As shown in Fig- 4d DPI maintains consistently positive alignment and exhibits sharp but smooth rises following event onsets, suggesting that its internal value representation reorients toward the most relevant objectives. In contrast, random, heuristic, and envelope baselines remain negative, showing no systematic relation to the reward structure. Together, these results demonstrate that DPI not only adapts its preferences to recover performance, but does in semantically interpretable way that reflects the true task demands.
4.7 ABLATION STUDY
Component-wise Ablation. To isolate the contribution of each design component in DPI, we com- pare against three variants: (i) w/o KI,, removing the KI. prior term from the ELBO objective; (ïi) w/o dir, removing the directional alignment constraint between preferences and return gradi- ents; (iii) w/o sta, removing the self-consistency regularization that anchors posterior predictions envelope-selected preferences. We also replace the weference-condiiionn actor-cut coor-critic with simple Q-learning baseline (w/ Q-learning) to test whether a purely value-basee based method suffices.
Fig. Sa reportss Post-Shift Performance (PS@ (K)@K) over K 1,...,& steps after each event, showing how quickly each variant recovers reward after distributional shifts. To provide a single-number summary of short-term recovery, also report the multistep average PS @ K (Fig. 5b), which aver- ages performance across K = 1. 8. Removing any of the three components leads to measurable performance drop, confirming that KL regularization, directional alignment, and posterior anchoring are all critical for stabilizing preference inference and improving adaptation. Moreover, replacing PPO with Q-Jearning significantly degrades recovery ability, highlighting the importance of using a actor-critic critic for fast on-policy adjustment.
History Window Size. We additionally study the effect of the history encoder's receptive field size H, varying the number of past observations fed into the preference encoder. repport the mean episodic return (MER) as a function of H (mean 95% CT). Performance degrades significantly for very small history sizes (c.g., H 1), indicating that the model needs sufficient temporal context to infer preferences. However, performance plateaus for moderately large windows (H 2 9), suggesting that approach is robust the exact choice of H. In all experiments, choose H as a good trade-off between performance and computational cost.
Hyperparameter ablations and training stability. See Appendix D.4 and Appendix D.6 for ab- lations and learning curves, which show that DPI robust to the main hyperparameters and trains stably without early collapse.

---

Published as a conference paper at ICLR 2026
5 CONCLUSION AND FUTURE DIRECTIONS
In this work, we introduced a cognitively inspired framework that abstracts value preference adjust- ment in dynamic multi-objeectve environments. We formalized setting in which preference weights are latent, context-dependent variables that must be inferred online, and proposed DPI, which com- bines a variational preference inference module with a preference-conditioned policy. Experiments in Queueing, Maze, and multi-objective continuous-conit aware preference adaptation and improves performance under event-driven distribution shifts. A key limitation is that our environments remain controlled and simulated rather than open-ended or real-world. Future work will focus on scaling DPI more realistic 3D) embodied and multi-agent settings, and on developing more expressive inference mechanisms for long-horizon and socially coupled preference dynamics.
tasks show that DPI enables context-
REFERENCES
Axel Abels, Diederik Roijers, Tom Lenaerts, Ann Nowé, and Denis Steckelmacher. Dynamic weights in multi-objective deep reinforcement learning. In International conference on machine learning, pp. 11-20. PMLR, 2019.
Mridul Agarwal, Vaneet Aggarwal, and Tian Lan. Multi-objectivv reinforcement learning with non- linear scalarization. In Proceedings of the 21st International Conference Autonomous Agents and Multiagent Systems, pp. 9-17, 2022.
Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Mané. Con- crete problems in ai safety. arXiv preprint arXiv: 06565, 2016.
Toygun Basaklar, Suat Gumussoy, and Umit Ogras. Pd-morl: Preference-driven multi-objective reinforcement learning algorithm. In The Eleventh International Conference on Learning Repre- sentations, 2023.
Richard Bellman and Robert Kalaba. Dynamic programming and statistical communication theory. Proceedings of the National Academy of Sciences, 43(8):749-751, 1957.
Hugo Bottemanne. Bayesian brain theory: Computational neuroscience of belief. Neuroscience, 566: 198-204, 2025.
Jerome R Busemeyer and James T Townsend. Decision field theory: a dynamic-cognitive approach to decision making in uncertain environment. Psychological review, 100(3):432, 1993.
Charles S Carver and Michael F Scheier. On the self-regulation of behavior. cambridge university press, 2001.
Charles S Carver and Michael F Scheier. Self-regulation of action and affect. Handbook of self- regulation: Research, theory, and applications, pp. 13-39, 2004.
Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Advances in neural information processing sys- tems, 30, 2017.
Matteo Colombo and Peggy Series. Bayes in the brain-or on bayesian modelling in neuroscience. The British journal for the philosophy of science, 2012.
Edward L Deci and Richard M Ryan. The general causality orientations scale: Self-determination in personality. Journal of research in personality, 1992):109-134, 1985.
Edward Deci and Richard M Ryan. Self-determinttion theory. Handbook of theories of social
psychology, 1(20):416-43. 2012.
Gabriel Dulac-Armold Nir Levine. Daniel J Mankowitz, Jerry Li, Cosmin Paduraru, Sven Gowal, and Todd Hester. Challenges of real-world reinforcement learning: definitions, benchmarks and analysis. Machine Learning, (49922411-268 2021.
Nico H Frijda. The place of appraisal in emotion. Cognition & Emotion, 7(3-)):35-3377, 1993.
10

---

Published as a conference paper at ICLR 2026
Karl Friston. The free-energy principle: unified brain theory? Nature reviews neuroscience, 11 (2):127-138, 2010.
Justin Fu, Katie Luo, and Sergey Levine. Learning robust rewards with adversarial inverse rein- forcement learning. In International Conference Learning Representations, 2018.
Borja Ibarz, Jan Leike, Tobias Pohlen, Geoffrey Irving, Shane Legg, and Dario Amodei. Reward learning from human preferences and demonstrations in atari. Advances in neural information processing systems, 31, 2018.
Daniel Kahneman. Thinking, fast and slow. macmillan, 2011.
Daniel Kahneman. 2 minds: How fast and slow thinking shape perception and choice [excerpt]. Scientific American, 15, 2012.
Daniel Kahneman and Amos Tversky. Prospect theory: An analysis of decision under risk. In Handbook of the fundamentals of financial decision making: Part I, pp. 99-127. World Scientific, 2013.
David C Knill and Alexandre Pouget. The bayesian brain: the role of uncertainty neural coding and computation. TRENDS in Neurosciences, 27(12):712-719, 2004.
Arie W Kruglanski, James Y Shah, Ayelet Fishbach, Ron Friedman, Woo Young Chun, and David Sleeth-Keppler. A theory of goal systems. In The motivated mind, pp. 207-250. Routledge, 2018.
Richard S Lazarus. Emotion and adaptation. Oxford University Press, 1991.
Sarah Lichtenstein and Paul Slovic. The construction of preference. Cambridge University Press, 2006.
Ruohong Liu, Yuxin Pan, Linjie Xu, Lei Song, Pengcheng You, Yize Chen, and Jiang Bian. Ef- ficient discovery of pareto front for multi-objective reinforcement learning. In The Thirteenth International Conference on Learning Representations, 2025.
Duncan Luce et al. Individual choice behavior, volume 4. Wiley New York, 1959.
Hossam Mossalam, Yannis M Assael, Diederik M Roijers, and Shimon Whiteson. Multi-objective deep reinforcement learning. arXiv preprint arXiv: 111.02160 2016.
John W Payne, James Bettman, and Eric J Johnson. The adaptive decision maker. Cambridge university press, 1993.
Deepak Ramachandran and Eyal Amir. Bayesian inverse reinforcement learning. In IJCAI, vol- ume 7, pp. 2586 2016,,259,,222 2007.
Diederik M Roijers, Peter Vamplew, Shimon Whiteson, and Richard Dazeley. A survey of multi- objective sequential decision-making. Journal of Artificial Intelligence Research, 48:67-113, 2013.
Stuart Jonathan Russell and Peter Norvig. Artificial intelligence: A modern approach: intelligent agent book]. Prentice hall, 1995.
KR Scherer. Appraisal theory. Handbook of cognition and emotion, pp. 637-663, 1999.
Amitai Shenhay, Mathhw M Botvinick, and Jonathan D Cohen. The expected value of control: an integrative theory of anterior cingulate cortex function. Neuron, 79(2):117-240, 2013.
Herbert Simon. A behavioral model of rational choice. The quarterly journal of economics, pp- 99-118, 118, 1955.
Paul Slovic. The construction of preference. American psychologist, 50(5):364, 1995.
Keith E Stanovich, Richard F West. and JE Alder. Individual differences in reasoning: Implications for the rationality debate?-open peer commentary-three fallacies. Behavioral and Brain Sciences, 23(5):665-666) 2000.

---

Published as a conference paper at ICLR 2026
Peter Vamplew, Richard Dazeley, Adam Berry, Rustam Issabekov, and Evan Dekker. Empirical evaluation methods for multiobjective reinforcement learning algorithms. Machine learning, 84 (1):51-80, 2011.
Kristof Van Moffaert and Ann Nowé. Multi-objective reinforcement learning using sets of pareto dominating policies. The Journal of Machine Learning Research, 15(1):3483-3512, 2014.
Carsten Wrosch, Michael F Scheier, Gregory E Miller, Richard Schulz, and Charles S Carver. Adap- tive self-regulation of unattainable goals: Goal disengagement, goal reengagement, and subjective well-being. Personality and social psychology bulletin, 29(12):194--1588 2003,
Runzhe Yang. Xingyuan Sun, and Karthik Narasimhan. A generalized algorithm for multi-objective reinforcement learning and policy adaptation. Advances in neural information processing systems, 32, 2019.
Stelios H Zanakis, Anthony Solomon, Nicole Wishart, and Sandipa Dublish. Multi-attribute decision making: A simulation comparison of select methods. European journal of operational research, 107(3):507-529, 1998.
Zhenliang Zhang, Zeyu Zhang, Ziyuan Jiao, Yao Su, Hangxin Liu, Wei Wang, and Song-Chun Zhu. On the emergence of symmetrical reality. In Proceedings of the IEEE Conference Virtual Reality and 3D User Interfaces (VR), pp. 639-649. IEEE, 2024.
12

---

Published as a conference paper at ICLR 2026
A DERIVATION OF THE PREFERENCE OPTIMIZATION OBJECTIVE
Problem definition. At each time step f, the agent maintains a latent preference logit vector 2; E Rd which mapped the probability simplex by
W/ = fo(2)) == softmax(z)) E Ad-1
(11)
Let the evidence at time be the recent history E = 8 14 Given a d-dimensional vector return Ge(e)) E It (estimated by the actor-critic from e₁; see Sec. 3.2 we define the scalar utility under preference Wa as
Ul(weee)) = (wr, Ge(e)))
(12)
Our goal is to infer 2t from the evidence ét.
Assumption (prior and noltzmann-rational evidence). Prior: Po(zz) (0,,,) I): (ii) Evi-
dence model: following quantal response and free-energy formulations, we posit the umnormalized likelihood
p(gp zz) (U, C(ex). B 0,
(13)
where C(es) does not depend on 21 (hence its gradient vanishes w.r.t. Φ) (unnormalized) target posterior is
The corresponding
p' ex) x po(z)) exp (BUr(wer; ex))
(14)
Objective: maximize marginal evidence.
The marginal evidence
p(et)= = / po(zz) plee | 2,) dze,
(15)
which is intractable because w, = fo(z,) (0z) nonlinear.
Variational family. We approximate the posterior by go(Zz ex) Nu,,, diag(o))) where (P, log o.) are the outputs of the encoder given éz.
KL expansion.
By Bayes' rule,
p² (z. ee) po((z) pler 2t)
(16)
p(e))
Thus
(L(go(z e p² (z, le))) Eyg loggo(zz (ea) log)o(zz) logple: zn) + logp(es).
(17)
Rearranging yields
p(ex( Egg [lggpee 21)] (T (e)) [ee +KT. le)) (1x l (18)
((())
ELBO. Therefore,
logp(e:) 2 L.(0) = (x [logp(e [x)]] (z (t) || I)).
(19)
Substituting the evidence model and dropping the C(ex) term, we obtain
Lx(0) = Beo(=sle [(fo)))))))(((()))))))))) [[f((z)) - (L ex) (1,)
(20)
Maximizing (d) is equivalent minimizing KL / || p²) and tightens the lower bound on log l(gp(e)). Over window t I,..., T, the overall prefereence-iffernnee objective is
T
T
max Σ C.( = Σ { (esle))) [(f((z) G.(e)))] - (L(gs(z, - (0), (21)
t=1 [=1
13

---

Published as a conference paper at ICLR 2026
Agent
(a) Queue environment
(b) Maze environment
Figure 6: Two environments used in our experiments. (a) Queue provides simple but illustrative toy problem. (b) Maze mainly used in our experiments.
B ENVIRONMENT DETAILS
In this section, we will introduce the detailed information of the designed environment in our exper- iments (as shown in Fig.
B. QUEUE
We design symbolic queueing environment to capture moral pragmatic trade-offs such as whether to wait patiently or cut in line under time pressure. The agent waits in a queue to be served and must decide between two discrete actions each step: WAIT (stay place) or CUT (jump k positions forward, capped at the front).
State space. The agent's state time is compact feature vector
8t = [pos,, queue len,, energy,- deadlines, recent cut,, service rate,),
where pos, is the agent's current position in the queue (normalized to [0, ID, queue_len, is the remaining queue length, energy, is remaining energy, deadling ine, remaining time budget, recent.cut, (0, 1} indicates whether the agent just cut the line, and service-rate, the current service rate (normalized).
Action space. The agent chooses from A [WAIT, CUT}. Cutting consumes additional energy and may incur a fairesss penalty.
Reward vector. Each step produces d = 5-dimensional reward vector:
Ir
progress time fairness .nergyy deadline
(22)
where r proggrsss +1 when advancing in the queue or being served, rlime -1 per waiting step. fairness -y when cutting ahead of others, r energy ー proportional energy expenditure, and deadline = +3 if served before the deadline.
Dynamic events. To induce contextual shifts, we introduce event triggers such
(i) Arrival burst: : a sudden inflow of new agents increases the queue length;
(ii) Service slowdown: the service rate drops, increasing expected waiting time;
(iii) Energy shock: the agent's remaining energy is reduced, raising the urgency of being served soon. These events create moral-pragmatic conflicts: a patient strategy may lead starvation energy runs out, while aggressive cutting risks collapsing the morality score. Thus, no fixed trade- off between fairness and survival can succeed universally, making this environment ideal for testing dynamic preference reweighting.
Equivalently, p(e: zz) x exp (3Ur(ws;e.)) e.)) with partition term that is independent of z;.
14

---

Published as a conference paper at ICLR 2026
Table 2: Detailed specifications of environments. Each domain introduces dynamic events that invalidate fixed preferences, requiring agents to adapt their value weights.
Environment State Space
Action Space
Reward Vector
Dynamic Events
progress, time penalty, fairness
Symbolic: pos, queue.len, energy, deadline
Arrival burst; Service slowdown; Energy shock
Queue
Wait Cut
penalty, energy penalty, deadline
penalty
progress, time senalty,haaan, penalty, energy penalty, deadline
Pixel:
Up Down
Deadline shock; Hazard surge; Energy drought
observation,
Maze
Left/ Right
time, energy
penalty
B.2
MAZE
We design Maze navigation environment with multiple interacting objectives to evaluate adaptive preference inference under dynamic constraints. The agent starts in the bottom-left corner and must reach a goal location the top-right corner. Each episode terminates upon reaching the goal or exceeding maximum step budget T' 200. At each step the agent observes its current (x, y) position, a global timer normalized to j0, 1], remaining energy, and a binary hazard map indicating nearby dangerous cells. The state encoded as an image $; E # x3,, where the three channels are [observation, time ratio, energy ratio], respectively.
Action space. The agent chooses from A {UP, DOWN, LEFT, RIGHT), moving one cell per step unless blocked by a wall.
Reward vector. Each transition produces d 5-dimensional reward vector:
F (r. r rt
prog time haxard energy ydeadline]
(23)
where y = +1 for moving closer to the goal, prtie -1 as a per-step time penalty, bazard - if stepping into a hazardous cell, and r sumption, pdeadlnee = +33 if reached before the deadline. No fixed scalarization is applied; the agent must learn to reweight these objectives.
ehergy ー proportional energy con-
Dynamic events. To induce non-stationarity, we introduce three event types at random steps:
(i Deadline shock: remaining time budget is suddenly shortened by 30%. increasing the ur- gency of reaching the goal.
(ii) Hazard surge: new region of static hazards appears, increasing collision risk along the shortest path.
(ii) Energy drought: the agent's energy consumption per step doubles, requiring more con- servative movement. Each episode may contain multiple events in sequence, forcing the agent to continuously reappraise what matters most (e.g., prioritize speed when the deadline is tight, or safety when hazards dominate).
Despite their surface differences (as Table 2) both domains share the same structural property: without dynamic preference adjustment, the agent rarely succeeds. In our implementations, we empirically observe that no single fixed preference vector over the exposed reward compo- nents achieves high MER and SR across all event configurations. This unified design highlights the necessity of our proposed framework, which equips agents with cognitive- like value reappraisal remain effective under shifting constraints.
15

---

Published as a conference paper at ICLR 2026
C IMPLEMENTATION DETAILS
C.1 NETWORK ARCHITECTURE
State Encoder
Value Appraisal Module
FC 128x128
Conv 64edad 64x3x3
FC 128x128
GRU 128
FC 128x128 128:5
Observation history Oz RUHITHIN
Figure 7: The architecture of our Value Appraisal Module, where the numbers represent channel numbers and kernel size
Value Appraisal Module. The network structure of the proposed value appraisal module used in Maze experiments is illustrated in Fig. / detail, the state encoder consists of two convolutional layers and fully connected layer for encoding observations. Then GRU network is employed to aggregate historical information, which is connected with two two-layer fully connected network to model the mean and the standard deviation. Each convolutional layer is activated by ReLU, and each fully connected layer is activated by leaky ReLU. particular, we use two fully connected layers state encoder for Queue.
State Encoder
Action Selection Module
Actor =(arls...))
Conv
FC 128x128
32x8x8
64x4x4
64x3x3
Critic v(ac.w")
V2
V3
V4
Observation 0, gR
Figure 8: The architecture of our Action Selection Module, where the numbers represent channel numbers and kernel size.
Action Selection Module. We employ a value preference conditional actor-critic our action selection module, as shown in Fig. 8 Specifically, the state encoder is the same the value appraisal module. The actor and critic are both consists of two fully connected layers activated by leaky ReLU.
C.2 OPTIMIZATION
The algorithm process for the DPI is shown in Algorithm I
The encoder fo outputs (H, log σt) for diagonal Gaussian (0(2) St st-H41:) N(u, (o)g(o?)) over the unconstrained latent 2;, and preferences are obtained via w(z)) = softmax(z:). We esti- mate the expectation in equation by the reparameterization trick 2, p4 + t WE,E ~ (N 7), using 1-K Monte Carlo samples per update. The temperature 3 controls evidence sensitivity (larger gives sharper posteriors. Without loss of generality, we set 3 = 1); we keep 3 fixed unless stated otherwise. Vector targets G, are computed per dimension from the critic (see Sec. 3.2 and are treated constants w.r.t. during preference updates.
16

---

Published as a conference paper at ICLR 2026
Algorithm Dynamic Preference Inference (DPI) with On-Policy Envelope Actor-Critic
Input:
Policy parameters 0, preference encoder ø Repeat for each training iteration:
Collect trajectories:
For t ....., T:
Encode recent history $1- via GRU fa obtain mean Mr, log-variance log o
(z 8t-H+11) (W (iag(())))
w softmax(2, (.)
(k) and select W. arg max (w) (k) Vo(se, w(())) Sample K candidates Zi
Sample action a, Te St, we) and step the environment. Optimize:
Compute vector returns: G. by multi-objeetivve GAE.
Compute scalar advantage: At = (w:, At).
Compute actor-critic loss: for Lppo Leritic-
Update 9 by minimizing frz.
Compute elbo CELBO:
CELBO = -Ezi~gs [softmax(zz), G.)] - KL (LL st- $t--+1::)|WN
(wotte G.)
Directional alignment regularization: Lair -1-
1i 1la 1|Grl
Self-consistency regularization: Lubb = prod
Update é by minimizing Lpeter LELBO + ALdie + +-su.
C.3 COMPUTE RESOURCES AND REPRODUCIBILITY
Compute Resources. All experiments were conducted on single workstation equipped with an NVIDIA RTX 4090 GPU (24GB VRAM) and an Intel Core i9- 14900K CPU (32 cores, 64GB RAM). This setup allows 6-8 experiments to be run in parallel without resource contention. Train- ing a single DPI agent on Maze for x 10 environment steps takes approximately 15 minutes, and completing all reported experiments across 10 random seeds requires 6 hours in total. Our imple- mentation is based on PyTorch 2.4.1 with CUDA 11.8, and uses Gymnasium 1.0.0 for environment single GPU; no distributed training is required.
simulation. All code is optimized
Reproducibility. We fix all random seeds for NumPy, PyTorch, and Gymnasium to ensure repro- ducibility. Hyperparameters are summarized in Table 3 We report results averaged over 10 inde- pendent seeds and present 95% confidence intervals (CI) to account for stochasticity. For ablation studies and sensitivity analyses (e.g.. history window size H), we sweep parameters in a controlled range and report mean # CI to ensure robustness.
D EXTRA EXPERIMENTS
D.I PARETO ANALYSIS OF PRE- AND EVEN- REGIMES IN QUEUE
To illustrate that the Pareto-optimal trade-off's before and after an event are structurally different, approximate Pareto fronts in the Queue environment under the pre-event (normal arrival and service rates) and post-event (arrival burst + service slowdown energy shock) regimes.
For this analysis, we fix family of linear scalarizations over the vector reward,
U(w e) (w, G(e)),
(24)
and sweep w over grid on the simplex. For each w, we train policy in the pre-event regime and in the post-event regime, and estimate the corresponding vector returns (w) and G Gpos(W)) (w) from rollouts. In Fig. we project these vectors onto 2D objective space: (i) progress (higher is better)
17

---

Published as a conference paper at ICLR 2026
Table Hyperparameters used all experiments, Values are shared across Maze and Queue unless otherwise noted.
Hyperparameter
Value
Learning rate
2.5 x 10
Batch size
1024 transitions per update
Discount factor
0.99
GAE parameter
0.95
PPO clip ratio (c)
0.2
Entropy cocfficient
0.01
Value loss coefficient (E)
0.5
KL regularization weight (cra)
0.1
Direction alignment weight (A)
0.1
Self consistency weight (?)
0.01
Optimizer
Adam
Number of epochs per update
1
History window size (H)
3 (unless otherwise varied in ablation)
Preference samples per step (K)
8
Training steps per run
1.5 x 102 environment steps
Number of seeds
10 (results reported mean 95% CI)
1.00
event samples
-2.00
Post-event semples
Pre-event Pareto front
Poreto front
-2.25
000
0.75
-7.0
argmon (d-11
deadline
-2.55
0.50
-3.00
-3.25
0.25
-3.50
-3.75
0.00
Figure 9: Queue environment: Pareto fronts before and after the event. We approximate Pareto fronts in 2D objective space (progress vs. deadline-related penalty) by training policies with fixed linear scalarizations w over the vector reward. Each point is a policy; circles and triangles denote pre- and post-event regimes, respectively, and colors encode the dominant component of The blue curve shows the pre-event Pareto front, which lies predominantly in the upper-left region (smaller deadline penalty with moderate progress). while the orange curve shows the post-event Pareto front, which shifts towards the lower-right (higher progress at the cost of larger deadline penalties). The scalarizations that are near-optimal the pre-event regime induce strongly suboptimal trade-off's in the post-event regime, and vice versa, indicating that the Pareto-optimal preference configurations before and after the event are largely disjoint. This supports our claim that no single fixed over the exposed reward vector is robust across all event configurations.
the x-axii, and (ii) deadline-related penalty on the y-axis (higher the plot corresponds to smaller penalty more deadline slack).
Each marker in Fig. corresponds to policy trained with fixed scalarization circles denote pre-event policies, triangles denote post-event policies. From these samples we extract the non- dominated points in each regime, which plot the pre-event and post-event Pareto fronts.
18

---

Published as a conference paper at ICLR 2026
D.2 RL BASELINES OPTIMIZING STATIC METRICS AND DENSE ORACLE BASELINE RESULTS
We add three extra baselines in which we tune the scalarization weights separately for MER and SR AMER-PPO and SR-PPO).
which are:
MER-PPO: PPO with a metric-specific scalar reward P (w FT), where the evalu- ation weight wexl given the agent (oracle static scalarization).
SR-PPO: PPO with a sparse success reward (1 on success, otherwise).
Dense Oracle: based on PPO that receives access to the same event signals and additionally uses time energy information continuously update preference weights at each timestep (e.g., gradually increasing deadline weight as the deadline approaches, increasing hazard weight in proportion hazard intensity).
We tuned two baselines carefully, and report their performance:
Table 4: Comparison results on Queue and Maze.
Queue
Maze
Method
MER
SR (%)
MER
SR (%)
MER-PPO
15.01=0046
0.98 0.05
85.55+0.12
0.05 + 0.01
SR-PPO
-5.64 + 5.22
46.91 + 0.00
-15.28 = 0.96
61.13 + 0.05
Dense Oracle
14.41 + 5.18
49.27 + 15.40
40.33 + 3.73
62.92 0.02
DPI-PPO
10.34 = 0.02
39.95 2.75
30.16 = 1.22
59.04 + 0.01
Note that, for MER-PPO and SR-PPO, while these baselines can do well on individual metrics in stationary settings, they struggle to maintain high performance across all metrics under non- stationary event sequences, whereas DPI maintains consistently good MER and SR. In addition, Dense-Oracle baseline serve strong upper bound.
Empirically, PPO-MER achieves competitive mean episodic return, but still suffers from low success rate, confirming that single fixed scalarization can hardly cope with all event configurations. Our DPI agent maintains comparable MER while substantially improving SR, and additionally provides interpretable, event-aligned preference trajectories (as shown in Fig. 4. This supports our claim that explicit dynamic preference inference offers benefits beyond standard single-objective RL. In addition, designing Markovian reward that directly optimizes P would require explicit access to change points and the evaluation horizon K, which we intentionally do not assume. We therefore treat PS @ K as an evaluation-only metric and focus on MER/SR for single-objective RL. baselines.
D.3 CONTINUOUS CONTROL ENVIRONMENT EXPERIMENTS
To demonstrate the scalability of the proposed method, we modified and implemented a continuous environment based on multi-objective mujoco. The experiment result reported in Table
We further construct a continuous-contro benchmark by extending the multi-objective HalfCheetah environment from mo gymnas i um (mo (mo-half wellffcheeah-ye with dynamic events and resource constraints. The agent controls a planar cheetah to run forward while trading off forward progress, control effort, and energy consumption under a stochastic schedule of events. Compared the grid- based Maze in the main text, this environment probes preference inference in a higher-dimenssonal, continuous state-action space.
State and observation. Let o, E d denote the standard MuJoCo observation of HalfCheetah at time & (joint positions/velocities, torso state, etc.). To provide short-term temporal context, expose to the agent a stacked history of length H:
8t [ot Hi-1+1,--- ... oz] E Hxd.
(25)
19

---

Published as a conference paper at ICLR 2026
with H 8 in all experiments. Internally, the environment also maintains remaining deadline Dr E N and an energy budget E, R.,, which govern termination but are not directly observed by the agent (making the task partially observable with respect to constraints). We set maximum horizon max = 200, an initial deadline Do = 200, and an initial energy budget Eg = 100.
Action space. The action a, E Rd is continuous torque vector applied the actuated joints, identical to the standard HalfCheetah control interface. We use the same box-constrained action space as mo
Reward vector and constraints. At each step the underlying mo mo-hachhhettthe environ- ment returns a vector reward
F ppppedd Fetal]
Pt
(26)
where rppeed is the forward speed reward (proportional to the x-velocity of the torso) and ictel encodes a control-cost penalty.
We construct d dimensional reward vector
Ft speed pepord
[r
Spopd
P rt
emergy
(27)
with
spend
ft
a, rtt
ctrl
aa
ctrl |
(28)
Tt
energy
Te ||aell rt
deadline
Here og speed oz .ostrl ?/ >> 0 are time-varying scales that may change when events occur (see below). The energy budget is updated as
E = Es - 7/t ||
(29)
and the deadline counts down as Dz+1 = D.
The deadline component r rdeadline is only non-zero at termination. If the episode terminates due to reaching a goal state before exhausting the deadline or energy budget, we grant positive bonus +
if it terminates due deadline expiration, exhaustion, we add an extra negative penalty on rf eneigy In our implementation we use 3 10. The environment terminates when either (i) the underlying MuJoCo simulator signals failure, (ii) D, S O (deadline reached), (iii) E2 S 0 (energy exhausted), or (iv) the maximum horizon Tmax is reached (in which case treat the episode truncated). No fixed scalarization is hard-coded into the environment; as in Maze, the agent must adaptively reweight the components of Fr.
assign penalty -B; and if it terminates due to energy
Dynamic events. To induce non-stationary trade-offs, we introduce three types of exogenous events that modify constraints and reward scales:
Deadline shock: the remaining time budget is suddenly shortened by a fraction. Concretely, event time sample shrink ratio p (0, 1) (default p = 0.5) and update
Dr max(1, Dt [pD:1}.
(30)
increasing the urgency of progress.
Speed surge: the importance of forward speed is increased by multiplying its scale:
speed + Expend a speed
(31)
with default multiplier Cspeed = 1.5. This models scenarios where performance pressure (e.g., tighter external requirement on forward velocity) suddenly rises.
Energy drought: the per-step energy cost increases, making actions more expensive:
f Cenergy Tl:
(32)
with default multiplier Cenergy = 2.0. This mimics a sudden reduction available power or effi- ciency, pushing the agent to adopt more conservative control.
20

---

Published as a conference paper at ICLR 2026
Table 5: Comparison results on the modified continuous control environment. To handle a continu- ous action space, we adapt the rule-based ENVELOPE Q-learning baseline to PPO actor-critic im- plementation (ENVELOPE-
Dense Oracle given privileged access to dynamic event signals and hand-crafted rules for updating the preference weights, whereas DPI-PPO (ours) only observes standard environment state and must infer preferences purely from interaction.
Method
MER
SR(%)
ENVELOPE-PPO
13.52 + 9.89
12.00 99.98 + 0.01
53.28
Dense Oracle
66.95 + 5.78
DPI-PPO (Ours)
42.10 = 6.25
81.00 1.00
D.4 HYPERPARAMETER SENSITIVITY ANALYSIS
To evaluate how sensitive DPI is to the auxiliary loss coefficients, we conduct a experiment on the QUEUE environment, varying the stability weight 7 and the directional-aligmment weight À in Eq. (10) while keeping all other hyperparameters fixed. For each setting, we train the agent with 10 random seeds and report the mean episodic return (MER) and success rate (SR) over 200 evaluation episodes in Table 6 Overall, DPI is fairly robust to moderate changes of y and X: performance under all tested configurations stays within one standard deviation the default setting, and only very small or very large coefficients lead to mild degradation in SR.
Table 6: Hyperparameter sensitivity analysis. Here and À are reported as relative multipliers around the default (7o Ap) used in the main experiments (i.e., 0.5 means 0.5%)
MER
SR(%)
0.5
1.0
10.84 1.33
38.57 2.10
1.0
1.0
10.34
39.95 + 2.75
2.0
1.0
0.62
35.85 1.02
1.0
0.5
9.70
30.43 3.27
1.0
2.0
8.85 + 1.62
36.93 0.46
D.5 DISCUSSION ON COLD-START DYNAMICS
In our implementation, early training is stabilized by two mechanisms.
(i) Gaussian prior and KL regularization. Because ( I-) is regularized toward (0, Z), the induced preference weights distribution over objectives. This prevents the appraisal module from committing extreme trade- offs when the critic is still inaccurate.
softmax(z,) are initially close to high-entropy, nearly uniform
(ii) Envelope selection with on-policy actor-critic. Even when the multi-objective returns G, are noisy. the envelope operator is applied over multiple samples drawn from a broad posterior, and the PPO update is on-policy with respect to the selected wi In practice, this behaves similarly to standard multi-objective actor actor-critic with stochastic exploration in preference space, rather than brittle deterministic scheduler.
Empirically, we do not observe systematic early mode collapse of preferences. Instead, the entropy of wi gradually decreases as the critic becomes more informative, consistent with a smooth transition from exploratory more specialized preference configurations.
D.6 LEARNING CURVES
To assess the training stability of DPI, we run 10 independent training the Queue environ- ment with different random seeds, using the default hyperparameters reported in Table 3 For each run, we log performance throughout training as follows:
21

---

Published as a conference paper at ICLR 2026
12.0
50
110
10.5
00
10.0
85
10-
010
20000
40000
60000
00000
100000
120000 1,4000
20000
80000 00000 100000
120000 140000
Total Environment Steps
Total Environment Steps
Figure 10: Mean episodic return (MER, left) and success rate (SR, right) as function of environ- ment steps. Solid lines show the mean over random seeds and shaded regions denote the 95% confidence interval. DPI-PPO exhibits smooth and monotone improvement without divergence or collapse.
After every 5,000 environment interaction steps, fix the current policy and derive greedy evaluation policy (breaking action ties uniformly at random).
We then evaluate this policy on 200 episodes without exploration noise and compute the episodic return (MER) and success rate (SR) for that evaluation point.
For each evaluation step, we aggregate MER and SR across seeds and report the mean and the 95% confidence interval, which are plotted in Fig. TO] shown in the figure, both MER and SR improve smoothly over time and eventually saturate, without divergence, collapse, large oscilla- tions. These observations indicate that the combined ELBO-based preference inference and PPO optimization yield stable training dynamics.
22


Reviews:
[{"review_id": "eV2ryos14R", "reviewer": "Reviewer_tNvs", "summary": "The paper considers dynamically changing preferences. It proposes an algorithm that leverages a signal providing information about these signals to modify its actions accordingly. Empirical results are presented on two settings (waiting in a queue, a maze problem) comparing the method to baselines.", "questions": "#1. In the \"waiting in line\" example used to motivate the paper and used in the numerical experiments, the phenomenon described can be modeled without the need to change goals.\n  The goal would be to maximize the expected value of the utility function 1{waiting_time < T} + c*cut_in_line.\n  The optimal strategy is to wait until T and, if we haven't gotten to the front of the line, cut.\n  The thing that is changing is not the goal but the optimal action (cut or not cut) given the current state (how long we've been waiting in line).\n \nLet me explain my example in more detail so that the authors can respond.  My example is a discrete time problem where there is a random variable W that is the time we would wait if we chose not to cut in line. This is drawn from a known probability distribution.  In each time period t, if we haven't gotten to the front of the line yet (i.e., t < W), then we would decide whether to cut in line or not based on t.  Any decision rule (a way to decide the action, i.e., whether to cut, based on the current state, i.e., t) results in a binary random variable cut_in_line, which is 1 if we eventually choose to cut in line and 0 if not. It is a function of W.  A decision rule also results in a waiting time, which is the minimum of W and when we choose to cut in line, if ever.  Using a dynamic programming argument where the state is the current time t, you can show that if c is large enough, an optimal strategy will be to wait until time T-1 and, if we haven't gotten to the front of the line, cut.\n\nThese same arguments extend to utility functions of the form f{waiting_time} + c*cut_in_line for a general function f. For most functions f and distributions over W, the optimal strategy will be to wait until some finite time that is strictly larger than 0 and then cut.\n\nThis is an optimal stopping problem.\n\n#2. The maze problem (the second numerical example) also seems to be better modeled by a fixed goal. What the paper positions as changes in goal really just seem like changes in the state in a Markov decision process (MDP) or partially observable MDP. For example, when trying to get out of a maze, a reasonable fixed goal is to minimize the expected time to get out of the maze. Over time, our knowledge about the maze changes. The maze itself may also change. As a result, the optimal action may change.\n\n#3. I am having trouble understanding the motivation for the paper.  I agree that there is a large body of evidence showing that humans \"switch goals\" over time. But this doesn't imply that algorithms should do this.\n\nIs the goal to develop a decision-making algorithm that is somehow better able to provide value to humans, in light of this empirically-observed goal switching?  Or is it to improve our understanding of human decision-making?\n\nMy sense is that the paper's goal is the not the second --- if it were, then we would presumably want to include human data in the experiments and make comparisons to other hypotheses.\nPresumably then it is the first.  But if this is the case, then it isn't clear that we want our algorithms to \"swich goals\".\n\nFirst, for settings like my comments #1 and #2 where human \"goal switching\" can be explained as the use of decision-making shortcuts to perform well against a larger goal that is static, wouldn't human utility be better maximiized by doing a good job of optimizing against the larger goal with a gool RL strategy?  More broadly, if \"goal switching\" can be explained by humans' bounded rationality because of biological limitations, then wouldn't a human prefer that an algorithm making decisions on their behalf be more rational than they are?\n\nI am not philosophically opposed to developing an algorithm for settings where humans switch goals but to be publishable at ICLR there would need to be a stronger set of examples where it is demonstrated that the new algorithm provides value to a human. To demanstrate that it provides more value to a human, it is important to somehow bring human data in to the paper, e.g., by having a human assess the quality of the outcomes provided by the proposed algorithm and baselines in a real-world setting.\n\n#4. Practically, how should a practitioner interested in using this method estimate p(s_t | omega_t) and p(omega_t | omega_{t-1}, xi_t)?\n\n#5. In the experiments, methods are evaluated relative to static success measures --- mean episodic return, success rate, and post-shift performance at K. If I want to maximize some combination of these (a weighted combination, or maximizing one subject to constraints on the others), then I should use a reinforcement learning algorithm designed for this task. The Queue environment seems particularly simple and I would expect that an RL algorithm could accomplish this. At a minimum, the paper should include as a baseline an RL algorithm that optimizes each of the single metrics reported.\n\n#6. Please help me understand some of the notation in section 3.1.\n- In equation 1, which is s_t not on the right-hand side of the equation?\n- In equation 1, presumably omega is a function of z_t even though the notation from equation 2 indicating this is not used here.  This notation from equation 2 is useful --- it should be introduced and defined in the text.\n- In equation 2, why does s_{t-H+1:t} not appear on the right-hand side?", "limitations": "", "rating": 4, "confidence": 2, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The paper considers an interesting under-explored area. There is strong empirical evidence that humans switch their stated goals over time. There is value in understanding this phenomenon and in designing algorithms that support humans despite these changes.", "weaknesses": "The motivation for the paper is unclear (see questions #1, #2, #3 below).  It is also unclear how to apply it to a practical problem (questions #1, #2, #4).  The experiments should have included stronger RL-based baselines that maximize the performance metrics reported (question #5).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "N6AEWol2bH", "reviewer": "Reviewer_b9am", "summary": "This paper studies the problem of the shifted preference as time and condition/state change. This problem is plural and complex, and the authors proposed a computational agent that comprises the Value Appraisal and Action Selection to model the dynamic decision-making systems.", "questions": "Introduction:\n- The statement of Line 28 needs some (psychological) literature to support. The instance given afterward does not fully support the claim \"rarely pursuing goals with fixed and immutable priorities\".\n- In line 36-38, I don't see a clear necessity between the modeling of such dynamic value adaptation and artificial intelligence. Why use AI in particular; why not some other type of modeling? The motivation for using AI is not clear in the writing. You may want to expand a bit on why current literature fails in the computational aspect and why AI can be helpful.\n- In line 32-34, I don't think \"psychology\" and \"cognitive science\" are two distinct/mutually exclusive terms. They might refer to similar stuff. \n- In line 50/51, what does \"decision-making models\" exactly mean?", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": 1, "contribution": 3, "strengths": "- The paper identifies an interesting problem that may not be fully aware of in the machine learning literature. This problem itself has connections with reinforcement learning and MDPs.\n- The proposed framework is quite interesting, and the Bayesian view of this problem is natural to follow. I like how the value shift is integrated in the framework, and I also feel that using the sample example for explaining the framework helps a lot.", "weaknesses": "- The paper starts with a lot of terminology but lacks an explanation of these terms. Although I'm quite familiar with the choice model/psychological literature, it is hard to tell if these terms are made up/invented by the authors. I strongly suggest avoiding overuse or misuse of terminology and approaching from an easy-to-understand tone. Besides, proper references for the well-defined terms are needed.\n-- I pointed out some points in the Questions section, but there are many more confusing points.\n\n- The paper does not highlight the computational contribution and what makes any other approach hard to address the research question.\n\nI find it quite hard to raise the score to a positive one because of the whole writing and idea delivery, but I would not mind giving a borderline reject if the author could clarify and improve the flow.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wRXT6aRfZh", "reviewer": "Reviewer_6Kvv", "summary": "The paper addresses the challenge of Multi-Objective Reinforcement Learning in dynamic environments where preference weights are not fixed or explicitly given, but are instead latent and subject to change based on contextual shifts. The authors propose a cognitive-inspired framework called Dynamic Preference Inference (DPI). DPI consists of two main components: a Value Appraisal Module that uses variational inference to maintain a posterior distribution over latent preference weights based on recent history, and an Action Selection Module that employs a preference-conditioned actor-critic to execute policies based on sampled preferences. The framework is validated on two synthetic environments (Queue and Maze), showing that it can adapt to sudden changes in task dynamics (e.g., \"deadline shocks\" or \"hazard surges\") better than static or heuristic baselines.", "questions": "1. Dense Oracle Baseline: Could you compare DPI against a \"Dense Oracle\" baseline that has access to analytically derived optimal preferences at every timestep (e.g., analytically derived from exact remaining time and energy)? This would provide a stronger upper bound to gauge the true effectiveness of DPI's inference module compared to standard envelope methods.\n\n2. Hyperparameter Sensitivity: Given the composite nature of the loss function, how sensitive is the model's performance to variations in the auxiliary loss coefficients $\\lambda$ (alignment) and $\\gamma$ (stability)?\n\n3. Cold-Start Dynamics: How does DPI behave early in training when the critic's vector return estimates $\\vec{G}_t$ are likely noisy or uninformative? Does this lead to early collapse into sub-optimal preference modes?\n\n4. Theoretical vs. Practical Gap: The generative assumption is $\\omega\\_{t}^{*}\\sim p(\\omega\\_{t} | \\omega\\_{t-1},\\xi\\_{t})$, but the implementation uses a static prior $p\\_{0}(z)=\\mathcal{N}(0,I)$. While the recurrent encoder handles history, did you experiment with strictly enforcing the temporal dependency in the ELBO (e.g., a learned transition prior)?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "- Novel Problem Formulation: The paper identifies and formalizes a significant gap in current MORL research: treating preference weights as latent and dynamic states that must be inferred online, rather than static inputs. This is well-motivated by cognitive theories of human decision-making.\n\n- Principled Framework: The DPI method offers a principled probabilistic approach, synthesizing variational inference for preference estimation (using ELBO) with established preference-conditioned RL techniques (like the envelope operator).\n\n- Demonstrated Adaptivity: Empirical results on the test environments clearly demonstrate the method's ability to recover performance (measured by Post-Shift Performance, PS@K) rapidly after unobserved environmental shifts, significantly outperforming fixed-preference baselines.\n\n- Interpretability: The inclusion of alignment analysis (cosine similarity between inferred preferences and instantaneous reward vectors) provides evidence that the learned latent preferences meaningfully track shifting task semantics.", "weaknesses": "- Limited Experimental Scope: The evaluation is restricted to two relatively simple synthetic environments: a symbolic queue task and a 2D grid-world. While illustrative, these do not fully demonstrate the method's scalability to complex, high-dimensional, or continuous control problems often seen in practical MORL settings.\n\n- Potentially Weak Oracle Baseline: The ENVELOPE baseline, described as having \"oracle access to event-dependent preference weights\" , performs surprisingly poorly (e.g., only 0.01% success rate in Maze ). This baseline appears to use sparse, static weights triggered by events. A true \"oracle\" in such highly dynamic environments should likely use dense preference updates at every timestep (e.g., exponentially increasing urgency as a deadline approaches, not just a flat jump in weight when a \"shock\" occurs). The large performance gap might simply reflect poorly tuned static weights for the baseline rather than the absolute necessity of inference.\n\n- Complexity of Objective: The total training objective is a composition of many terms: PPO loss, dual critic loss, ELBO, directional alignment ($\\mathcal{L}\\_{dir}$), and self-consistency ($\\mathcal{L}\\_{stab}$). The ablation study confirms standard components alone (e.g., w/o $\\mathcal{L}\\_{dir}$ or $\\mathcal{L}\\_{stab}$) perform significantly worse, suggesting the method might be highly sensitive to the hyperparameters balancing these diverse loss terms ($\\lambda, \\gamma, \\xi, \\beta$).\n\n- Dependence on Return Vectors for Alignment: The directional alignment loss relies on vector returns $\\vec{G}\\_t$ estimated by the critic. In environments with very sparse rewards or difficult exploration challenges, these return estimates might be highly inaccurate early in training, potentially leading to poor preference inference that is hard to recover from.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "I3oARGquYx", "reviewer": "Reviewer_9DQE", "summary": "A cognitively inspired RL framework is proposed for dynamically inferring agent preferences in non-stationary, multi-objective environments. The method jointly trains preference inference and control via variational Bayesian optimization, achieving interpretable and adaptive behavior. Experiments in two environments demonstrate the effectivenss of the proposed method.", "questions": "1. In Lines 93–94, the definition of $\\omega_t$ is unclear. Please clarify its meaning and justify why it is denoted in bold.\n\n2. The choice of a unit Gaussian prior lacks sufficient motivation. It would be helpful to provide a rationale or empirical justification for this assumption.\n\n3. In Line 203, the derivation of the presented formula is not fully explained. Additional details or intermediate steps would improve clarity.\n\n4. In the Maze environment, the authors claim that “without dynamic preference adjustment, the agent cannot succeed.” Please elaborate on how this statement was verified or implemented in practice.\n\n5. In Table 1, the success rate (SR) should also be reported with the mean and 95% confidence interval (CI) across $N$ episodes, consistent with the other metrics.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The central idea of treating value preferences as latent, context-sensitive cognitive states inferred through Bayesian variational methods is novel and interesting.\n\n2. The manuscript is clearly organized and flows logically from motivation to methodology to experiments.\n\n3. The experimental design is good, featuring comprehensive analyses and ablation studies that offer valuable insights into the model’s behavior.", "weaknesses": "1. The experiments are confined to relatively small, synthetic domains (toy symbolic and 2D Maze). Although cognitively illustrative, these environments may not fully demonstrate the scalability or generalizability of the proposed DPI framework. More complicated continous environments or real-world cases that better reflect the changing preferences are needed.\n\n2. Although the paper introduces a principled ELBO-based training objective with regularization terms, it lacks sufficient analysis of training stability, including convergence behavior and sensitivity to hyperparameters.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)