MTRec: Learning to Align with User Preferences via Mental Reward Models
Authors: Mengchen Zhao
Yifan Gao
Yaqing Hou
Xiangyang Li
Pengjie Gu
Zhenhua Dong
Ruiming Tang
Yi Cai
Venue: NeurIPS 2025 poster ()
Keywords: recommender systems
reinforcement learning
Primary area: applications
TL;DR: We propose MTRec, a novel sequential recommendation framework which uses a learned mental reward model to guide the recommendation model to align with users' real preferences.

Abstract: Recommendation models are predominantly trained using implicit user feedback, since explicit feedback is often costly to obtain. However, implicit feedback, such as clicks, does not always reflect users' real preferences. For example, a user might click on a news article because of its attractive headline, but end up feeling uncomfortable after reading the content. In the absence of explicit feedback, such erroneous implicit signals may severely mislead recommender systems. In this paper, we propose MTRec, a novel sequential recommendation framework designed to align with real user preferences by uncovering their internal satisfaction on recommended items. Specifically, we introduce a mental reward model to quantify user satisfaction and propose a distributional inverse reinforcement learning approach to learn it. The learned mental reward model is then used to guide recommendation models to better align with users’ real preferences. Our experiments show that MTRec brings significant improvements to a variety of recommendation models. We also deploy MTRec on an industrial short video platform and observe a 7\% increase in average user viewing time.

MTRec: Learning to Align with User Preferences via Mental Reward Models
Mengchen Zhao'
Yifan Gao?
Yaqing Hou Xiangyang Li Pengjie Gu
Zhenhua Dong²
Ruiming Tang² Yi Cai
School of Software Engineering, South China University of Technology School of Computer Science and Technology, Dalian University of Technology "Nanyang Technological University {zzmc, ycai>@scut. edu. en {otz, houyq>@mail, dlut. .lut.edu. edu...dd... cn (lixiangyang34, dongzhenhua, tangruiming) @huawei com
HHuawei Noah's 's Ark Lab
Abstract
Recommendation models are predominantly trained using implicit user feedback, since explicit feedback is often costly to obtain. However, implicit feedback, such as clicks, does not always reflect users' real preferences. For example, user might click on a news article because of its attractive headline, but end up feel- ing uncomfortable after reading the content. In the absence of explicit feedback, such erroneous implicit signals may severely mislead recommender systems. In this paper, we propose MT'Re, novel sequential recommendation framework designed to align with real user preferences by uncovering their internal satisfac- tion on recommended items. Specifically, we introduce a mental reward model to quantify user satisfaction and propose a distributional inverse reinforcement learning approach to learn it. The learned mental reward model then used to guide recommendation models to better align with users' real preferences. Our experiments show that MTRec brings significant improvements to a variety recommendation models. We also deploy MTRec platform and observe a 7% increase in average user viewing time.
industrial short video
Introduction
In interactive recommender systems, explicit feedback (e.g., ratings) is inherently sparse. Conse- quently, recommendation models predominantly rely on implicit signals (e.g., clicks) for training. However, such signals frequently fail to capture users' real preferences. For instance, clicking video may not indicate satisfaction with its content, while skipping a video could stem from prior exposure to similar content other platforms rather than genuine dislike. These observations highlight fundamental misalignment between recommendation models and users' real preferences.
To mitigate such misalignment caused by erroneous feedback signals, a natural approach is to incentivize users to provide explicit feedback. However, in real-world scenarios, users exhibit low propensity to offer such feedback due to cognitive burdens and interface constraints. Prior studies treat erroneous feedback signals as noisy labels and applying denoising techniques address them, yet their effectiveness remains limited because erroneous feedback is not random noise by its nature Wang al. 202T]. Some alternative methods attempt to mitigate erroneous feedback via nulti-feedback fusion, yet they often struggle when confronted with confficting feedback Chen al. [2021a]. Overall, existing works focus on data mining approaches, lacking a deep understanding of the mismatch between users' implicit feedback and their real preferences.
"Corresponding author.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

In this work, we aim to quantify and uncover users' internal satisfaction with recommendations, thereby bridging the gap between the recommendation model and users' real preferences. In fact, each time user takes an action (e.g., consume an item), private feeling will be generated in her mind, telling how she satisfied by taking the action. We summarize such private feeling as mental reward. We have following two observations on the mental reward. O1: The mental reward will influence user's short-term interests and her subsequent behaviors. For example, if an user clicked on a news but felt uncomfortable with the content, she would lose interests that topic and probably not click on similar news again. O2: Users are maximizing their accumulated mental rewards. This is reasonable because users naturally pursue good experiences during interaction with the recommender system. The above observations indicate that mental reward plays an important role in user's sequential decision making. If we can directly optimize user's mental rewards, the recommendation model would be better aligned with users' real preferences.
To this end, we propose MTRee, a novel sequential recommendation framework which uses learned Mental Reward Model to guide the recommendation model to align with users' real preferences. First of all, we model the user's decision making as Markov Decision Process (MDP). With the assumption that the user always maximizes her accumulated mental rewards, we use Inverse Reinforcement Learning (IRL) to infer a mental reward function from users' behavioral data. However, plain IRL recovers a deterministic mental reward function, which fails to capture the random nature of the mental rewards. To address this, we propose Quantile Regression Inverse Q-Learning (QR-IQL) approach to learn a distributional mental reward function, which maps a state-action pair to distribution of mental rewards. Hence, use the rewards predicted by the mental reward model as complementary supervision signals to guide the training of recommendation model. In such a way, the misalignment between recommendation model and user's real preferences can be greatly reduced. Experiments on two public datasets show that MTRec significantly improves the performance of several popular recommendation models, in terms of Area Under Curve (AUC) and Normalised Capped Importance Sampling (NCIS). We also test MTRec in Virtual Taobao to demonstrate its effectiveness on reinforcement learning based recommendation models. Moreover, we deployed MTRec in a real-world industrial short video recommendation platform and observed 7% increase in average user viewing time over 7-day period during the online A/B test.
Our main contributions are summarized as follows.
We identify the misalignment problem in sequential recommendation, where erroneous user feedback could severely deviate recommendation model from users' real preferences.
We introduce MTRec, novel sequential recommendation framework that aims to bridge the gap between the recommendation model and users' real preferences by a learned mental reward model, which uncovers users' internal satisfaction with recommendations.
To capture the random nature of the mental rewards, we develop distributional variant of IRL called QR-IQL to learn the mental reward model. We show how to use the learned mental reward model to guide the optimization of sequential recommendation models.
We conducted extensive offline and online experiments to demonstrate the improvements brought by MTRee. Additionally, we deployed MTRec real-world industrial short video recommendation platform and observed a significant increase in user engagement.
2
Related Works
Implicit user feedback in recommendation. Since explicit feedback is very sparse, industrial recommender systems rely on implicit feedback (e.g., click, video watching) to train recommendation models Covington al. [2016), Zhou et al. 20118) Dai et al. [2021] Yang et al. (2021). However, these models ignore the fact that implicit feedback may not reflect users' real preferences. Some works treat such erroneous feedback as random noise and try to eliminate its influence via denoising, yet these methods have limited accuracy because it is hard to distinguish it from genuine responses Yuu and (200)),
, WWang al.) (202T). Another line of works address various types of biases in user feedback, including position bias, selection bias, popularity bias and exposure bias woong Lee et al. 2021 Yi et al. [ However, errors implicit user feedback differ from the aforementioned biases. In some sense, such errors can be regarded as certain type of inductive bias Jiawei et al. 2023 because implicit feedback is wrongly assumed reflect users' real preferences.
2

---

AI alignment through reward model. Emerging from Natural Language Processing, alignment algorithms have proven effective due their ability to guide Large Language Models (LLMs) in matching human values. Many popular alignment methods employ reward model to provide fine-tuning signals. For example, Reinforcement Learning from Human Feedback (RL.F) learns human preferences through reward model trained with human-rated outputs Ouyang et al. [2022]. Reward rAnked Fine Tuning (RAFT) uses a reward model to select the best set of training samples based on model outputs Dong et al. [022]] Inspired by these works, we aim to learn reward model to guide recommendation models in aligning with users' real preferences. Unlike prior approaches that rely human annotators to provide reward model labels, our reward model is directly learned from existing user behavioral data, making it more calustry-friendde
RL and IRL for sequential recommendation. Sequential recommendation is typically modeled as interactions between users and recommender systems. Prior works have explored using reinforcement learning (RL. to optimize recommendation policy, where the recommender system modeled as an agent and the users are treated key components of environment [hen et al. [2019]. Zheng jet al. 2018). RL-based methods have great potential to maximize recommender systems' long-term revenue, but they often suffer from the bias of user simulators. Inverse RL (IRL) aims to recover the agent's 's reward function from expert trajectories. Some works apply IRL to infer the reward function for the recommender system, assuming expert recommendation policies are available Chen et al.) (2021b). et al. [2023a], However, these works ignore that the users are also active agents, and understanding user behaviors is essential for improving recommendation policies. In our work, model users as agents and use IRI. to infer the optimal reward model from their behaviors. Although similarly employing IRL techniques, work differs fundamentally from existing works by inferring a user-centric reward model, as opposed to system-centric reward modeling.
3 Preliminaries
Sequential recommendation. Typically, a sequential recommendation model takes a sequence of user-item interactions as input and predicts the next items that mostly attract the users. Due to the lack of explicit feedback, sequential recommendation tasks are usually formulated as predicting the next item that is most likely to induce target user behaviors such as clicks:
L = arg max (āli,, h1-1), SEI
where à represents user's behavior to be predicted, 1 = (i1, y.1.... iN } is the set of candidate items. The parameterized function Pa measures the probability of the target behavior à. The interaction history hy consists of list of tuples up to time step where each tuple < u, i, a > consists of u, an item i and the user's action a E A. In simplified case, user's action space A can be restricted to {click, skip), hence the recommendation task is reduced to predicting the user's click-trough rate.
Inverse reinforcement learning. Conventionally, maximum entropy IRL aims to learn deterministic reward function r: S x A R by solving Problem 2 Brian et al. 20088.
max min E.zg (r(s, a)) R. -(()))
rER #Ell
where H(T) A Ez log [a]ss] denotes the entropy of policy т. Intuitively, this formulation learns reward function that assigns high reward to the expert policy ag and a low reward to other policies, while searching for the optimal policy under the reward function in the inner loop. The expectation over policies can be replaced by the occupancy measure p=(s, a) T(als) Σ P'Ps s|T),
which specifies a probability distribution over (8, a) pairs. Then, Problem 2 can rewritten as:
maxmin Epg (rss, a)] - EE (s.sss.)) a)] - H(T) - o(r), PER wEll
(3)
where & is a convex regularizer on r Jonathan and Ermon 2016 If we define the reg- ularizer as (bx) = x ((x)), then Ep (psgr(s, a]] ((r) can be compactly represented by Epe ()r(((((s)))) Moreover, the reward function can be represented by the soft Q-function as
3

---

r(s, a) = ((s,a) - --By--(ss))) V(s), where V(s) = Ba-z( (-10)(111) (-18) Q(s a) log क (a]s)] Also the optimal policy nT can be represented by the soft Q-function as π (a|s) = A(s exp(Q(s, a)), where (ss) = Ee exp(Q(s,,) (()))) the normalization factor. Then, the inner problem becomes trivial so
that Problem can be reformulated as follows (rrian et al. 2008). Haarnoja et al. 2018
max B, sampE (Q(, - P(s.) V QER
(1 po (V [Vo(s)]],
where V (s) = log E. exp Q(. a). Note that the objective of Problem 4 depends only on Q. which allows us to solve the problem by directly optimizing Q-network. Solving Problem 4 with the expert data Dg results in the optimal Q-function, and the deterministic rewards can be recovered by:
r(s, a) = Q(s, a) - -EEE log Σ expQ(ss'aa)
(5)
The misalignment problem in recommendation. In the context of LLM, the alignment problem is defined as optimizing the outputs of LLMs towards matching human values Ouyang et al. [022]] For recommender systems, the general goal is to maximize users' satisfaction by selecting appropriate items for them. This is naturally in accord with the alignment problem in LLMs,, in the sense that aligning with human values is similar to aligning with user preferences. However, directly maximizing user satisfaction is very challenging for recommender systems. Therefore, most existing works focus on optimizing some surrogate objectives such as click-through rate and conversion rate Wang et al. 2017]. Zhou et al.) 2019]. Although the surrogate objectives make it convenient to optimize recommendation models, they suffer from inductive biases since users' implicit feedback may not reflect their real preferences. As a consequence, the optimization goal of recommendation models may deviate from users' real preferences. Compared with one-shot recommendation, the misalignment problem in sequential recommendation even worse due to the accumulation of errors.
Inspired by recent advances in LLM alignment, we aim to develop a reward model that helps to align recommendation models with users' real preferences. The challenges two-fold. First, do not have an off-the-shelf reward model that reflects users' real preferences, under the lack of their explicit feedback. To address this, we propose the Mental Reward model learned from rich user behavioral data to approximate their real preferences. Second, unlike static human values, users' preferences could be highly stochastic, potentially due unpredictable environmental factors and preference shifts. To resolve this, we propose a distributional Inverse RL approach to capture the randomness of the mental reward model, which will be illustrated in Section
Method
Motivation. User behaviors have been extensively studied in the literature of recommender systems. However, existing works treat user behaviors as labels for recommendation models. In fact, during the interaction with recommender systems, users are active agents rather than static label providers. Moreover, users are implicitly maximizing their own reward functions by taking actions in recommender systems. Based on the above insights, we believe that uncovering the user's reward function would significantly benefit recommendation models in aligning with users' real preferences.
Overview. We propose MenTal reward based Recommendation framework MTRec, which consists of three main parts. First, we introduce novel User-Centric Markov Decision Process, where uses are modeled as active agents during their interaction with recommender systems. Second, we develop distributional IRL method called Quantile Regression Inverse Q-learning (QR-IQL) for learning the mental reward model. Last, we show how to use the learned mental reward model to guide existing recommendation models align with users' real preferences.
4.1 User-Centric Markov Decision Process
Reinforcement learning has been applied to model the interaction between users and recommender systems. Existing works usually model the recommender system as agent, who recommends items and receives user feedback reward Zheng et al. 20188 Chen et al. 20019. However, they often

---

Mental Reward
Interaction Data
Model
Tt
St
St+1
User Policy
Recommendation
Model
at
The overall framework MTRec. The solid lines represent the interaction process. The dashed lines represent the information flow between data and models. Our goal is to recover the mental reward model and use it to improve the recommendation model.
Figure I:
ignore the strategic behavior of the users. In this work, we focus on studying users' behaviors, whose decision making can be modeled as Markov Decision Process M = = P, T))
S denotes the state space. A state s, (he, it) includes the interaction history up time t. At each time step, the interaction is recorded by tuple < u, >,, consisting of an user displayed item and the user's action a.
A denotes user's action space. An action displayed to her. User's responses could be of various types, including explicit feedback such news clicks and video watchings. In this work, consider general case where users' responses are either positive or negative, leading to a simplified action space.
A represents user's response to the item
S - denotes transition function. Following existing works on sequential recommendation Zhou et al. (2018) 2019). Wang-Cheng and McAuley (2018), after an user takes an action, new state 5c-1 = St (h141,44+1) will be generated.
R: x + (rr) denotes the mental reward function which maps a state and an action to a distribution of mental reward E R. Note that we model the mental reward as random variable instead deterministic value, since could be influenced by unpredictable environmental factors and preference shifts. Such a modeling allows us to capture the intrinsic randomness and potentially richer information on user's mental rewards.
S plaa denotes the user's behavioral policy, which maps actions. A stochastic user policy facilitates reasoning her reward function using IRI.
distribution
Figure Iillustrates the interaction process between user and recommendation model. At each round, the user receives recommended item and select action to respond. After that, mental reward that summarizes user's satisfaction about the current item generated. Note that the mental reward highly correlated with user's real preference, but is unknown to the recommendation model. However, the mental reward will influence the user's ''s subsequent behaviors as she seeks to maximize the accumulated mental rewards. We will focus on estimating the mental rewards in the following parts.
4.2 Uncovering the Mental Reward Model
Inverse reinforcement learning (IRL) aims find reward model that explains behaviors of the expert policy (Saurabh and Doshi (202T) Since the user naturally maximizes her mental rewards, her policy can regarded as the expert policy. Therefore, IRL could be used to recover the mental reward model using the interaction data. However, existing IRL methods focus deterministic reward models, which fail to capture the intrinsic randomness of the user's 's mentul rewards. this end, we will develop distributional version of IRL algorithm for uncovering the mental reward model.
4.2.1 A distributional perspective on IRL
Conventional IRI. methods such as MaxEntIRL. iteratively optimize the reward function and the policy until convergence Brian et al. 20088. Bayesian IRL methods assume that there are multiple reward
5

---

functions and focus on estimating their posterior distribution using Bayes' rule Deepak and Amir [2007]. Jaedeug and Kim [2012). Although Bayesian IRL methods introduce various distributions over reward functions, they are still restricted to deterministic reward functions. By contrast, we aim to learn a distributional reward function where the reward can be stochastic.
Since we model the user's mental reward as random variable, Equation can be rephrased using distributional operator as
( >E (log Σ exp 2(s
Exxp
where X _ U denotes equality of probability laws, that is, the random variable X distributed according to the same law as U. As each Q(s, a) function uniquely determines a distribution of r(s, a), learning a distributional reward function is reduced to learning a distributional Q-function.
4.2.2 Quantile Regression Inverse Q-learning (QR-IQL)
Problem aims to learn deterministic Q-function, while we aim learn distributional Q- function. Following the QR-DQN Will et al. (2018), the distribution of Q can characterized by quantile distribution. We denote by Z the variable associated with the distribution of Q, that is, (s,, a) = (s, a)]. Let : S x A -> R. be parametric model, where N is the number of quantiles. Then quantile distribution Zx maps state-action pair (s, a) to uniformly probability distribution supported on {Ae(s, a) y Instead of learning a scalar value Q(s, a), our model will estimate the positions supports {A, a)) and calculate Qx(8, a) as:
N
Qx(s, = N
Σ 21
A(ssa))
Recall that Problem 4 we optimize a Q-network that outputs a scalar Q-value, order to learn the distributional Q-function, we change the output layer of the Q-network to be of size |A] x N. To derive the objective for the quantile regression inverse Q-learning, we made two modifications on Problem 4 First, since our mental reward model is learned from offline data, the second term other Ex m0 [so)] is irrelevant with the initial state distribution po. Second, as is suggested in Garg et al. 2022 J, we choose the regularizer as (bx)) = x _= for the ease of optimization while bounding the rewards. Finally, we formulate the objective of the quantile regression inverse
in Problem (i.c., (1 7 [V (so)) can be replaced by (ss.a)=p (V(s) - (s')).
words,
Q-learning as Problem 6 The complete derivation is provided in Appendix A.
max Ep [Qx(s, a) log Σ exp Qx(s.))) Qx
- Eps [[Qx a) log Σ exp Qx(s' (6)
1
4a
We apply the Pinball loss get the optimal quantile distribution supports a)) and the optimal Q3, which we can use to calculate the mental rewards as:
r r' a) = (i. PEp Dlog Σ exp (8 a)).
While building upon ideas from QR-DQN for RL, our QR-IQL is the first distributional IRL algorithm that uncovers the underlying distribution rewards, which is key to capture the randomness of users' mental rewards.
Detailed optimization algorithm is provided Appendix A.2
4.3 Applications of the Mental Reward model
Generally, sequential recommendation model takes user features, candidate item features, interaction histories and some contextual features as input, and output score for the candidate item used for ranking. For the sake of brevity, we represent the recommendation model as F; (e which is parameterized by Ç. Although the mental reward model r(s, a) indicates users' preferences to some extent, it lacks sufficient feature-level modeling and thus cannot be directly used for recommendation. Instead, we use r(s, a) to provide additional learning signals for recommendation models, However, combining r(s, a) with existing recommendation models is non-trivial due to the

---

variety of learning objectives. Fortunately, these objectives fall into several categories. We will use the following two typical examples to illustrate how use the mental reward model practice.
Classification-baseed models. Many sequential recommendation tasks are formulated as binary classification problems with the following Cross Entropy loss:
Cce(C) = la log(F(((lA)) + a log(1 Fc(lbb)),
(8)
where af = and ax = 0 indicate user's positive and negative responses, and (ilhh represents the estimated probability clicking on item based on history h. We want the recommendation models to also maximize the expectation of user's mental rewards, leading to the following alignment loss:
Aliign (S) -D,,, [r" (s, a²) FG(())) + r" (s, a²) ) (1 Fc(()N))]:
Then, the final loss for training the recommendation model can be written as weighted combination of the two losses:
Crinat LCr(S) + (S)
(10)
RL-based models. In typical setting, the recommender system is modeled as an agent, who maximizes the accumulated system rewards. We denote by f(h, i, a) the system reward after recom- mending item i and receiving user feedback a given interaction history h. In this context, (i(h)) represents the RL-based recommendation policy, whose goal is to maximize the expectation of accumulated rewards
(S) = E |> ((h1-1,
(11)
Since we want the recommendation model to also maximize the mental rewards of the user. We simply add the mental reward r* f and obtain the following objective.
(Fi Fi E (( + +-- (h arl
(12)
See Appendix A.3 for more implementation details.
5
Experiments
In this section, we report the performance of MTRec in both offline and online settings, with focuses on answering the following research questions (RQs).
(RQ1:) How does MTRec improve classification baase recommendation models?
(RQ2:) How does MTRec improve RL-based recommendation models?
(RQ3:) Does the learned mental reward model provide useful information?
(RQ4:) How does MTRec perform in online A/B test?
5.1 Experiments on Public Datasets (RQ1)
Datasets. The Amazon dataset McAuley et al. [2015] collects user review data from Amazon e-commerce platform. We use two subsets of the Amazon dataset: Books and Electronics in our offline experiments. More details on processing the datasets are provided in Appendix A.4
Baselines. We use eight widely used recommendation models baselines and combine each of them with MTRec to test the improvements brought by MTRec. Wide& Cheng et al. [2016] hybrid recommendation model combining a wide linear model and deep neural network for collaborative filtering PNN Qu et al.] 2016) is neural network architecture designed for CTR prediction recommender systems. DeepFM Guo al. (207)] a hybrid recommendation model combining factorization machines and deep neural networks. SASRec Wang -henng and McAuley 2018) uses the self-attention mechanism to model sequential patterns for recommendation systems. DIN et al. (0088) is attention-based neural model for sequential recommendation, where the attention mechanism aims to distinguish the interest of a user's historical behaviors. DIEN Zhou et al. (2019) designs sequential architecture to model interest evolution for recommendation.
7

---

which uses an auxiliary loss to capture temporal interests. LinRec Liu et al. [003b]] is lightweight linear recommendation model designed for efficient computation and scalability with large datasets. SIGMA Liu et al. (2024) is sequential recommendation model that uses selective gating mechanism to focus on the most relevant user behaviors for improved performance.
Evaluation metries. We use the following two metrics: Area Under Curve (AUC) Fawceti 2006 and Normalised Capped Importance Sampling (NCIS) Swaminathan and Joachims 2013 AUC is used to measure the model's 's ranking ability and NCIS is used approximate the model's online performance Gilotte et al. (2018). Formally, the score of NCIS can be calculated by:
E P((M) L;: Σ (:
JNCIS(M)
where (MM = IIET (cM is the probability that the CTR model M follows the request trajectory of the user i. ((MM) the click-through rate estimate of model M for item f and n the number of users in the test set for NCIS. In the experiments, we use the complete trajectories of 10% of users to calculate NCIS. Moreover, we obtain the final NCIS score by substracting the NCIS of the untrained model from that of the trained model to eliminate the impact of random parameters among different models. Intuitively, JNCIS (M) awards CTR model with high score if the model has large probability to follow long trajectories.
Results on Amazon datasets. Table summa- rizes the experimental results on the two Ama- zon datasets. The suffix "IRL" indicates that the mental reward model is learned by a non- distributional IRL algorithm IQ-Learn Garg et al. [2021]. while the suffix "MTRec" indicates that the mental reward model is learned by our algo- rithm QR-IQL. It can be observed that integrat- ing MTRee with existing models consistently improves their AUC and NCIS across almost all baseline models. Moreover, the improvements on NCIS are generally more significant than AUC, This aligns with the motivation of MTRec that is, to maximize overall user satisfaction and long-term engagement. Note that the AUC is also slightly improved, demonstrating that the improvement of users' long- long-term engagement is not at the sacrifice of the model's 's ranking ability. In addition, the comparisons between IRI. and MTRec demonstrates the benefit of learning distributional version of mental reward model.
Electronics
Books
Model
AUC
NCIS
AUC
NCIS
Weee&
0.8290
0.6749
0.8605
2.0967
Wide&Deep-IRL
0.8342
0.8738
0.8661
3.1264
0.8351
0.9063
0,8657
3,2043
PNN
0.8396
0.6618
0.8603
2.3316
PNN-IRL
0.8547
0,9297
0,8673
3.4567
PNN-MIRee
0.8542
0.9515
0.8679
3,4971
DeepI'
0.8424
0.6993
0.8614
DeepFM-IRI
0,8458
0.8542
0,8715
3.7296
DeepEM
McTReM MTRec
0,8468
0.8742
3.79IM
SASRe:
0.8325
0.8241
0,8675
3.1755
SASRec-IRL
0.8366
0,8681
0,8681
3, 1949
SASRec-MTRO
0.8328
0.8833
0.8798
3.4634
DIN
0.8523
0.6044
0.8653
2.1324
DIN-IRL
0.8533
0.8168
0.8701
3,1675
DIN-MTRec
0.8542
0.8728
0.8732
3,2208
DIEN
0.8448
0.7766
0.8686
2.2685
DIEN-IRK
0.8461
0.8857
0.8723
3,0476
DIEN-MTROC
0.8472
0.9324
0.8757
3,1185
Linkee
0.8579
0.8077
0.8754
2.6533
LinRec-IRI
0.8597
0,9365
0,8771
3,8005
LinRee MTRec
0.8594
0,9782
0,8792
3.7928
SIGMA
0,8581
0.7945
0.N762
2,3975
SIGMA-IRL
0.8592
0.9261
0.8802
3,7556
SIGMA MTRec
0.8604
0.9563
0,8814
3,8025
Table I: Experimental results on Amazon datascts.
5.2 Experiments on Virtual Taobao (RQ2)
In this set of experiments, we choose RL-based recommendation models as our baselines and combine them with MTRec to test their performance. Both training and testing of the algorithms are conducted in simulated interactive recommendation environments on Virtual Taobao [Shi et al. (2019) We construct an expert dataset containing 100,000 high-quality trajectories by recording trajectories with averaged CTR:0.5, After training the mental reward model using the expert data, we add the predicted mental rewards to the original rewards and train the baselines again.
Baselines. Virtual Taobao allows training recommendation policies by RL. We use Proximal Policy Optimization (PPO) Schulman et al. [011]] and Soft Actor- (SAC)Haarmoj) et al. [2018) as baselines. According to Equation] 12 RL-based recommendation models can be adjusted by simply adding the mental rewards to the base rewards ffinal Tene Tmental- In Virtual Taobao, Tenr provided by a pre-trained user model. We set K = 0.2 for trade-off between the two reward signals.
8

---

0.8
CTR
0.4
0.2
PPO
SAC
PPO-MIRce
SAC-MTRee
10000
20000
30000
40000
50000
0
500
1000
1500
2000
Steps
Steps
Figure 2: Training curves of RL models, Averaged CTR reported with 95% confidence interval.
6:77
01
Staps
Figure 3: Illustrations of the predicted mental rewards. (a) Averaged mental rewards by steps in all trajectories; (b-e) Expected and counterfactual mental rewards given actual user actions.
Evaluation metric. The major evaluation metric used in Virtual Taobao is the episodic Click- Through-Rate (eCTR) during the simulated online interaction. The eCTR is calculated as:
Tepiiode
cCTR
10 * Nstep"
where 10 is the number of items recommended in single page, repisode is the total number of clicks in an episode and Natep is the total number of steps.
Results on Virtual Taobao. From Fig we can see that the baselines attain average CTRs of 0.5435 and 0.7055 respectively. By incorporating the mental rewards during training. PPO and SAC show significantly improved performance at average CTRs of 0.678 and 0.909 respectively. This demonstrates that the mental rewards provide more useful information about users' real preferences and successfully boost the performance of RL-based recommendation models.
5.3 Evaluation of the Mental Reward Model (RQ3)
is almost impossible to evaluate the learned mental reward model directly due to the lack of ground- truth mental reward labels. Yet, we evaluate the mental reward model indirectly by its correlation with the datasets. First, we calculate the averaged mental rewards at different steps in all trajectories. Figure 3 (a) shows the results of the first ten steps on Amazon Book dataset. We can see that the averaged mental rewards decrease obviously with the increase of steps. This result aligns with our intuition that users may get tired and receive less mental rewards during their interaction with the recommender system.
Second, we visualize the distribution of the mental rewards conditioned on the users' real responses. For instance, Figure (b) and (c) show the distributions of expected mental reward r" r*( a" (areal aP) and the counterfactual mental reward r" rs a [Great = aP) when the users take positive action areal a.. Specifically, (s"s a areal a represents the predicted user's mental reward after she takes action a while r" r"(ss, a |Greal = aP) represents the counterfactual mental reward she had taken a negative action N Intuitively, an user should receive a relatively high reward for the action she actually taken (a high r* areat (dreat)). Yet, the experimental results show that a nog-negligible proportion of y (s, Great (reat) relatively low, which suggests that there mismatch between users' actions and their real preferences (recall the example in the abstract: user

---

might click on a news article because of its attractive headline, but end up feeling uncomfortable after reading it). These observations indirectly validate the effectiveness of the mental reward model.
5.4
Online A/B test on Industrial Platform (RQ4)
To further validate the effectiveness of MTRec, we deploy it on an industrial short-video recommendation platform, which has tens of millions of Daily Active Users (DAU). Short-video recommendation is typical sequential recommendation scenario, where the goal is to improve the user engagement with the platform. The baseline model a Deep Cross Network (DCN) Wang et al.) [2017). which is trained using binary labels indicating whether the users click on videos. However, intuitively, clicking on a video does not necessarily mean that the user is satisfied after watching the video. Therefore, we expect that MT'eec could help to improve the overall users' satisfaction on the recommended videos and hence improve their engagement.
We improve the DCN model by incorporating the align- ment loss, as illustrated in Equation TO We find that the averaged video viewing time stably improve by about 7% compared with the baseline model, which demonstrates that MTRec indeed improves the overall recommendation quality and leads to better user en- gagement. Note that MTRee quite industrial friendly because of two reasons: (1) the dataset used to train MTRec could be same with that used to train the recom- mendation model; (2) only need to add an auxiliary alignment loss to the original recommendation loss.
1000
665
stewinng tine (4)
v00
DCN
DCN-MTRee
503
Days
Figure 4: Online A/B test results.
Conclusions
The general goal of recommender systems is to satisfy users by providing items that align with their real preferences. However, existing works focus on optimizing surrogate objectives based on users' implicit feedback, ignoring that the implicit feedback may accurately reflect their real preferences. Consequently, recommendation models could be systematically biased. In this work, we aim to fill this gap by studying the users' behaviors and uncovering the distribution of their mental rewards. We propose a novel distributional IRI. algorithm to learn the mental reward model and use it to guide the training recommendation models. Finally, we validate the effectiveness of MTRec via both offline and online experiments, including the A/B on an industrial short video recommendation platform. In the end, believe that existing studies on user modeling far from totally understanding users.
Acknowledgements
This research is supported by Guangdong Basic and Applied Basic Research Foundation (202541))5 and the Fundamental Research Funds for the Central Universities (204ZZYXZZR069,
10

---

References
Ziebart Brian, Andrew Maas, Andrew Bagnell, and Anind Dey. Maximum entropy inverse rein- forcement learning. In Proceedings of the 23rd AAAI Conference Artificial Intelligence, pages 1433-1438, 2008.
Hong Chen, Yudong Chen, Xin Wang, Ruobing Xie, Rui Wang, Feng Xia, and Wenwu Zhu. Curricu- lum disentangled recommendation with noisy multi-feedback. In Advances in Neural Information Processing Systems, pages 26924-26936, 202
Minmin Chen, Alex Beutel, Paul Covington, Sagar Jain, Francois Belletti, and Ed H. Chi. Top-k off-policy correction for REINFORCE recommender system. In Proceedings of the Twelfth ACM International Conference on Web Search and Data Mining, pages 456-464, 2019.
Xiaocong Chen, Lina Yao, Aixin Sun, Xianzhi Wang. Xiwei Xu, and Liming Zhu. Generative inverse deep reinforcement learning for online recommendation. In Proceedings of the 30th ACM International Conference on Information and Knowledge Management, pages 201-210, 2021b.
Heng-Tze Cheng, Levent Koe, Jeremiah Harmsen, Tal Shaked, Tushar Chandra, Hrishi Aradhye, Glen Anderson, Greg Corrado, Wei Chai, Mustafa Ispir, et al. Wide & deep learning for recommender systems. In Proceedings of the 1st workshop on deep learning for recommender systems, pages 7-10, 2016.
Paul Covington, Jay Adams, Emre Sargin, and Mengehen Zhao. Deep neural networks for youtube recommendations. In Proceedings of the 10th ACM Conference on Recommender Systems, page 191-198, 2016.
Xinyi Dai, Jianghao Lin, Weinan Zhang, Shuai Li, Weiwen Liu, Ruiming Tang, Xiuqiang He, Jianye Hao, Jun Wang. and Yong Yu. An adversarial imitation click model for information retrieval, In Proceedings of the Web Conference, pages 1809-1820, 2021.
Ramachandran Deepak and Eyal Amir. Bayesian inverse reinforcement learning. In Proceedings of the 20th International Joint Conference on Artificial Intelligence, pages 2586 2586-2591, 2007.
Hanze Dong, Wei Xiong, Deepanshu Goyal, Rui Pan, Shizhe Diao, Jipeng Zhang, Kashun Shum, and Tong Zhang. Raft: Reward ranked finetuning for generative foundation model alignment, 2023.
Tom Fawcett. An introduction to roc analysis. Pattern recognition letters, 27(():861-877 2006.
Divyansh Garg, Shuvam Chakraborty, Chris Cundy, Jiaming Song, and Stefano Ermon. Iq-learn: Inverse soft-q learning for imitation. In Advances in Neural Information Processing Systems, pages 4028-4039, 2021.
Alexandre Gilotte, Clément Calauzènes, Thomas Nedelec, Alexandre Abraham, and Simon Dollé, Offline a/b testing for recommender systems. In Proceedings of the Eleventh ACM International Conference on Web Search and Data Mining, pages 206, 2018
Huifeng Guo, Ruiming Tang, Yunming Ye, Zhenguo Li, and Xiuqiang He. Deepfm: a factorization- machine based neural network for ctr prediction. In Proceedings of the 26th International Joint Conference on Artificial Intelligence, 2017
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforeement learning with stochastic actor. In International conference on machine learning. pages 1861-1870. PMLR, 2018.
Choi Jaedeug and Kee-Eung Kim. Nonparametric bayesian inverse reinforcement learning for multiple reward functions. In Advances in Neural Information Processing Systems, pages 305-313, 2012.
Chen Jiawei, Hande Dong, Xiang Wang, Fuli Feng, Meng Wang, and Xiangnan He. Bias and debias in recommender system: A survey and future directions. ACM Transactions on Information Systems, 41(3):1-39, 2023.
Ho Jonathan and Stefano Ermon. Generative adversarial imitation learning. In Advances Neural Information Processing Systems, 2016.

---

Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations, 2015.
Jialin Liu, Xinyan Su, Zeyu He, Xiangyu Zhao, and Jun Li. Adversarial batch inverse reinforcement learning: Learn to reward from imperfect demonstration for interactive recommendation. In arXiv preprint arXiv:2310. 19536, 2023a.
Langming Liu, Liu Cai, Chi Zhang, Xiangyu Zhao, Jingtong Gao, Wanyu Wang, Yifu Lv, Wenqi Fan, Yiqi Wang, Ming He, et al. Linrec: Linear attention mechanism for long-term sequential recommender systems. In Proceedings of the 46th International ACM SIGIR Conference on 2023b.
Research and Development in Information Retrieval, pages
Ziwei Liu, Qidong Liu, Yejing Wang, Wanyu Wang, Pengyue Jía, Maolin Wang, Zitao Liu, Yi Chang, and Xiangyu Zhao. Bidirectional gated mamba for sequential recommendation. arXiv preprint arXiv:2408. 1 1451,, 2024.
Julian McAuley, Christopher Targett, Qinfeng Shi, and Anton Van Den Hengel. Image-based recommendations on styles and substitutes. In Proceedings of the 38th international ACM SIGIR conference on research and development in information retrieval, pages 43-52, 2015.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Gray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback. In Advances in Neural Information Processing Systems, pages 27730-27744, 2022.
Yanru Qu, Han Cai, Kan Ren, Weinan Zhang, Yong Yu, Ying Wen, and Jun Wang. Product-based neural networks for user response prediction. In 2016 IEEE 16th international conference data mining (ICDM), pages 1149-1154, IEEE, 2016.
Arora Saurabh and Prashant Doshi. A survey of inverse reinforcement learning: Challenges, methods and progress. Artificial Intelligence, 297 2021.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy 2017.
optimization algorithms. arXiv preprint
Jing-Cheng Shi, Yang Yu, Qing Da, Shi- Yong Chen, and An-Xiang Zeng. Virtual-taobao: Virtualizing real-world online retail environment for reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 190224909 2019.
Adith Swaminathan and Thorsten Joachims. The normaiedd estimator for counterfactual learning. advances in neural information processing systems, 28, 2015.
Ruoxi Wang, Bin Fu, Gang Fu, and Mingliang Wang. Deep & cross network for ad click predictions. In Proceedings of the ADKDD' 17, 2017.
Wenjie Wang, Fuli Feng, Xiangnan He, Liqiang Nie, and Tat-Seng Chua. Denoising implicit feedback for recommendation. In Proceedings of the 14th ACM International Conference on Web Search and Data Mining, pages 373-381, 2021.
Kang Wang-Cheng and Julian McAuley. Self-attentive sequential recommendation. In Proceedings of the International Conference on Data Mining, pages 197-206, 2018.
Dabney Will, Mark Rowland, Mare Bellemare, and Rémi Munos. Distributional reinforcement learning with quantile regression. In Proceedings of the 33rd AAAI Conference on Artificial Intelligence, 2018
Jac woong Lee, Seongmin Park, Jongwuk Lee, and mengehen zhao. Dual unbiased recommender learning for implicit feedback. Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 1647-1651,
Jiaqi Yang, Xiang Li, Shuguang Han, Tao Zhuang, Dechuan Zhan, Xiaoyi Zeng, and Bin Tong. Cap- turing delayed feedback in conversion rate prediction via elapsed-time sampling. In Proceedings of the AAAI Conference Artificial Intelligence, pages 4582-4589, 2021.
12

---

Ren Yi, Hongyan Tang, Jiangpeng Rong, and Siwen Zhu. Unbiased pairwise learning from implicit feedback for recommender systems without biased variance control. In Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval, 2023.
pages
Wenhui Yu and Zheng Qin. Sampler design for implicit feedback data by noisy-label robust learning. In Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 861-870, 2020.
Guanjie Zheng, Fuzheng Zhang, Zihan Zheng, Yang Xiang, Nicholas Jing Yuan, Xing Xie, and Zhenhui Li. DRN: A deep reinforcement learning framework for news recommendation. In Proceedings of the 2018 world wide web conference, pages 167-176, 2018.
Guorui Zhou, Xiaoqiang Zhu, Chenru Song, Ying Fan, Han Zhu, Xiao Ma, Yanghui Yan, Junqi Jin, Han Li, and Kun Gai. Deep interest network for click-throuu rate prediction. In Proceedings of the 24th ACM SIGKDD international conference on knowledge discovery and data mining, pages 1059-1068, 2018.
Guorui Zhou, Na Mou, Ying Fan, Qi Pi, Weijie Bian, Chang Zhou, Xiaoqiang Zhu, and Kun Gai. Deep interest evolution network for click-through rate prediction. In Proceedings of the AAAI conference on artificial intelligence, volume 33, pages 5941-5948, 2019.
NeurIPS Paper Checklist
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Our main contribution is the MTRec framework with the QR-IQL algorithm, which is described in Section 4 and justified in Section 5
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer:
[Yes]
Justification:
The limitation discussed in Appendix A.5
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
13

---

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [Yes]
Justification:
The derivations details are provided in Appendix A
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification:
The implementation details are provided in Appendix A.3
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed
14

---

instructions for how to replicate the results, access to a hosted model (e.g., g. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Answer: [No]
Justification: We provide implementation details in Appendix A.3 and will release the code upon acceptance of this paper.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips... ec/
cablic/guides httgg//ddeoooeeouaaiiieeeseeeess (CodeSummissionooo coonoollcy)) for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https gww.ss///eee/eegee/ooo (wwdeSumiissioonoooo.
/
for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: All the data processing, training and evaluation details are described in Section Appendix[A]
Guidelines:
15

---

The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification:
We report the mean CTR with 95% confidence interval in Figure
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
The computational resources and training times are provided in Appendix A.3
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 'ttntt make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips. httprrip.....ubbblcchhccceeeceie
Answer: [Yes]
Justification: We respect the NeurIPS Code of Ethics.
16

---

Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: This work focuses on the technical foundations of sequential recommendation.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (c.g.. deploymeet of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
Our paper does not release data or models with high risk for misuse.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
17

---

Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: All the baselines are properly cited and introduced in Section
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, wwwpppppppsateeea thcode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license dataset.
For existing datasets that are re- re-packaged, both the original license and the license of the derived asset (if has changed) should be provided. If this information is available online, the authors are encouraged to reach out the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing or research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
Institutional review board (IRB) approvals or equivalent for research with human subjects
18

---

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Our paper does not involve crowdsourcing research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with
human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification:
The core method development in this research does not involve LLMs.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM policy (https: Dectpa://neurppeee heurips cc/ cc//onnfrrences/2225/////// for what should or should not be described.
A Technical Appendices and Supplementary Material
A.1 Derivations of Problem 6
In this section, we show that Problem 4 can be simplified and translated to Problem 6 under the assumption that (ss, a) follows quantile distribution Zx(s, a). Given Problem 4
max B, (aaepe/l@((0 (Q(, 7 (() - (1 7) Boo (V (so))
(4)
QEf
The second term be expanded
(1- (1=7)Bng~rroV"" (1
Σ Eng (8:)) (1 (71--) Σ Eng [V'(s)]
Ep V (s) (s)
(13)
Since we already have the expert trajectory dataset DE. we will use pg to estimate Equation
13
Then, on substituting (z) = t/
_x² and (*(s) log Σ. exp Q(s, a) in Problem 4
we have:
max Epx - log Σ exp Q(s.a))s))))))))) a)) - 4a
E (Q(s, a) - log Σ exp Q(s'
(14)
As Q(s,a) is parameterized by quantiles (Mr a) N replace Q with Qx and obtain Problem
19

---

A.2 Optimization steps for QR-IQL
In order to learn the distributional Q-function, we change the output layer of the Q-network to be of size |A] x N, where |A] denotes the size of the action space and N denotes the number of quantiles. For given action, each of the N heads implicitly correlates Xs. We adopt the widely used Pinball loss to learn the positions of {A:) which is defined as:
{ A A.u,
u 2 0
pa(u)
(2-
2 <<0
where u represents the error between the predicted value and the target value at quantile A. Intuitively, the Pinball loss pushes the quantile À to the right position so that the predicted value distribution matches the target distribution. Based on the objective of Problem we define two errors as: 6
uy (s,a) a) log Σ exp (s,
1 (2)(), log Σ Ax aa))² 4a
" (s,0,8)
where A; is the i-th quantile and (s,s,a) a) the corresponding head of the Qx network.. Complete training procedures are provided in Algorithm
Algorithm QR-IQL Optimization Steps
Input:
Interaction (expert) data Dg, number of quantiles
1: Initialize network Qx;
2: repeat
3: Sample (batched) data (s, a, s');
4: Compute errors 3 (s, a) and ux, (s, a, s') for each quantile in {A:)
5:
Compute the Pinball losses px, (u') and px, (u²) for each quantile in {M.)
6:
Compute the total loss as: Σ/= (ux + Px, ( (u²)
7:
Minimize the total loss by Adam Kingma and Ba [015];
8:
until convergence
Output: Q
A.3
Implementation Details
Our experiments run on server with 2xAMD EPYC 7542 32-Core Processor CPU and 2xNVIDIA RTX 3090 graphics. For the offline experiments on Amazon datasets, it takes about hours for 50,000 iterations of training with 4000 batch size. For online experiments Virtual Taobao, it takes about 4 hours for 50,000 RL training steps.
Algorithm 2 describes an overview of the implementation procedures. Basically, there are two stages. At stage we focus on learning Q3, and at stage 2 we focus on learning Fİ. practice, the architecture of the recommendation model F'c could be of various types. For example, in Section
5.1 we test seven widely used recommendation models: Wide&Deep Cheng et al. [2016] PNN Qu
al. [0166] DeepFM Guo al. [2017]. DIN Zhou al. 22018) Zhou Zhou al. 2019 LinRec
Liu et al. (2023b and SIGMA Liu et al. (2024). We will use the same network architecture of Fş
to construct Qx (except the output layer) to ensure that the features are processed properly. All the
hyper-parameter of the backbone models follow their official codes. For implementation of MTRec,
we select the number of quantiles N 10 and the weight a 0.5 in Problem 6
Actually, our MT'ee framework is industrial friendly due to the following merits. First, all the training procedures are run in an offline manner, saving the cost of building online user-system interaction environments. Second, since QA shares the model architecture with Fc, we do not need to build mental reward model from scratch, saving lot of work for adaptation.
A.4 Details the Amazon datasets
The Amazon dataset McAuley et al. (0155) collects user review data from amazon.com. The crawled reviews have time span from May 1996 to July 2014. The dataset can be divided into many subsets
20

---

Algorithm 2 Overall Implementation of MTRec
Input:
Interaction (expert) data DE
1: Initialize networks Qx and Fç:
2: Learn Q3 according to Algorithm
3: Obtain the mental rewards by
r(s, a) < (x, a) - -s'','',, Vs, ~~ DE:
4: Patch the mental rewards De:
5: Learn F according to Equation 10 or Equation 12
Output: Aligned Recommendation Model F
according the various product categories. To verify the effectiveness of MTRee, utilized two subsets of the Amazon dataset: Books and Electronics. We treated the reviews as user behaviors and sorted the reviews from each user chronologically. Based user's historical behaviors, our goal was to predict whether the user would write a review.
Dataset
User
Item
Categoriy
Sample
Books
603,668
367,982 63,001
1,600
603,668
Electronics
192,403
801
192,403
Table The statistics of the Amazon datasets.
A.5 Limitations
While our work opens a door for studying users' intrinsic rewards during their interaction with the recommender systems, it still lacks systematic method thoroughly evaluate the learned mental reward model. In future works, we plan to construct comprehensive benchmark involving large-scale human studies to further evaluate the mental reward model.
21


Reviews:
[{"review_id": "2wL6U5igur", "reviewer": "Reviewer_pp7H", "summary": "This paper proposes MTRec, a sequential recommendation framework that introduces a mental reward model to better align recommendations with users’ real preferences. The mental reward is defined as a latent signal reflecting internal user satisfaction, and is learned from behavioral data via a distributional inverse reinforcement learning method called Quantile Regression Inverse Q-learning (QR-IQL). The learned reward is then used to provide auxiliary supervision for training recommendation models. Experiments are conducted on public datasets, a simulated environment (Virtual Taobao), and an industrial short-video platform, showing improvements in offline metrics and online engagement.", "questions": "Q1. The paper assumes that users are maximizing their cumulative “mental rewards,” but this concept is treated as a latent variable without clear grounding. How sensitive is the proposed method to this assumption, and is there any empirical evidence or user-behavior analysis to support it?\n\nQ2. How does MTRec conceptually and practically differ from existing reinforcement learning-based recommendation frameworks? Could the authors clarify the complementary or novel aspects compared to those methods?\n\nQ3. The mental reward model is used as an auxiliary supervision signal, but its interpretability remains unclear. Have the authors considered any methods to analyze or visualize what the model has learned?", "limitations": "yes", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths\n\nS1. The paper addresses a core challenge in recommendation — the discrepancy between implicit feedback and true user preferences — and proposes a novel approach that attempts to model latent user satisfaction signals directly.\n\nS2. The proposed QR-IQL method offers a technically interesting contribution by capturing the stochastic nature of user feedback through a distributional IRL formulation, extending prior work on inverse Q-learning.\n\nWeaknesses\n\nW1. The concept of mental reward is central but underdefined. The paper lacks theoretical grounding or empirical validation to support that the learned reward meaningfully reflects user satisfaction.\n\nW2. The learned reward model lacks interpretability and analysis. There is limited insight into what the model captures, how it varies across users or items, or whether it corresponds to known signals of user preference.\n\nW3. The experimental comparison omits several recent and relevant RL-based recommendation models that similarly aim to optimize long-term user satisfaction or retention, such as Decision Transformer-based frameworks or offline RL approaches. Without comparing to these baselines (e.g., [1-4]), it is difficult to assess whether the proposed method offers a significant advantage over the current state of the art.\n\nRef:\n\n[1] Sequential recommendation for optimizing both immediate feedback and long-term retention, SIGIR 2024.\n\n[2] User retention-oriented recommendation with decision transformer, WWW 2023.\n\n[3] Maximum-entropy regularized decision transformer with reward relabelling for dynamic recommendation, KDD 2024.\n\n[4] Causal decision transformer for recommender systems via offline reinforcement learning, SIGIR 2023.", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your helpful suggestions to our work. Please see our responses below.\n\n**Q1: On the definition of the mental reward.**\n\n**A1:** Thanks for raising this question. In the third paragraph of the Introduction, we define the user's mental reward as \"the summary of her private feelings about how she is satisfied\". We also use multiple examples to illustrate this concept. However, we agree with you that we did not provide a very specific definition of the mental reward, because it is essentially **a highly abstract concept and can be interpreted in different ways depending on the scenario contexts**. For example, a user might click on a news article because of its attractive headline, but end up feeling uncomfortable after reading the content. Here, the mental reward is a reflection and quantification of how the user feels uncomfortable. \n\nOn the other hand, despite its definition, the mental reward **does exist**, and it lies between user's implicit feedback (e.g., click) and her real preferences. Our work aims to narrow this gap by using the mental reward as an effective tool (and the experimental results demonstrate this). In summary, our work makes an important step towards understanding the gap between users' implicit feedback and their real preference. We contribute a novel and theoretically solid algorithm to uncover the mental rewards and demonstrate its effectiveness by extensive experimental results. \n\n**Q2: How sensitive is the proposed method to the assumption that users are maximizing their cumulative mental rewards?**\n\n**A2:** This is an insightful question. Actually, this assumption is the basic motivation of using inverse RL, where we aim to seek a reward function that rationalize users' behaviors. If the users are not maximizing their mental rewards, which means that learned mental rewards are **noises**, they should not bring improvements to the recommendation models. However, our experiments (including offline, online, real-world deployment) demonstrate the effectiveness of the learned mental rewards, which in turn **justifies the correctness of this assumption**.  \n\n**Q3: How does MTRec conceptually and practically differ from existing reinforcement learning-based recommendation frameworks?**\n\n**A3:** **MTRec is actually a complementary to RL-based methods.** RL-based methods heavily rely on accurate rewards to learn good policies, yet existing approaches simply use user implicit feedback (e.g., click) as rewards. Due to the gap between the implicit feedback and their true preferences, the implicit feedback may not be the optimal reward signal. MTRec makes a non-trivial correction to the reward signals, which significantly improves the performance of RL-based methods. Please refer to Section 5.2 for more details. \n\n**Q4: Have the authors considered any methods to analyze or visualize what the model has learned?**\n\n**A4:** If we understand corretly, this question is related to Q1. Perhaps answer A1 could clarify some of the concerns. In a word, the mental reward model learns to summarize and differentiate user's feelings after she takes an action. For example, if a user clicks and reads a piece of news, how much does she like it actually? In Section 5.3, we visualize some analysis on the mental reward model. We can see that the mental rewards actually correlated with users' behaviors, which also aligns with our intuitions. Although we don't know exactly how the learned mental reward correspond to users' feelings (e.g., like, neutral, dislike, angry, favorite), it indeed provides useful information to the recommender systems and helps to better understand the users.\n\n**Q5: Comparisons to Decision Transformer based methods.**\n\n**A5:** Thanks for raising this question. In the paper we consider online RL setting and test MTRec on PPO and SAC. As the reviewer mentioned, it is also interesting to test MTRec on offline RL methods such as Decision Transformers. We use the dataset containing 100,000 user-RS interaction trajectories, as described in Section 5.2, to train a DT model (including 2 Transformer layers, 8 heads, and an embedding size of 128) for 5 epochs, and test it the same way with PPO and SAC in the online Virtual Taobao environment. The comparative results are shown below. \n\n\n| Model      | Averaged CTR |\n| :---        |    :----:   |\n|   PPO  | 0.5435       | \n| PPO-MTRec   | 0.6782       |\n|   SAC  | 0.7055       | \n| SAC-MTRec   | 0.909       |\n|   DT  | 0.4816      | \n| DT-MTRec   | 0.5472       |\n\nIt can be observed that the original DT performs worst across all models, since DT in trained in offline setting while PPO and SAC are trained in online setting. Therefore, it is not a really fair comparison. However, we can see that DT-MTRec still improves DT, which demonstrates that MTRec indeed brings more useful information than simply treating click as reward signal.  \n\nNote that all of the four references mentioned by the reviewer are based on DT, therefore we only test the original DT and combine it with MTRec. In the next version of our paper, we will discuss more about the differences between our work and existing RL-based methods, including but not limit to the references mentioned by the reivewer.\n\nThanks again for your time spent on our paper. We would be very grateful if you could re-evaluate our work based on our rebuttal."}, {"review_id": "aTIc2HFrRx", "reviewer": "Reviewer_7bFs", "summary": "This paper proposes a novel framework designed to align recommendation models with users' intrinsic preferences. The key innovation involves the construction of a mental reward model that quantitatively captures user satisfaction levels. To effectively learn this model, it introduces a distributional inverse reinforcement learning methodology. The optimized mental reward model subsequently serves as a guiding mechanism for training the recommendation system. Extensive experiments are conducted to verify the effectiveness of the proposed method.", "questions": "1. Regarding Appendix A.3 setting the number of quantiles N=10, it does not explain why 10 quantiles are chosen instead of 5 or 20 (e.g., the balance of probability interval coverage). Additionally, the logic behind the non-uniform distribution of quantile levels (e.g., 0.2, 0.3, ..., 0.9) is not clarified (e.g., the theoretical basis for dense sampling in medium-high probability intervals). It is recommended to supplement comparative experiments with N=5 or 20 to demonstrate robustness.\n\n2. Concerning the engineering details of industrial deployment, the online experiment mentions integrating the alignment loss with the DCN model but does not specify the real-time calculation latency of mental rewards (e.g., whether it affects the recommendation response time).\n\n3.  The evaluation section indirectly validates the model through counterfactual mental rewards, but lacks a correlation analysis (e.g., Pearson coefficient analysis) between reward values and explicit user metrics (such as viewing duration and revisit rate). This may weaken the core assumption that \"rewards reflect real preferences.\"", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 4, "strengths_and_weaknesses": "Strengths\n1. The Idea of this paper is interesting and novel. Departing from conventional recommendation paradigms that treat implicit feedback (e.g., clicks) as deterministic preferences, this work proposes modeling user behavior as \"a stochastic process of mental reward maximization\".\n2. To address the limitation of conventional inverse reinforcement learning (IRL) methods in capturing reward distributions, this work introduces the Distributional Inverse Reinforcement Learning algorithm (QR-IQL).\n3. Comprehensive experiments and analysis are provided, including offline and online experiments, as well as testing on a large-scale industrial recommendation platform.\n\nWeakness:\nit still lacks a systematic method to thoroughly evaluate the learned mental reward model.", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your appreciation and helpful suggestions to our work. Please see our responses below.\n\n**Q1: Why the number of quantiles N=10 is chosen instead of 5 or 20?**\n\n**A1:** Thanks for raising this question. Intuitively, a larger N (more quantiles) provides more flexibility to model the underlying distribution of $Q_\\lambda$, however, it may also bring overfitting issues. We agree that searching for the optimal N is necessary. Therefore, we add a new set of experiments on the Electronic dataset over three models, as shown below. \n\n|    Model    | AUC (N=5)| AUC (N=10)| AUC (N=15)| AUC (N=20)| NCIS (N=5)| NCIS (N=10)| NCIS (N=15)| NCIS (N=20)|\n| :----------- | :-----------: | :-----------: | :-----------: | :-----------: | :-----------: | :-----------: | :-----------: | :-----------: |\n| DeepFM-MTRec      | 0.8462      |**0.8468**    | 0.8467       | 0.8463      | 0.8960      |0.8961      | 0.8960       |**0.8975**      |\n|DIN-MTRec               | 0.8544      |0.8542     | 0.8544      | **0.8547**      | 0.8722      | 0.8728     | **0.8733**       |0.8731      | \n| LinRec-MTRec         | 0.8593      |0.8594     | **0.8597**      | 0.8589      | 0.9773      |0.9782      | 0.9780       |**0.9785**      | \n\nFrom the above table, we can see that different choices of N do not result in significant changes in terms of AUC and NCIS.  In addition, N=5 performs the worst, perhaps due to limited flexibility to represent the underlying quantile distribution.  \n\nWe also test different choices of N on Virtual Taobal over two RL-based methods. The results are reported below. \n\n|    Model    | Averaged CTR (N=5)| Averaged CTR (N=10)| Averaged CTR (N=15)| Averaged CTR (N=20)| \n| :----------- | :-----------: | :-----------: | :-----------: | :-----------: |\n| PPO-MTRec      | 0.665      |0.678   | 0.686       | **0.688**      | \n|SAC-MTRec               | 0.885      |0.909     | 0.915      | **0.917**      | \n\nSurprisingly, we obverve a clear trend of improving performance, with respect to N, of both PPO and SAC. These results show that a larger number of quantiles helps to learn more accurate mental reward models. Note that this trend is not that obvious in supervised learning models (refer to the first table). We hypothesis that it is because RL-based methods are more sensitive to rewards.\n\n\n**Q2: Whether the calculation of mental rewards affects the recommendation response time?**\n\n**A2:** In fact, according to Algorithm 2 in Appendix A.3, we only calculate the mental rewards during training. There is no extra computation that might increase the response time during testing. We are sorry for the misunderstanding caused and will state more clearly in the paper. \n\n**Q3: Whether the calculation of mental rewards affects the recommendation response time?**\n\n**A3:** Thanks for your insightful suggestion! Since there is no direct way to evaluate the mental reward model, computing correlations is an effective approach. In the first figure of Figure 3, we report the correlation between users' mental rewards and the lengths of their interaction with the recommender system. We can see that users' mental rewards gradually decrease with the number of interaction steps increases. This aligns with the intuition that people might lose interest and get tired after consuming enough recommendation contents. \n\nThanks again for reviewing our paper, we hope that the above rebuttal can address your concerns."}, {"review_id": "jGx7DFRMA1", "reviewer": "Reviewer_bFhU", "summary": "The paper address the critical issue that implicit feedback often misrepresents user satisfaction. Then, the authors proposes MTRec, a sequential recommendation framework that leverages a mental reward model learned via distributional inverse reinforcement learning (IRL) to align recommendation models with users’ real preferences. Experiments on public datasets and an industrial platform demonstrate improvements.", "questions": "None", "limitations": "Light", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n* Novel Problem Formulation: Addresses the fundamental mismatch between implicit feedback and user satisfaction, a long-standing issue in recommendation systems.\n* Technical Innovation: The proposed QR-IQL algorithm, a distributional variant of IRL, effectively captures the stochasticity of user satisfaction—a significant advancement over deterministic reward models.\n* Practical Impact: The industrial deployment of MTRec in  real-world and achieve good improvment. \n\nWeaknesses:\n* Mental Reward Interpretation: While the paper argues that mental rewards reflect user satisfaction, the model’s direct interpretability is limited. The absence of ground-truth mental reward labels necessitates indirect validation (e.g., correlation with user actions), which leaves open questions about how accurately the model captures true preferences.\n* Oversimplified Modeling of User Feedback: The framework assumes user mental rewards can be modeled via distributional IRL, but real-world feedback exhibits fragmentation and uncertainty that may limit model accuracy.(For instance, a user might skip a video due to network issues or just  bad mood).\n* Inadequate Experiments: The paper only gives main experiments and online experiments, but lack of ablation experiments to prove their performance of method. (for example: Hyperparameter Selection(different alignment loss weights and number of quantiles N), quantile regression vs. distributional rewards , Computational cost....)", "was_revised": true, "final_justification": "The difficulty of achieving a well-defined mental reward remains and the effort on the extended ablation in the rebuttal partially addressed my concern. Though the concerns in the discussion are only partially addressed in my view and there is still room to improve the convincingness of the paper, I believe the current revised state of the work is reaching the bar of NeurIPS so I am raising the rating to 4.", "rebuttal": "Thanks for your valuable suggestions to our work. Please see our responses below.\n\n**Q1: The model’s direct interpretation on the mental reward is limited.**\n\n**A1:** Thanks for raising this question. In the third paragraph of the Introduction, we define the user's mental reward as \"the summary of her private feelings about how she is satisfied\". We also use multiple examples to illustrate this concept. However, we agree with you that we did not provide a very specific definition of the mental reward, because it is essentially **a highly abstract concept and can be interpreted in different ways depending on the scenario contexts**. For example, a user might click on a news article because of its attractive headline, but end up feeling uncomfortable after reading the content. Here, the mental reward is a reflection and quantification of how the user feels uncomfortable. \n\nOn the other hand, despite its definition, the mental reward **does exist**, and it lies between user's implicit feedback (e.g., click) and her real preferences. Our work aims to narrow this gap by using the mental reward as an effective tool (and the experimental results demonstrate this). In summary, our work makes an important step towards understanding the gap between users' implicit feedback and their real preference. We contribute a novel and theoretically solid algorithm to uncover the mental rewards and demonstrate its effectiveness by extensive experimental results. \n\nIn addition, we agree with the reviewer that evaluation about how accurately the model captures true preferences is an open question. In our paper, we compute the **correlation between the mental rewards and the lengths of user-RS interaction trajectories**. We also **visualize the distribution of the mental rewards conditioned on the users’ real actions** (please refer to Section 5.3). We believe that these results are instructive in answering the above question. \n\n**Q2: Oversimplified Modeling of User Feedback.**\n\n**A2:** Thanks for mentioning this issue. Real-world recommender systems may involve many types of user feedback, depending on specific application scenarios. For research works, we have to make appropriate simplification. In fact, we follow existing literature (see related works and Section 2 and 5) and classify user feedback as positive and negative, in order to cover as many scenarios as possible. However, we aggree that extending MTRec to include more diverse user feedback is an interesting future direction. \n\nIn addition, the reviewer mentioned that the uncertainty in user feedback may influence the accuracy of the mental reward model. This is true. Actually, such uncertainty affects almost all the recommendation models, because **user feedback is random variable by its nature**. To mitigate this issue, we model the mental reward also as random variable, which motivates the design of our algorithm QR-IQL (see Secion 4.2).  QR-IQL addresses the uncertainty in the mental rewards and achieves better performace than conventional IRL algorithm (please refer to Section 5.1).\n\n**Q3: More ablation experiments.**\n\n**A3:** We agree that including more experimental results would strengthen our claims. Therefore, we add three new set of experiments to test different choices of hyper-parameters. The results are shown below. \n\n**Exp1: Ablation on different choices of N (number of quantiles)**. Intuitively, a larger N (more quantiles) provides more flexibility to model the underlying distribution of $Q_\\lambda$, however, it may also bring overfitting issues. We agree that searching for the optimal N is necessary. Therefore, we add a new set of experiments on the Electronic dataset over three models, as shown below. \n\n|    Model    | AUC (N=5)| AUC (N=10)| AUC (N=15)| AUC (N=20)| NCIS (N=5)| NCIS (N=10)| NCIS (N=15)| NCIS (N=20)|\n| :----------- | :-----------: | :-----------: | :-----------: | :-----------: | :-----------: | :-----------: | :-----------: | :-----------: |\n| DeepFM-MTRec      | 0.8462      |**0.8468**    | 0.8467       | 0.8463      | 0.8960      |0.8961      | 0.8960       |**0.8975**      |\n|DIN-MTRec               | 0.8544      |0.8542     | 0.8544      | **0.8547**      | 0.8722      | 0.8728     | **0.8733**       |0.8731      | \n| LinRec-MTRec         | 0.8593      |0.8594     | **0.8597**      | 0.8589      | 0.9773      |0.9782      | 0.9780       |**0.9785**      | \n\nFrom the above table, we can see that different choices of N do not result in significant changes in terms of AUC and NCIS.  In addition, N=5 performs the worst, perhaps due to limited flexibility to represent the underlying quantile distribution.  \n\nWe also test different choices of N on Virtual Taobal over two RL-based methods. The results are reported below. \n\n|    Model    | Averaged CTR (N=5)| Averaged CTR (N=10)| Averaged CTR (N=15)| Averaged CTR (N=20)| \n| :----------- | :-----------: | :-----------: | :-----------: | :-----------: |\n| PPO-MTRec      | 0.665      |0.678   | 0.686       | **0.688**      | \n|SAC-MTRec               | 0.885      |0.909     | 0.915      | **0.917**      | \n\nSurprisingly, we obverve a clear trend of improving performance, with respect to N, of both PPO and SAC. These results show that a larger number of quantiles helps to learn more accurate mental reward models. Note that this trend is not that obvious in supervised learning models (refer to the first table). We hypothesis that it is because RL-based methods are more sensitive to rewards.\n\n**Exp2: Ablation on different choices alignment loss weights $\\kappa$**. As RL-based methods are more sensitive to rewards, we test 7 different choices of $\\kappa$ under the PPO and SAC experiments. The results are shown below. \n|    Model    | Averaged CTR ($\\kappa$=0.1)| Averaged CTR ($\\kappa$=0.2)| Averaged CTR($\\kappa$=0.3)| Averaged CTR ($\\kappa$=0.4)| Averaged CTR ($\\kappa$=0.5)| Averaged CTR ($\\kappa$=0.6)| Averaged CTR ($\\kappa$=0.7)| \n| :----------- | :-----------: | :-----------: | :-----------: | :-----------: |:-----------: | :-----------: | :-----------: |\n| PPO-MTRec      | 0.698      |0.678   | 0.706         | **0.718**      | 0.703   | 0.693       | 0.704     | \n|SAC-MTRec        | 0.892      |0.909     | **0.927**   | 0.917      | 0.904   | 0.915       | 0.896      | \n\nWe can see that the optimal choices of $\\kappa$ are 0.4 and 0.3 respectively. We can see that increasing $\\kappa$ does not always bring improvement. This indicates that the mental reward is better used as regularizer, instead of the main source of the reward. We hypothesis that the mental rewards, while provide useful supervision information, may also bring inductive bias.\n\n**Exp3: Computational costs**.\nOur experiments are run on a server with 2×AMD EPYC 7542 32-Core Processor CPU and 2×NVIDIA RTX 3090 graphics. For the offline experiments on Amazon datasets, it takes about 3 hours for 50,000 iterations of training with a 4000 batch size. For online experiments on Virtual Taobao, it takes about 4 hours for 50,000 RL training steps.\n\nFurthermore, although training MTRec's additional mental reward model takes several hours, this cost is worthwhile since recommender systems are primarily concerned with inference latency, and MTRec adds no computation during inference.\n\nThanks again for your time spent on our paper. We would be very grateful if you could re-evaluate our work based on our rebuttal."}, {"review_id": "pvRutiLSOb", "reviewer": "Reviewer_nSej", "summary": "This paper introduces a mechanism to align recommendation systems with user preferences by modeling a user as an agent with their own mental reward function, which is used to provide a learning signal to two classes of recommendation systems: (1) classification models and (2) RL agents. The authors introduce an algorithm to learn mental reward function using a novel quantile-based inverse RL to learn a quantile Q function, which is then used to produce the reward value. Their evaluations included performance on standard recommendation system benchmarks, and end-to-end performance from the real video recommendation system. The results suggest improvement over the baseline methods.", "questions": "Thank you for your work on this paper. I enjoyed reading it.\n\n* I’d suggest including more discussion about mental reward function, and its connection to other sciences. This can help validate this concept and make the result stronger.\n* I’d suggest addressing minor comments:\n  * Adding standard deviation to Figure 3A\n  * Adding missing citations (see my comment in Strengths and Weaknesses)", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "# Quality\n\n* (+) A nice paper that provides mathematical rigor with practical algorithms.\n* (+) Good discussions about interpretability of the mental reward function.\n\n# Clarity\n* (+) Clear description of research questions and their evaluation.\n* (+) Math derivations are clear to follow.\n* (-) Missing citations for some claims made in the paper.\n  * “User behaviors have been extensively studied in the literature of recommender systems”\n* (-) Figure 3a doesn’t include the standard deviation. It would be useful to see how noisy the measurements are for each step.\n\n# Significance\n\n* (+) This work provides practical algorithms to improve recommendation systems. The quantile IRL algorithm learns from offline data, which makes it easier to adopt. \n\n# Originality\n\n* (+) Authors model users as RL agents, where the reward function is modeled as a stochastic function, which previous approaches didn’t do (prev. used are deterministic reward functions).\n* (-) It wasn’t clear to me what inspired the mental reward function. It seems like it can be related to some concepts in psychology (i.e. dopamine). I’d like to see some discussion and potential connections to other sciences.", "was_revised": true, "final_justification": "This was a strong paper with novel contributions about mental reward model. I described its strength and weaknesses in my original review. Authors clarified connection of mental reward models to psychology, especially the Likert evaluation methods. This clarified the novelty of this work, which is why I raised my score.", "rebuttal": "Thanks for your appreciation and useful suggestions to our work. Please see our responses below.\n\n**Q1: How does the concept of mental reward relate to other sciences?** \n\n**A1:** Thanks for raising this interesting question. To be honest, we don't have a clear answer for now. But we would like to share some of our thoughts on this. First, our study is definitely related to psychology, and perhaps biology. In psychology, if we want to evaluate a person's preference on an item, we usually ask a set of questions from multiple perspectives (e.g., Likert Scale) and build some models (e.g., regression) to analyze the person's feedback. However, in the recommendation process, we don't have the chance to ask user many questions about their feelings on an item, therefore the aforementioned psychology models may not work here. But perhaps there are other psychology models that focus on estimating user's preferences from very limited feedback. We will look into this in future works. \n\n\n**Q2: Adding standard deviation to Figure 3A.** \n\n**A2:** This is a very helpful suggestion! Unfortunately, we can't upload figures in the rebuttal phase according to NeurIPS's policy.  We will update the figure in the next version of our paper. \n\n\n**Q3: Adding more citations on user behavior modeling in recommender systems.** \n\n**A3:** Sure, we will add those citations and check other parts of the paper. Thanks for your advice!\n\nThanks again for your time spent on our paper. We would be very grateful if you could re-evaluate our work based on our rebuttal."}]

Author rebuttal:

Meta-review: This paper proposes MTRec, a sequential recommendation framework that introduces the concept of a “mental reward” to bridge the gap between implicit feedback and true user preferences (explicit feedback). The method leverages a distributional inverse reinforcement learning approach (QR-IQL) to learn mental reward models and demonstrates their utility in recommender systems. Reviewers acknowledged the novelty of the problem formulation, the technical soundness of the algorithm, and the comprehensive evaluations across public datasets, simulated environments, and an industrial platform. The primary concerns centered on the vague definition of “mental reward,” limited interpretability, and the need for stronger comparisons and ablation analyses. The rebuttal and discussion phases helped alleviate these issues, as the authors clarified the interpretation of mental reward, added ablations (quantiles, loss weights, and computational cost), and extended comparisons with Decision Transformer baselines. Given the paper’s strong empirical results, clear algorithmic innovations, and demonstrated practical impact, I recommend acceptance.
Decision: Accept (poster)