Personalized Reward Learning with Interaction-Grounded Learning (IGL)
Authors: Jessica Maghakian
Paul Mineiro
Kishan Panaganti
Mark Rucker
Akanksha Saran
Cheng Tan
Venue: ICLR 2023 poster (2023)
Keywords: interaction-grounded learning
recommendation systems
interactive machine learning
contextual bandits

Abstract: In an era of countless content offerings, recommender systems alleviate information overload by providing users with personalized content suggestions. Due to the scarcity of explicit user feedback, modern recommender systems typically optimize for the same fixed combination of implicit feedback signals across all users. However, this approach disregards a growing body of work highlighting that (i) implicit signals can be used by users in diverse ways, signaling anything from satisfaction to active dislike, and (ii) different users communicate preferences in different ways. We propose applying the recent Interaction Grounded Learning (IGL) paradigm to address the challenge of learning representations of diverse user communication modalities. Rather than requiring a fixed, human-designed reward function, IGL is able to learn personalized reward functions for different users and then optimize directly for the latent user satisfaction. We demonstrate the success of IGL with experiments using simulations as well as with real-world production traces.


Published as a conference paper at ICLR 2023
PERSONALIZED REWARD LEARNING WITH INTERACTION-GROUNDED LEARNING (IGL)
Jessica Maghakian
Paul Mineiro
Stony Brook University
Microsoft Research NYC
jessica wwggaaaaaeeettttorrokooooko
Kishan Panaganti Texas A&M University kpb@tamu, edu
Mark Rucker
University of Virginia
mrZan@virginia.. edu
Akanksha Saran
Cheng Tan
Microsoft Research NYC akanksha www.neaiiraoooeeeeeeeeee com
Microsoft Research NYC
tan. www.nhheennnnnoooccccc cnlengemicroooft.coom
ABSTRACT
In an era of countless content offerings, recommender systems alleviate informa- tion overload by providing users with personalized content suggestions. Due to the scarcity of explicit user feedback. modern recommender systems typically op- timize for the same fixed combination of implicit feedback signals across all users. However, this approach disregards a growing body of work highlighting that (i) implicit signals can be used by users in diverse ways, signaling anything from satisfaction to active dislike, and (ii) different users communicate preferences different ways. We propose applying the recent Interaction Grounded Learning (IGL) paradigm to address the challenge of learning representations of diverse communication modalities. Rather than requiring fixed, human-designed reward function, IGL is able to learn personalized reward functions for different users and then optimize directly for the latent user satisfaction. We demonstrate the success of IGL with experiments using simulations as well as with real-world production traces.
INTRODUCTION
From shopping to reading the news, modern Internet users have access an overwhelming amount of content and choices from online services. Recommender systems offer way to improve user experience and decrease information overload by providing customized selection of content. A key challenge for recommender systems the rarity of explicit user feedback, such as ratings or more readily available implicit signals, such as clicks (Hu et al.] (008), webpage dwell time (Yi et al. 2014), or inter-arrival times (Wu al. 2017) as a proxy signal for satisfaction. These implicit signals are used as the reward objective in recommender systems, with the popular Click-Through Rate (CTR) metric as the gold standard for the field (Silveira al. 2019). However, directly using implicit signals as the reward function presents several issues.
likes/dislikes (Grëar et al.] 2005). Rather than explicit feedback, practitioners typically
Implicit signals do not directly map to user satisfaction. Although clicks are routinely equated with user satisfaction, there are examples of unsatisfied users interacting with content via clicks. Clickbait exploits cognitive biases such as caption bias (Hofmann et al., 2012) or the curiosity gap (Scot) 2021) so that low quality content attracts more clicks. Direct optimization of the CTR degrades user experience by promoting clickbait items (Wang et al.) 2021). Recent work shows that users will even click on content that they know priori they will dislike. In a study of online news reading, Lu et al. (2018a) discovered that 15% of the time, users would click on articles that they strongly disliked. Similarly, although longer webpage dwell times associated with satisfied users, study by Kim et al. (2014) found that dwell time is also significantly impacted by page topic, readability and content length.

---

Published as a conference paper at ICLR 2023
Different users communicate in different ways. Demographic background is known to have im- pact on the ways in which users engage with recommender systems. A study by Beel et al. (2013) shows that older users have CTR more than 3x higher than their younger counterparts. Gender also has an impact on interactions, e.g. men are more likely to leave dislikes on YouTube videos than women (Khan 2017). At the same time, growing body of work shows that recommender sys- tems do not provide consistent performance across demographic subgroups. For example, multiple studies on ML fairness in recommender systems show that women on average receive less accurate recommendations compared to men (Ekstrand et al. 2018 Mansoury et al., 2020). Current systems are also unfair across different age brackets, with statistically significant recommendation utility degradation as the age of the user increases (Neophytou et al.) (022). The work of Neophytou et al. identifies usage features as the most predictive mean recommender utility, hinting that the inconsistent performance in recommendation algorithms across subgroups arises from differences in how users interact with the recommender system.
These challenges motivate the need for personalized reward functions. However, extensively mod- eling the ways in which implicit signals are used or how demographics impact interaction style is costly and inefficient. Current offte-of-the-arr systems utilize reward functions that manually engineered combinations of implicit signals, typically refined through laborious trial and error meth- ods. Yet recommender systems and their users evolve, do the ways in which users implicitly communicate preferences, Any extensive models or hand tuned reward functions developed now could easily become obsolete within few years time.
To this end, we propose Interaction Grounded Learning (IGL. Xie et al. (2021) for personalized reward learning (IGL-P). IGL is learning paradigm where learner optimizes for unobservable rewards by interacting with the environment and associating observable feedback with the true la- tent reward. Prior IGL approaches assume the feedback either depends on the reward alone (Xie) et al. 2021), or on the reward and action (Xie et al. 2022). These methods are unable to disam- biguate personalized feedback that depends on the context. Other approaches such as reinforcement learning and traditional contextual bandits suffer from the choice of reward function. However our proposed personalized IGI., TGL-P, resolves the above challenges while making minimal as- sumptions about the value of observed user feedback. Our new approach is able to incorporate both explicit and implicit signals, leverage ambiguous user feedback and adapt to the different ways in which users interact with the system.
Our Contributions: We present IGL-P, the first IGL strategy for context-dependent feedback, the first use of inverse kinematics an IGL. objective, and the first IGL strategy for more than two latent states. Our proposed approach provides an alternative to agent learning methods which require hand- crafted reward functions. Using simulations and real production data, we demonstrate that IGL-P is able to learn personalized rewards when applied to the domain of online recommender systems, which require at least reward states.
2
PROBLEM SETTING
2.1
CONTEXTUAL BANDITS
The contextual bandit (Auer et al. 2002 Langford & Zhang 2007) is a statistical model of myopic decision making which is pervasively applied in recommendation systems (Bouneffouf et al.) (2220). IGL operates via reduction to contextual bandits, hence, we briefly review contextual bandits here.
The contextual bandit problem proceeds over T rounds. At each round t E [T]. the learner receives a context 2, E X' (the context space), selects an action a, E A (the action space), and then observes reward (ax), where rt A + [0, 1] is the underlying reward function. We assume that for each round t. conditioned on 2t, ri is sampled from a distribution Pr,, x1). A contextual bandit (CB) algorithm attempts to minimize the cumulative regret
Regce (T) = Era(T - (((())
(1)
1=1
relative to optimal policy nt over a policy class II.
In general, both the contexts 21...., FT and the distributions Fr....., Pyy can be selected in an arbitrary, potentially adaptive fashion based on the history. In the sequel we will describe IGL in a stochastic environment, but the reduction induces a nonstationary contextual bandit problem, and therefore the existence of adversarial contextual bandit algorithms is relevant.
2

---

Published as a conference paper at ICLR 2023
like this!
y
NEMS
NEMS
NEMS
NEMS
NEWS
recommeader system
Figure 1:
IGL in the recommender system setting. The learner observes the context x, plays an action a, and then observes a feedback y (that is dependent on the latent reward r), but not itself.
2.2 INTERACTION GROUNDED LEARNING
IGL is a problem setting in which the learner's goal is to optimally interact with the environment with no explicit reward to ground its policies. IGL extends the contextual bandit framework by
eliding the reward from the learning algorithm and providing feedback instead (Xie et al.) 2021). We describe the stochastic setting where (Ia, It, Mt
triples are sampled iid from an unknown distribution; the learner receives the context It E X, selects an action
A. and then observes the feedback ye (ar)), where Ve A -0, is the underlying feedback function. Note ra(a)) is never revealed to the algorithm: nonetheless, the regret notion remains the same as Eq. ① An information- therrrrticcc argument proves assumptions relating the feedhack to the underlying reward are necessary to succeed (Xie et al.) 2022).
[0,1]
2.2.1 SPECIALIZATION TO RECOMMENDATION
For specific application in the recommendation domain, we depart from prior art in IGL (Xie al. 2021 2022) in two ways: first, in the assumed relationship between feedback and underlying reward; and second, in the number of latent reward states. For the remainder the paper, we represent the users as (i.e.. context), content items action a, the user satisfaction with recommended content as r, and the feedback signals in response to the recommended content (user interactions with the system interface) y.
Feedback Dependence Assumption Xic et al. 2021 assumed full contextual independence of feedback on context and chosen action, i.c. y 2, a r. For recommender systems, this implies that all users communicate preferences identically for all content. In subsequent paper, Xiee et al.) (2022) loosen the full conditional independence by considering context conditional independence, y x.a,r. r. For setting, this corresponds to the user feedback varying for combinations of preference and content, but remaining consistent across all users. Neither of these two assumptions are natural in the recommendation setting because different users interact with recommender systems in different ways (Beel et al. 2013 Shin 2020). In this work, we assume 1 a|z, i.c., the feedback y is independent of the displayed content given the user x and their disposition toward the displayed content Thus, we assume that users may communicate in different ways, but a given user expresses satisfaction, dissatisfaction and indifference to all content in the same way.
Number of Latent Reward States Prior work demonstrates that a binary latent reward assump- tion, along with an assumption that rewards are rare under a known reference policy, are sufficient contrast between a learned policy and the oblivi-
for IGL to succeed. Specifically, optimizing ous uniform policy is able to succeed when feedback is both context and action independent (Xie) ctt al. 202I;; ; and optimizing the contrast between the learned policy and constant- policies succeeds when the feedback is context independent (Xie et al. 2022).
Although the binary latent reward assumption (c.g., satisfied or dissatisfied) appears reasonable for recommendation scenarios, fails to account for user indifference versus user dissatisfaction. This observation was first motivated by our production data, where a 2 state IGL policy would sometimes maximize feedback signals with obviously negative semantics. Assuming users ignore most content most of the time (Nguyen et al.) 2014), negative feedback can be difficult to elicit as positive feedback, and a 2 state IGL model is unable to distinguish between these extremes. Hence, we posit minimal latent state model for recommender systems involves 3 states: (i) 1, when users satisfied with the recommended content, (ii) r = O, when users are indifferent or inattentive, and (iii) -1, when users are dissatisfied. See Appendix A.T and Maghakian al. (2022) for details.
3

---

Published as a conference paper at ICLR 2023
3 DERIVATIONS
Prior approaches to IGL use contrastive learning objectives (Xie et al. 2021 2022), but the novel feedback dependence assumption in the prior section impedes this line of attack. Essentially, given arbitrary dependence upon æ, learning must operate each example in isolation without requiring comparison across examples. This motivates attempting to predict the current action from the current context and the currently observed feedback, i.e., inverse kinematics.
Inverse Kinematics Traditionally in robotics and computer animation, inverse kinematics is the mathematical process used to recover the movement (action) of a robot/object in the world from some other data such the position and orientation of robot manipulator/video of moving object (in our case, the other data is context and feedback). We motivate our inverse kinematics strategy using exact expectations. When acting according to any policy P(a[z), we can imagine trying to predict the action taken given the context and feedback; the posterior distribution is
x) P(alx)
(Bayes rule)
P(y(
Plylr,a,7 2()la,z)
= P((a/))
Σ
(Total Probability)
P(y[z)
P(ylr, 2) P(rla,x) P(ylæ))
=((a/)) Σ
(y 1 (.,r)))
P(r y, x) P(r(z)
P(dlz) E
P(rla,2)
(Bayes rule)
P(rla,x)P(alt)
Σ P(rly,x)
P(rra,,)))P(aa
(Total Probability)
We arrive at the inner product between reward decoder term (P(rly,a)) a)) and a reward predictor term (P(rla,.)).
Extreme Event Detection Direct extraction reward predictor using maximum likelihood on the action prediction problem with Eq. (2) is frustrated by two identifiability issues: first, this ex- pression is invariant to permutation of the rewards on a context dependent basis; and second, the relative scale of two terms being multiplied is not uniquely determined by their product. To miti- gate the first issue, we assume Σ. P(r = Oja, P(a ()) > 3. i.e., nonzero rewards are rare under P(a[z); and to mitigate the second issue, assume the feedback can be perfectly decoded, i.e., P()ly. E {0, 1}. Under these assumptions, we have
r = = P(aly, x) = P(r P((a
Σ. P(r 0/a, x)) P(alz)
< 2P = (((lz) s 2P(a))).
(3)
Eq. (3) forms the basis for our extreme event detector: anytime the posterior probability of an action predicted to be more than twice the prior probability, we deduce + 0.
Note a feedback merely being apriori rare frequent (i.c., the magnitude of P(y|z) under the policy P(a[=)) does not imply that observing such feedback will induce an extreme event detection; rather the feedback must have a probability that strongly depends upon which action is taken. Because feedback is assumed conditionally independent of action given the reward, the only way for feedback to help predict which action is played via the (action dependence of the) latent reward.
Extreme Event Disambiguation With 2 latent states, r #0 = r r= and we can reduce to a standard contextual bandit with inferred rewards 1 ((P(aly, > 2P(a)x)). With 3 latent states, r # = r = =1, and additional information is necessary to disambiguate the extreme events. We assume partial reward information is available via "definitely negative" function DN X x This reduces extreme event disambiguation one-sided learning (Bekker & Davis 2020 applied only to
y + (-1, 0} where P(DN(z, y) = O/r 1) = 1 and P(DN(z,)) y) = -1/r
"Definitely positive" information he incorporated analogously.
4

---

Published as a conference paper at ICLR 2023
extreme events, where try to predict the underlying latent state given (z, a). We assume partial labelling is selected completely at random Elkan & Noto (2008) and treat the (constant) negative labelling propensity as a hyperparameter. We arrive our 3-state reward extractor
P(aly, S 2P(a
pl 0, =
a
P(aly, > 2 and DN( v) -1,
(4)
otherwise
equivalent to Bekker & Davis 2020
Equation 11). Setting a = embeds 2-state IGL.
Algorithm IGI.; Inverse Kinematics; 2 or 3 Latent States; On or Off-Policy.
Input: Contextual bandit algorithm CB-Alg.
Input: Calibrated weighted multiclass classification algorithm MC-Alg.
Input: Definitely negative oracle DN.
# DN(...) = 0 for 2 state IGL # a = for state IGL
Input: Negative labelling propensity a.
Input: Action set size K.
π f- new CB-Alg.
2: IK <- new MC-Alg.
3: for 1, do
4: Observe context and action set A, with AA. K.
5: if On-PPllyy IGL then
6:
P(-]z)) w..redict(22, At).
7: Play ag ~ P(-(2)) and observe feedback yt-
8:
else
9:
Observe (2i,, az, P(-Ix))).
W. + 1/(KP (x.)).
# Synthetic uniform distribution
P(a, (1; 24) < IK. I...KK..pdiii... (re.peeeee(((re (4), Ar, at).
# Predict action probability
12:
if P(allut, xx) < then
# c
13:
T.leam Lt, 0t; AG, 7' 0) else
14:
# fz + 0
15:
if (N(...) then
16:
π. I.eean (Zt, a,, A,, F1 1, P(-1x))
else
# Definitely negative
18:
I.eear( at, A,, -a P(-22))
IK.lear((rg, W1), Ar, 0e, w.).
Implementation Notes In practice, P(a|z) is known but the other probabilities are estimated. P(aly, æ)) is estimated online using maximum likelihood on the problem predicting a from (z, y), i.e. on data stream of tuples ((I, a). The current estimates induce p(xx, a, y) based upon the plug-in version of Eq. 4.. In this manner, the original data stream of (x, a, y) tuples is transformed into a stream of (x, a, p(x, a, y)) tuples and reduced a standard online CB problem.
As an additional complication. although P(a)z) is known, it is typically a good policy under which rewards
rare (e.g., offline learning with good historical policy; acting online according to the policy being learned by the IGL procedure). Therefore, we use importance weighting to synthesize uniform action distribution P(a[z) from the true action distribution Ultimately arrive at the procedure of Algorithm
EMPIRICAL EVALUATIONS
Evaluation Settings: Settings include simulation using a supervised classification dataset, online news recommendation on Facebook, and production image recommendation scenario.
When the number of actions varies from round to round, use importance weighting to synthesize non-uniform action distribution with low rewards, but we elide this detail for of exposition.
5

---

Published as a conference paper at ICLR 2023
Abbreviations: Algorithms denoted by the following abbreviations: Personalized IGL for latent states (IGL-P (2) Personalized IGL for 3 latent states (IGL-P (3)); Contextual Ban- dits for the Facebook news setting that maximizes for emoji, non-like click-based reactions (CB-emo j 1 ); Contextual Bandits for the Facebook news setting that maximizes for comment in- teractions (CB-comment); Contextual Bandits for the production recommendation scenario that maximizes for CTR (CB- (CB-Click).
each time step f, the context zy is provided from either the simulator (Section 4.1 Section 4.2 or the logged production data (Section 4.3 The learner then selects an action a₂ and receives feedback yt- In these evaluations, each user provides feedback in exactly one interaction and different user feedback signals are mutually exclusive, so that y/ is one-hot vector. In simulated environments, the ground truth reward is sometimes used for evaluation but never revealed to the algorithm.
General Evaluation Setup:
Code: Our code is available for all publicly replicable experiments (i.e. except production data).
4.1 COVERTYPE IGL SIMULATION
To highlight that personalized IGL can distinguish between different user communication styles, we create a simulated 2-state IGL scenario from a supervised classification dataset. First, we apply supervised-to. no-dbandit transform to convert the dataset into contextual bandit simulation (Bietti et al.) 202T]. i.e., the algorithm is presented the example features as context, chooses one of the classes as an action, and experiences binary reward which indicates whether or not it matches the example label. In the IGL simulation, this reward is experienced but not revealed to the algorithm. Instead, the latent reward is converted into feedback signal follows: each example is assigned one of N different user ids and the user id is revealed to the algorithm as part of the example features. The simulated user will generate feedback in the form of one of M different word ids. Unknown to the algorithm, the words are divided equally into good and bad words, and the users are divided equally into original and shifted users. Original users indicate positive and zero reward via good and bad words respectively, while shifted users employ the exact opposite communication convention
User Embeddings
Word Embeddings
original
good
shifted
bad
Figure 2: The proposed personalized IGI. algorithm successfully disambiguates both different user communication styles and diffferent event semantics. See Section 4.1 for details.
We simulated using the Covertype (Blackard & Dean 1999) dataset with M = N = 100, and an (inverse kinematics) model class which embedded both user and word ids into 2 dimensional space. Fig. 2 demonstrates that both the user population and the words are cleanly separated into two latent groups. Additional results showcasing the learning curves for inverse kinematics, reward and policy learning are shown in Appendix A.2 along with detailed description of model parameters.
4.2 FAIRNESS IN FACEBOOK NEWS RECOMMENDATION
Personalized reward learning is the key to more fair recommender systems. Previous work (Neophy- tou et al. 2022) suggests that inconsistent performance in recommender systems across user sub- groups arises due to differences in user communication modalities. We now test this hypothesis in the setting of Facebook news recommendation. Our simulations are built on dataset (Martinchek)
2016) of all posts by the official Facebook pages of popular news outlets (ABC News, CBS News, CNN, Fox News and The New York Times) that span the political spectrum. Posts range from May to November 2016 6 and contain text content information, as well as logged interaction counts, which include comments and shares, as well as diverse click-based reactions (see Fig. 3)
'http::/ggthhub.. http:://gghub..co/ssaaa///CCC com/asaran/IGII.P
'Our motivation arises from the phenomenon of semantic shift, where word's usage evolves to the point that the original and modern meaning are dramatically different (e.g. terrific originally meant inspiring terror).

---

Published as a conference paper at ICLR 2023
Love
Hala
Wow
Sad
Angry
Figure 3: Facebook's 's click-based reactions: like, love, haha, wow, sad and angry (Meta 2016). The reactions allow users engage with content using diverse communication signals.
Constructing hand-engineered reward signal using these feedbacks is difficult, and Facebook itself came under fire for utilizing reward weights that disproportionately promote toxic, low quality news. One highly criticized iteration of the reward ranking algorithm treated emoji reactions as five times more valuable than likes (Merrill & Oremus 202Tp. Future iterations of the ranking algorithm pro- moted comments, in attempt bolster "meaningful social interactions" (Hagey & Jeff Horwitz, 2021]. Our experiments evaluate the performance of CB algorithms (Li et al. 2010) using these two reward functions, referring to them CB-emo j and CB-comment.
We model the news recommendation problem as 3 latent reward state problem, with readers of different news outlets as different contexts. Given curated selection of posts, the goal of the learner is to select the best article to show to the reader. The learner can leverage action features including the post type (link, video, photo, status or event) as well as embeddings the text content that were generated using pre-trained transformers (Reimers & Gurevych] (019).. User feedback is drawn from fixed probability distribution (unknown to the algorithms) that depends on the user type and latent reward of the chosen action. As approximation of the true latent reward signal, we use low dimensional embeddings of the different news outlets combined with aggregate statistics from the post feedback to categorize whether the users had positive (r 1), neutral (r 0) negative experience (r -)) with the post. This categorization is not available to the evaluated algorithms. Finally, IGL (3) uses the angry reaction negative oracle.
IGL-P(3) (3) CB-emoji CB-comment
Wwenne rewand
0.3
www.
0033
500
1000
1500
2000
2500
3000
3500
number of interactions
Figure 4: Of the three algorithms, only IGL- (3) consistently displays posts that users enjoy.
(3)
CB CB-emoj1
CS comment
TGI.-P(3)
CR-emoji
CB-comment
Erection rewards
0.8
Friction of rewwards
0.6
0.6
0.4
0.2
0.2
ABC
CBS
CNN
Fox
NYT
ABC
CBS
CNN
Fox
NYT
(a) Average fraction of rewards that positive
(b) Average fraction of rewards that negative
Figure 5: IGL-P P (3) uses personalized reward learning achieve fair news recommendations across diverse reader bases. CB policies based on those used by Facebook perform inconsistently, with subsets users receiving both fewer high quality and more low quality recommendations.
Figs. and 5 shows the results of our online news recommendation experiments. While the per- formance of both CB CB-emoji and CB-comment varies significantly across the different reader groups, IGL-P (3) provides fair performance for both positive and negative rewards. Although the CB-comment objective that was introduced to decrease low quality news succeeded for CBS readers, it significantly increased dissatisfaction among 3 of the 5 reader types (ABC News, Fox News and The New York Times). Additional details of this experiment, including model parameters and learning curves available in Appendix A.3

---

Published as a conference paper at ICLR 2023
4.3 PRODUCTION RESULTS
Our production setting is real world image recommendation system that serves hundreds of mil- lions of users. our recommendation system interface, users provide feedback in the form of clicks, likes, dislikes or no feedback. All four signals are mutually exclusive and the user only provides one feedback after each interaction. For these experiments, we use data that spans millions of interac- tions. The production baseline is contextual bandit algorithm with hand-engineered multi-task reward function which dominates approaches that only use click feedback Any improvements over the production policy imply improvement over any bandit algorithm optimizing for click feedback.
We implement IGL-P (2) and IGL-P (3) and report the performance relative lift metrics over the production baseline. Unlike the simulation setting, we no longer have access to the user's latent reward after each interaction. As a result, evaluate IGL by comparing all feedback signals. An increase in both clicks and likes, and a decrease in dislikes, are considered desirable outcomes. [ the results of empirical study. IGL-P (2) exhibits an inability to avoid extreme negative events. Although the true latent state is unknown, IGL-P (2) is Pareto-dominated due increase in dislikes. IGL P (3) does not exhibit this pathology. These results indicate the utility of a 3 latent state model in real world recommendation systems. To clearly illustrate the improvement of IGL-P (3) and provide an additional benchmark, we also report that IGL-P (3) significantly outperforms a contextual bandit policy CB-C1 ck using the industry standard CTR reward in Table
Algorithm
Clicks
Likes
Dislikes
IGL-B (3)
0.999, 1.067, 1.152]
[0.985, 1.029, 1.054] [0.914, 0.988
[0.751, 1.072, 1.274]
TGL-P (2)
0.926, 1.005, 1.091]
1.141, 1.557
Table 1: Relative metrics lift over a production baseline. The production baseline uses hand- engineered reward function which not available IGL algorithms. Shown are point estimates and associated bootstrap 95% confidence regions. IGL (2) erroneously increases dislikes to the detriment of other metries. IGL-P P (3) is equivalent to the hand-engineered baseline.
Algorithm
Clicks
Likes
Dislikes
IGL-P (3)
1.000, 1.010, 1. 1.020]
[1.006, 1.026, 1.049]
[0.890, 0.918, 0.955
CB -C-click
0.968, 0.979, 0.990)
0.993,, 0.959, 0.977] [12234, 1.311, 1.348)
Table 2: Relative metries lift over a CB policy that uses the click feedback as the reward. Point estimates and associated bootstrap 95% confidence regions are reported. TGL-P (3) significantly outperforms CB-Click at minimizing user dissatisfaction. Surprisingly, IGL-2 (3) even outper- forms Click with respect to click feedback, providing evidence for the latent state model 6
RELATED WORK
Eliminating reward engineering. Online learning algorithms in supervised learning, contextual
bandits and reinforcement learning all optimize over hand-crafted scalar rewards (Hoi et al.) [202T)
Previous work in areas such as inverse reinforcement learning (Arora & Doshi 2021). imitation learning (Hussein et al. 2017) and robust reinforcement learning (Zhou 2022) attempt to learn from historical data without access to the underlying true reward models. However these methods either require expert demonstration data, use hand-crafted objectives to incorporate additional human cues (Saran et al., 2020a by 2022) or are incapable of generalizing to different feedback signals (Chen et al. 2022), and generally need expensive compute and enormous sampling capabilities for learning. In this context, IGL (Xie et al., 2021 (022) a novel paradigm, where learner's goal to optimally interact with the environment with no explicit reward or
2021 Panaganti et al.)
'The utility of multi-task learning for recommendation systems is well-established, e.g., Chen al. 2021 Lu et al. (2018h); Chen et a (2019).
Data for Table and Table 2 are drawn from different date ranges due to compliance limitations, the IGL-P (3) performance need match.
8

---

Published as a conference paper at ICLR 2023
demonstrations to ground its policies. Our proposed approach to IGL, IGL-P (3), further enables learning of personalized reward functions for different feedback signals.
Recommender systems. Traditional vanilla recommendation approaches can be divided into three types. Content-based approaches maintain representations for users based their content and rec- ommend new content with good similarity metries for particular users (Lops et al. 2019). In con- trast, collaborative filtering approaches employ user rating predictions based on historical consumed content and underlying user similarities (Schafer et al. 2007). Finally, there are hybrid approaches that combine the previous two contrasting approaches to better represent user profiles for improved recommendations (Javed et al. 2021).
Our work is significant departure from these approaches, in that we learn representations of user's communication style via their content interaction history for improved diverse personalized recom- mendations. Specifically (1) propose novel personalized IGL algorithm based on the inverse kinematics strategy as described in Section 3 (2) we leverage both implicit and explicit feedback signals and avoid the costly, inefficient, status quo process of reward engineering, and (3) we formu- late and propose a new recommendation system based on the IGL paradigm. Unlike previous work on IGL, we propose personalized algorithm for recommendation systems with improved reward predictor models, more practical assumptions on feedback signals, and more intricacies described Section 2.2.1
DISCUSSION
We evaluated the proposed personalized IGL approach (IGL-P) in three different settings: (1) A simulation using a supervised classification dataset shows that IGL-P can learn to successfully dis- tinguish between different communication modalities; (2) A simulation for online news recommen- dation based on real data from Facebook users shows that IGL-P -P leverages insights about different communication modalities to learn better policies and achieve fairness with consistent performance among diverse user groups; (3) A real-world experiment deployed in an image recommendation product showcases that the proposed method outperforms the hand-engineered reward baseline, and succeeds in a practical application.
Our work assumes that users may communicate in different ways, but given user expresses (dis)satiffaction or indifference to all content in the same way. This assumption was critical to deriving the inverse kinematics approach, but in practice user feedback can also depend upon tent (Freeman et al., 2000).. IGL with arbitrary joint content-action dependence of feedback is in- tractable, but plausibly there exists a tractable IGL setting with constrained joint content-action dependence. We are very much interested in exploring this as part of future work. Additionally, although established the utility of three state model over two state model in our experiments, perhaps more than three states is necessary for more complex recommendation scenarios.
Although we consider the application of recommender systems, personalized reward learning can benefit any application suffering from a mne-size-itts-ap approach. One candidate domain is adap- tive brain-computer and human-computer interfaces (Xie et al.. (022). An example is self- calibrating eye tracker where people's response to erroneous agent control is idiosyncratic (Gao et al. 2021] Zhang et al.) 22200) (Saaa et al. 2018). Other domains include the detection of under- lying medical conditions and development of new treatments and interventions. Systematic misdi- agnoses in subpopulations with symptoms different than those of white men tragically standard in US medicine (Carter), with early stroke signs overlooked in women, minorities and young pa- tients (Newman- Toker al.] 2014), underdetected multiple sclerosis in the Black community (Okai et al. 2022). and the widespread misclassification of autism in women (Supekar & Menon 2015). IGL-P can empower medical practitioners address systematic inequality by tuning diagnostic tools via personalization and moving past patient-oblivious diagnostic approach. Other possible applications that can achieve improved fairness by using IGL-P include automated resume screen- ing (Parasurama & Sedoc. 2022) and consumer lending (Dobbie et al., 202T).
7 ACKNOWLEDGEMENTS
The authors thank John Langford for the valuable discussions and helpful feedback. This material is based upon work supported by the National Science Foundation under Grant No. 1650114 14.

---

Published as a conference paper at ICLR 2023
ETHICS STATEMENT
In paper, two woaldworld interaction datasets. The first is publicly available dataset of interactions with public Facebook news pages. All interactions are anonymous and the identities of users are excluded from the dataset, preserving their privacy. Similarly, our production data does not include user information and was used with the consent and permission of all relevant parties.
REPRODUCIBILITY STATEMENT
We have taken considerable measures to ensure the results are reproducible as possible. We have provided the code to replicate our experiment results (except for the production results in Section 4.3) as part of the supplementary material. The code will be made publicly available at {url redacted). We used publicly available datasets for our simulated experiments in Section 4.1 (Blackard & Dean 1999) and Section (.2 (attiinchek,220666 The experiment code for Section 4.1
when executed, will automatically download the dataset. The dataset for Section 4.2 is included as part of our supplementary material. Finally, our supplementary material also includes a conda environment file help future researchers recreate development environment their machines when running the experiments.
REFERENCES
Saurabh Arora and Prashant Doshi. A survey of inverse reinforcement learning: Challenges, meth- ods and progress. Artificial Intelligence, 297:103500, 2021.
Peter Auer, Nicolo Cesa-Bianchi, Yoav Freund, and Robert E Schapire. The nonstochastic multi- armed bandit problem. SIAM journal on computing, 32(1):48-1) 2002.
Joeran Beel, Stefan Langer, Andreas Nürnberger, and Marcel Genzmehr. The impact of demo- graphics (age and gender) and other user-characteristics on evaluating recommender systems. In International Conference on Theory and Practice of Digital Libraries, pp. 396-400. Springer, 2013.
Jessa Bekker and Jesse Davis. Learning from positive and unlabeled data: A survey. Machine Learning. 109(4):719-760, 2020.
Alberto Bietti, Alekh Agarwal, and John Langford. contextual bandit bake-off. Mach. Learn. Res., 22: 12 2021.
Jock A Blackard and Denis J Dean. Comparative accuracies of artificial neural networks and dis- criminant analysis in predicting forest cover types from cartographic variables. Computers and electronics in agriculture, 2313):131-15,, 1999.
Djallel Bouneffouf, Irina Rish, and Charu Aggarwal. Survey applications of multi-armed and contextual bandits. In 2020 IEEE Congress on Evolutionary Computation (CEC), pp. 1-8. IEEE, 2020.
Kiera Carter. The frustrating reasons why doctors constantly misdiagnose women. Prevention. URL www.mmiiiiaannnnom women/
https: / //wwww/w.///////////////// www.:pprever.. prwvent ion. cwwm wtt.r//oom/eeeaaaa/////000100220
Lei Chen, Jie Cao, Guixiang Zhu, Youquan Wang, and Weichao Liang. A multi-task learning ap- proach for improving travel recommendation with keywords generation. Knowledge-Bard Sys- tems, 233:107521, 2021.
Xiaocong Chen, Lina Yao, Xianzhi Wang, Aixin Sun, and Quan Z Sheng. Generative adversarial reward learning for generalized behavior tendency inference. IEEE Transactions on Knowledge and Data Engineering, 2022.
Zhongxia Chen, Xiting Wang, Xing Xie, Tong Wu, Guoqing Bu, Yining Wang, and Enhong Chen. Co-attentive multi-task learning for explainable recommendation. In IJCAI, pp. 2137-2143, 2019.
Will Dobbie, Andres Liberman, Daniel Paravisini, and Vikram Pathania. Mcasuring bias in con- sumer lending. The Review of Economic Studies, 88(6):2799-2833 2021.
10

---

Published as a conference paper at ICLR 2023
Michael D Ekstrand, Mucun Tian, Ion Madrazo Azpiazu, Jennifer D Ekstrand, Oghenemaro Anuyah, David McNeill, and Maria Soledad Pera. All the cool kids, how do they fit in?: Pop- ularity and demographic biases in recommender evaluation and effectiiveness. In Conference fairness, accountability and transparency, pp. 172-186. PMLR, 2018.
Charles Elkan and Keith Noto. Learning classifiers from only positive and unlabeled data. In Proceedings of the 14th ACM SIGKDD international conference on Knowledge discovery and data mining, pp. 213-220, 2008.
Cole Freeman, Hamed Alhoori, and Murtuza Shahzad. Measuring the diversity of facebook reac- tions to research. Proceedings of the ACM on Human- Comoun- Interaction, 4(GROUP):1-17, 2020.
Jensen Gao, Siddharth Reddy, Glen Berseth, Nicholas Hardy, Nikhilesh Natraj, Karunesh Ganguly, Anca D. Dragan, and Sergey Levine. X2T: training an x-to-text typing interface with online learning from user feedback. In 9th International Conference on Learning Representations, ICLR, 2021.
Miha Grčar, Dunja Mladenič, Blaž Fortuna, and Marko Grobelník. Data sparsity issues in the collaborative filtering framework. In International workshop on knowledge discovery the web, pp. 58-76. Springer, 2005.
Keach Hagey and Jeff Jeff Horwitz. Facebook tried to make its platform a healthier place. it got angrier instead. The Wall Street Journal, 2021. URL.https: www. wa wwm/ cow.articcls//////ocmccm.m..m..t Facebook - algorithm change wwwker 11631654215
Katja Hofmann, Fritz, Behr, and Filip Radlinski. On caption bias in interleaving experiments. In Pro- ceedings of the 21st ACM international conference on Information and knowledge management, pp. 115-124, 2012.
Steven CH Hoi, Doyen Sahoo, Jing Lu, and Peilin Zhao. Online learning: A comprehensive survey. Neurocomputing, 459:249-2888 2899 2021.
Yifan Hu, Yehuda Koren, and Chris Volinsky. Collaborative filtering for implicit feedback datasets. In 2008 Eighth IEEE international conference data mining, pp. 263-272. leee, 2008
Ahmed Hussein, Mohamed Medhat Gaber, Eyad Elyan, and Chrisina Jayne. Imitation learning: A survey of learning methods. ACM Computing Surveys (CSUR), 50(2):1-35, 2017.
Umair Javed, Kamran Shaukat, Ibrahim A Hameed, Farhat Iqbal, Talha Mahboob Alam, and Suhuai Luo. A review of content-based based and context-based recommendation systems. International Jour- nal of Emerging Technologies in Learning (iJE7), 16(3):274-366, 2021.
M Laeeq Khan. Social media engagement: What motivates user participation and consumption on YouTube? Computers in human behavior, 66:236-247, 2017.
Youngho Kim, Ahmed Hassan, Ryen W White, and Imed Zitouni. Modeling dwell time predict click-level satisfaction. In Proceedings of the 7th ACM international conference on Web search and data mining, pp. 193 903-202.200 2014.
John Langford and Tong Zhang. The epoch-greedy algorithm for contextual multi-armed bandits. Advances in neural information processing systems, 20(1):96-1, 2007.
Lihong Li, Wei Chu, John Langford, and Robert E. Schapire. A contextual- approach to personalized news article recommendation. Proceedings of the 19th international conference on World wide web, pp. 661-670, 2010.
Pasquale Lops, Dietmar Jannach, Cataldo Musto, Toine Bogers, and Marijn Koolen. Trends in content-based recommendation. User Modeling and User-Adapted Interaction, 29(2):239-249,
2019.
Hongyu Lu, Min Zhang, and Shaoping Ma. Between clicks and satisfaction: Study multi-phase user preferences and satisfaction for online news reading. In The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval, pp. 435 444, 2018 Sa.

---

Published as a conference paper at ICLR 2023
Yichao Lu, Ruihai Dong, and Barry Smyth. Why i like it: multi-tast learning for recommendation and explanation. In Proceedings of the 12th ACM Conference on Recommender Systems, pp- 4-12, 2018b.
Jessica Maghakian, Kishan Panaganti, Paul Mineiro, Akanksha Saran, and Cheng Tan. Interaction- grounded learning for recommender systems. 2022.
Masoud Mansoury, Himan Abdollahpouri, Jessie Smith, Arman Dehpanah, Mykola Pechenizkiy, and Bamshad Mobasher. Investigating potential factors associated with gender discrimination in collaborative recommender systems. In The Thirty-Third International Flairs Conference, 2020.
Patrick Martinchek. 2012-2016 Facebook Posts.
2012-2016 fww.bock fa01--ffccbook-posst 2016.
Jeremy Merrill and Will Oremus. Five points for anger, one for 'like': How face- book's formula fostered rage and misinformation. The Washington Post, 2021. URL
https://wwet www. washingtonpost. com////chooooooy//22///22222 -acebook-angzy-emoo1 aalpyyyyyyemm---lgoraattm
Meta.
Reactions available globally. Facebook News, 2016. URL https: http:p::///bbuu....
Nicola Neophytou, Bhaskar Mitra, and Catherine Stinson. Revisiting popularity and demographic biases in recommender evaluation and effectiveness. European Conference on Information Retrieval, pp. 641-654. Springer, 2022.
David E Newman- Toker, Ernest Moy, Ernest Valente, Rosanna Coffey, and Anika L Hines. Missed diagnosis of stroke in the emergency department: a cross-sectional analysis of a large population- based sample. Diagnosis, 1(2):155-16)) 166, 2014.
Tien T Nguyen. Pik-Mai Hui, F Maxwell Harper, Loren Terveen, and Joseph A Konstan. Exploring the filter bubble: the effect of using recommender systems content diversity. In Proceedings of the 23rd international conference on World wide web, pp. 677-686, 2014.
Annette F Okai, Annette M Howard, Mitzi J Williams, Justine D Brink, Chiayi Chen, Tamela I. Stuchiner, Elizabeth Baraban, Grace Jeong, and Stanley Cohan. Advancing care and outcomes for african american patients with multiple sclerosis. Neurology, 98(24):1015-1020 2022.
Kishan Panaganti, Zaiyan Xu, Dileep Kalathil, and Mohammad Ghavamzadeh. Robust reinforce- ment learning using offline data. In Advances in Neural Information Processing Systems, 2022.
Prasanna Parasurama and João Sedoc. Gendered information in resumes and its role in algorith- mic and human hiring bias. In Academy of Management Proceedings, volume 2022, pp. 17133. Academy of Management Briarcliff Manor, NY 10510, 2022.
Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bert- networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, 2019.
Akanksha Saran, Srinjoy Majumdar, Elaine Schaertl Short, Andrea Thomaz, and Scott Niekum. Human gaze following for human-rohot interaction. In 20/8 JEEF/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 8615-8621. IEEE, 2018.
Akanksha Saran, Flaine Schaertl Short, Andrea Thoma7, and Scott Niekum. Understanding teacher gaze patterns for robot Icarning. In Conference on Robot Learning, pp. 1247-1258. PMLR, 2020a.
Akanksha Saran, Ruohan Zhang, Elaine Schaertl Short, and Scott Niekum. Efficiently guiding imitation learning agents with human gaze. arXiv preprint arXiv:2002.12500, 2020b.
Akanksha Saran, Kush Desai, Mai Lee Chang, Rudolf Lioutikov, Andrea Thomaz, and Scott Nickum. Understanding acoustic patterns of human teachers demonstrating manipulation tasks to robots. In 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 9172-9179. IEBE, 2022.
12

---

Published as a conference paper at ICLR 2023
J Ben Schafer, Dan Frankowski, Jon Herlocker, and Shilad Sen. Collaborative filtering recommender systems. In The adaptive web, pp. 291-324. Springer, 2007.
Kate Scott. You won't I believe what's in this paper! Clickbait, relevance and the curiosity gap.
Journal of pragmatics, 175:53-66, 2021.
Donghee Shin. How do users interact with algorithm recommender systems? The interaction of
users, algorithms, and performance. Computers in Human Behavior, 109:106344, 2020.
Thiago Silveira, Min Zhang, Xiao Lin, Yigun Liu, and Shaoping Ma. How good your recommender system is? A survey on evaluations in recommendation. International Journal of Machine Learn- ing and Cybernetics, 10(5):813 831, 2019.
Kaustubh Supekar and Vinod Menon. Sex differences in structural organization of motor systems and their dissociable links with repetitiverrestricted behaviors in children with autism. Molecular Autism, 6(1):1-13, 2015.
Wenjie Wang. Fuli Feng, Xiangnan He, Hanwang Zhang, and Tat-Seng Chua. Clicks can be cheat- ing: Counterfactual recommendation for mitigating clickbait issue. In Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 1288-1297, 2021.
Qingyun Wu, Hongning Wang, Liangjie Hong, and Yue Shi. Returning is believing: Optimizing long-term user engagement in recommender systems. In Proceedings of the 2017 ACM on Con- ference Information and Knowledge Management, pp. 1927-1936, 2017.
Tengyang Xie, John Langford. Paul Mineiro. and Ida Momennejad. Interaction -rounnded Learning. In International Conference on Machine Learning pp. 11414-11423 PMLR, 2021.
Tengyang Xie, Akanksha Saran, Dylan J Foster, Lekan Molu, Ida Momennejad, Nan Jiang, Paul Mineiro, and John Langford. ineeactiionGrountle Learning with Action-inclusive Feedback. arXiv preprint wxxh:2206.00834, 2022.
Xing Yi, Liangjie Hong. Erheng Zhong, Nanthan Nan Liu, and Suju Rajan. Beyond clicks: dwell time for personalization. In Proceedings of the 8th ACM Conference on Recommender systems, pp. 113-120, 2014.
Ruohan Zhang, Akanksha Saran, Bo Liu, Yifeng Zhu, Sihang Guo, Scott Niekum, Dana Ballard, and Mary Hayhoe. Human gaze assisted artificial intelligence: A review. In IJCAI: Proceedings of the Conference, volume 2020, pp. 4951. NIH Public Access, 2020.
Zhengqing Zhou, Zhengyuan Zhou, Qinxun Bai, Linhai Qiu, Jose Blanchet, and Peter Glynn. Finite- sample regret bound for distributionally robust offline tabular reinforcement learning. In Interna- tional Conference on Artificial Intelligence and Statistics, pp. 3331-3339. PMLR, 2021.
13

---

Published as a conference paper at ICLR 2023
A APPENDIX
A.1 THE 3 STATE MODEL FOR RECOMMENDER SYSTEMS
Here we provide background on the 3 latent state model for recommendation systems from Maghakian et al. (0022) In particular, we demonstrate that the traditional 2 latent state reward model used by prior IGL methods (Xie et al. 2021 2022) is not sufficient for the recommender sys- tem scenario. We further illustrate that latent reward state model can overcome the limitations of the two latent state model, and achieve personalized reward learning for recommendations.
Simulator Design. Before the start of each experiment, user profiles with fixed latent rewards for each action are generated. Users are also assigned predetermined communication styles, with fixed probabilities of emitting a given signal conditioned on the latent reward. The available feedback includes mix of explicit (likes, dislikes) and implicit (clicks, skips, none) signals. Despite receiving no human input on the assumed meaning of the implicit signals, IGL can determine which feedback are associated with which latent state. addition to policy optimization, IGL can also be tool for automated feature discovery. To reveal the qualitative properties of the approach, the simulated probabilities for observing a particular feedback given the reward are chosen so that they can be perfectly decoded, i.e., each feedback has nonzero emission probability in exactly one latent reward state. Although production data does not obey this constraint (c.g., nonzero accidental feedback emissions), the not perfectly decodable setting is topic for future work.
IGL-P(2). We implement Algorithm for latent states as IGL-P (2). Our experiment shows the following two results about IGL-P (2) (i) it is able to succeed in the scenario when there are 2 underlying latent rewards and (ii) it can no longer do so when there are latent states. Fig. showss the simulator setup used, where clicks and likes are used to communicate satisfaction, and dislikes, skips and no feedback (none) convey (active or passive) dissatisfaction.
distiler
sldp
skip
(a) 2 latent state model
(b) 3 latent state model
Figure 6: Simulator settings for state and state latent model. In Fig. 6a
r" corresponds to
anything other than the user actively enjoying the content, whereas in Fig. 6b is split into indifference and active dissatisfaction.
lack of user enjoyment
Fig. shows the distribution of rewards for IGL-P (2) as a function of the number of iterations, for both the 2 and 3 latent state model. When there are only 2 latent rewards, IGL P (2) consistently improves; however for 3 latent states, IGL-P (2) oscillates between r = and r = -1, resulting in much lower average user satisfaction. The empirical results demonstrate that although IGL-P (2) can successfully identify and maximize the rare feedbacks it encounters, is unable to distinguish between satisfied and dissatisfied users.
fraction of rewards
1.0
fraction of rewards
1.0
0.8
0.8
0.6
0.6
0.4
0.4
0.2-
www.
0.2
0
200
400
600
800
1000
1200
1400
0
200
400
600
800
1000
1200
1400
number of interactions
mumber of interactions
(a) Two latent states
(b) Three latent states
Figure 7: Although IGL-P (2) is successful with the 2 state simulator, it fails on the state simu- lator and oscillates between attempting to maximize = 1 and -1.
14

---

400000 Published as a conference paper at ICLR 2023
IGL-P(3): Personalized Reward Learning for Recommendations, Since IGL-P (2) is not suf- ficient for the recommendation system setting, we now explore the performance of IGL-P (3).
Using the same simulator as Fig. [b] we evaluated IGL-P (3). Fig. Sa demonstrates the distribu- tion of the rewards over the course of the experiment. TGL-P (3) quickly converged, and because of the partial negative feedback for dislikes, never attempted maximize the r -1 state. Even though users used the ambiguous skip signal to express dissatisfaction 80% of the time, IGL P (3) was still able learn user preferences.
In order for IGL-P (3) succeed, the algorithm requires direct grounding from the dislike signal. We next examined how IGL-P (3) is impacted by increased or decreased presence of user dislikes. Fig. [bb] was generated by varying the probability p of users emitting dislikes given r = 1, and then averaging over 10 experiments for each choice of p. While lower dislike emission probabilities are associated with slower convergence, IGL-P (3) is able to overcome the increase in unlabeled feedback and learn to associate the skip signal with user dissatisfaction. Once the feedback decoding stabilizes, regardless of the dislike emission probability, IGL-P (3) enjoys strong performance for the remainder of the experiment.
0.8
0.4
0.2
0.1
Fractiio of rewaardi
0.6
N/
0.4-
0.2
0.2
0.0
200
600
800 1000
1200
1400
200
400
600
800
1000
1200
1400
number of interactions
mumber of interactions
(a)
Ground
truth
learning
curves,
(b) Effect of varying (diilikke)r - 1).
Pddiliike|r- = 1) 0.2.
Figure 8: Performance of IGL-P (3) in simulated environment. In Fig. 8a IGL-P (3) suc- cessfully maximizes user satisfaction while minimizing dissatisfaction. Fig. demonstrates how IGL-P (3) is robust to varying the frequency of partial information received, although more data needed for convergence when "definitely bad" events are less frequent.
A.2 ADDITIONAL. RESULTS FOR THE COVERTYPE IGL SIMULATIONN
Model Parameters For The Covertype IGL Simulation. For the Covertype IGL simulation we trained both a CB and IK model (as shown in Algorithm [T We use online multiclass learning to implement the IK model, where the classes are all the possible actions and the label is the action played. Both CB and IK are linear logistic regression models implemented in PyTorch, trained using the cross-entropy loss. Both models used Adam to update their weights with learning rate of 2.5e The first layer of weights for both models had 512 weights that were initialized using a Cauchy distribution with 0.2. The learning curves are shown in Figure 9b
0.6
IB accupac
roward
0.4
0.2
0.2
20
100020
200000
300000
500000
100000
200000 number of interaations
300000
400000
500000
number of internctions
(a) Reward learning.
(b) Action prediction accuracy.
Figure 9: Learning curves for the Covertype simulated experiment in Sec. 4.1 Figure shows the accuracy with which the inverse kinematies model recovered the actions.
15

---

Published as a conference paper at ICLR 2023
A.3 ADDITIONAL RESULTS FOR FACEBOOK NEWS SIMULATION
Model Parameters For The Facebook News Simulation. For the Facebook simulation there were 2 CB models and an IK model. All models used Adam to update their weights with a learning rate of 10 batch size 100, and cross-entropy loss function. The reward learning curves shown in Figures 10 11 and 12 Further details about implementation are available in provided code ?
fraction of rewards
ABC News
CBS News
CNN
Fox News
NY Times
1.0
1.0
1.0
1.0
1.0
0.75
0.75
M
0.76
0.75
0.75
0.5
0.5
0.5
fpm
0.25
0.25
0.25
0.25
0.25
50lk
25k
75k
50k
25k
50k
number of interactions
Figure 10: Reward learning curves for IGL-P (3) in the Facebook news recommendation exper- iment. Over time, IGL-P P (3) learns the difference between the three states and tries to minimize r 0 and r = while maximizing r = across all contexts.
r=-1
fraction of rewards
ABC News
CBS News
CNN
Fox News
NY Times
1.0
1.0
1.0
1.0
1.0
mmmm
0.5
0.5
0.5
0.5
0.5
0.25
www.
M mm
0.0
0.0
0.0
0.0
0.0
08
25k
Silk
75k
0x
25k
50k
75k
03
25k
5ilk
75k
0k
25k
50lk
75k
03
25k
50lk
75k
number of interactions
Figure 11: Reward learning curves for CB-emoji i in the Facebook recommendation experi- ment. Two of the five reader types (CBS News and CNN) communicate dissatisfaction using many of the emoji feedback. As result, CB-emo ji ultimately maximizes -1 for those readers.
r=-1
r=0
fraction of rewards
ABC News
CBS News
CNN
News
Times
0.75
0.75
0.75
0.75
0.75
0.5
0.5
0.5
0.5
0.5
0.25
0.25
0.25
0.25
0.25
0.0
0.0
0.0
0.0
0.0
50k
75k
60k
75k
50k
number interactions
Figure 12: Reward learning curves for CB comment in the Facebook news recommendation ex- periment. Although CB-comment performs successfully with CBS News readers, it maximizes the unhappiness of ABC News, Fox News and The New York Times readers.
16


Reviews:
[{"review_id": "x6sHJxvkZz", "reviewer": "Reviewer_4RJN", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this paper, the authors propose a new algorithm for Personalized RecSys under Interaction Grounded Learning (IGL) paradigm where a personalized policy is learned to maximum unobservable user satisfaction with only implicit feedbacks. The authors alter the independence assumption used in vanilla IGL (Xie et al 2021/2022) to enable personalized feedback that depends on the context. They also increase the reward state from 2 to 3 to capture non-extreme (neutral) user state. Based on the modified assumptions, the authors propose a new IGL algorithm with Inverse Kinematics (different from contrastive learning algorithm proposed in original IGL in 2022) and a novel reward decoder that handles three states.  Empirical results are shown with both simulation and real-world News recommendation dataset, where better fairness is achieved when feedbacks are synthetically designed to be user dependent. They also AB test in production setting and is comparable with contextual bandit with reward engineering.", "summary_of_the_review": "The paper proposes novel solution for personalized reward learning with implicit feedback in IGL paradigm, and come up with important modification of original IGL (Xie et al 2022). Despite limited theoretical result and baselines, the contribution is potentially significant to the community.", "strength_and_weaknesses": "Strength:\n\n- The authors find a novel application of IGL paradigm on personalized RecSys and propose key modifications that adapt to user modeling use case. The resulting model solves an important problem in RecSys where we usually lack explicit feedback on the objective that need to be optimized, and implicit feedbacks are sensitive to user subgroups such that ad-hoc reward engineering is undesirable.\n- The application of Inverse Kinematics on IGL problem is novel, and it is technically non-trivial.\n- The introduction of 3-state reward decoder is non-trivial and interesting.\n- The result on synthetic feedback scenario shows improved fairness, and the algorithm is comparable to SOTA production candidate with hand-crafted reward engineering.\n\nWeakness:\n\nDespite the novelty and contributions, there are some ambiguities that may need further clarification\n\n- When and how is the DN(…) function learned? It seems PU-learning is used where only negative labels are partially provided, and the labeling probability is somehow known. But in the empirical experiments it is unclear how these set of information is obtained. Moreover, it is non-trivial to estimate labeling probability in real-world, and there is not much explanation on how this hyperparameter is chosen and how it impact the performance.\n- The synthetic experiments with Facebook and Covertype datasets are designed such that feedbacks are heavily dependent on user type, making it in favor of the proposed algorithm. And only simple CB are used as baselines for comparison. The real-production result is comparable to reward-engineering, while the significance of the lift is not clear as the confidence lower bound <1 for likes and there is a positive lift in dislike. Comparison to action-inclusive IGL in Xie et al 2022 is also lacking, making it less convincing whether the user dependence assumption is the main reason for the lift.\n- The authors propose different assumption and learning algorithm than vanilla IGL. Yet no theoretical analysis is provided to prove whether the new independence assumption and the learning algorithm converges to the optimal solution. The enhancement on fairness is also supported with only empirical evidence, thus might limit the impact of the work.\n\nMinor issues:\n\n- There might be a typo in the last line of equation (2) where the prior on numerator is `p(a,x)` whereas in equation(3) it is `p(a|x)`.\n- Missing figure for learning curves for IK, reward and policy learning. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "5LM4zsgKag", "reviewer": "Reviewer_7wvr", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper posits that reward signals from users are not necessarily clear and cannot be easily attributed to genuine preference but might be the result of various biases. In particular implicit signals (watches/clicks) do not map directly to user satisfaction. Moreover, different users can have different ways of signaling preferences. The authors propose Interaction Grounded Learning or personalized\nreward learning (IGL). IGL is a learning paradigm where a learner optimizes for unobservable\nrewards by interacting with the environment and associating observable feedback with the true latent reward.\nThe adaptations to the case of recommendation of IGL are presented in the paper and the final formulation of the algorithm involves a mapping between conditional probabilities and estimated potential rewards. On that basis, a Bandit-type algorithm is trained and experiments are conducted on artificial offline data and real news recommendation data with positive results. \n\n", "summary_of_the_review": "Overall an interesting work with a novel treatment of a problem in the area of recommender systems. \nI'm a bit unsure about the impact of the work in a general ML conference. ", "strength_and_weaknesses": "The paper is well written and well motivated, in particular, the issue of extracting reward out of implicit signals in recommendations is an open question. The paper also includes results from a production system that is not typically the case in this type of work. Moreover, the idea of applying IGL principles in the area of recommender systems is to the best of my knowledge novel and justified well. \nThe experimental section is somewhat restricted to results from the live experiment and there is no clear comparison to e.g. bandit algorithms with a simpler implicit signal to reward mapping hence it is unclear how much of an advantage the method hold compared to potentially simpler alternatives. It is also somewhat unclear against what baseline the method is tested in the live experiments. \nOne somewhat negative point is that this work is relatively narrow even in the space of recommender systems and might interest only a small subset of the ICLR audience, a more IR/recommender systems conference might be a more appropriate venue. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "1LxQaRxP2D", "reviewer": "Reviewer_iWcq", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a personalized reward learning method for recommender systems. The authors apply the recent Interaction Grounded Learning (IGL) paradigm to address the challenge of learning representations of diverse user communication modalities. The proposed personalized IGL is designed for context-dependent feedback, with inverse kinematics as an IGL objective and the capability of modeling more than two latent states. Both simulations and experiments on real production data have demonstrated the proposed IGL is able to address two typical challenges for modern online recommender systems, i.e., feedback-reward dependence assumption and the number of latent reward states.", "summary_of_the_review": "Given the listed strengths and weaknesses, I think this is a borderline paper.\n\n## After rebuttal\nThe authors' response has somewhat alleviated my concerns. Thus I decide to raise my score from 5 to 6.", "strength_and_weaknesses": "Strengths:\n1. This paper tackles an important research problem, optimizing latent user satisfaction in recommender systems, through an interesting angle, i.e., learning personalized reward functions.\n2. The proposed IGL method is reasonable and capable of solving domain-specific challenges.\n3. The proposed method is rigorously derived in Section 3.\n\nWeaknesses:\n1. Background on contextual bandit and interaction grounded learning should be strengthened.\n2. The authors claimed that this method could handle both implicit and explicit feedback, while only implicit feedback is considered in experiments.\n3. The empirical performance improvements seem to be barely marginal, and only three types of contexts are considered in the facebook news recommendation scenario. What is the number of contexts on production datasets?\n4. Many details of experiments and implementations are missing, like the definition of metrics, scale of production dataset, model parameters, etc.\n5. The paper is generally well-written, but its clarity can be further improved. For example, the meaning of $a$ / $r$ in recommender systems should be clearly defined.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster