Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper posits that reward signals from users are not necessarily clear and cannot be easily attributed to genuine preference but might be the result of various biases. In particular implicit signals (watches/clicks) do not map directly to user satisfaction. Moreover, different users can have different ways of signaling preferences. The authors propose Interaction Grounded Learning or personalized
reward learning (IGL). IGL is a learning paradigm where a learner optimizes for unobservable
rewards by interacting with the environment and associating observable feedback with the true latent reward.
The adaptations to the case of recommendation of IGL are presented in the paper and the final formulation of the algorithm involves a mapping between conditional probabilities and estimated potential rewards. On that basis, a Bandit-type algorithm is trained and experiments are conducted on artificial offline data and real news recommendation data with positive results. 


strength_and_weaknesses: The paper is well written and well motivated, in particular, the issue of extracting reward out of implicit signals in recommendations is an open question. The paper also includes results from a production system that is not typically the case in this type of work. Moreover, the idea of applying IGL principles in the area of recommender systems is to the best of my knowledge novel and justified well. 
The experimental section is somewhat restricted to results from the live experiment and there is no clear comparison to e.g. bandit algorithms with a simpler implicit signal to reward mapping hence it is unclear how much of an advantage the method hold compared to potentially simpler alternatives. It is also somewhat unclear against what baseline the method is tested in the live experiments. 
One somewhat negative point is that this work is relatively narrow even in the space of recommender systems and might interest only a small subset of the ICLR audience, a more IR/recommender systems conference might be a more appropriate venue. 
clarity,_quality,_novelty_and_reproducibility: Overall the paper is well written, and there is some degree of novelty in that IGL is being applied to the recommendation problem and in particular to mapping implicit signals to rewards. The treatment of the problem and the derivation of the mapping to rewards could be clearer as it is somewhat difficult to follow. It would be helpful if more of an intuition is provided at every step.  
The author seem to be willing to publish the code and most datasets from the experimental section which would provide adequate reproducibility of a big part of the experimental section. 

summary_of_the_review: Overall an interesting work with a novel treatment of a problem in the area of recommender systems. 
I'm a bit unsure about the impact of the work in a general ML conference.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a personalized reward learning method for recommender systems. The authors apply the recent Interaction Grounded Learning (IGL) paradigm to address the challenge of learning representations of diverse user communication modalities. The proposed personalized IGL is designed for context-dependent feedback, with inverse kinematics as an IGL objective and the capability of modeling more than two latent states. Both simulations and experiments on real production data have demonstrated the proposed IGL is able to address two typical challenges for modern online recommender systems, i.e., feedback-reward dependence assumption and the number of latent reward states.
strength_and_weaknesses: Strengths:
1. This paper tackles an important research problem, optimizing latent user satisfaction in recommender systems, through an interesting angle, i.e., learning personalized reward functions.
2. The proposed IGL method is reasonable and capable of solving domain-specific challenges.
3. The proposed method is rigorously derived in Section 3.

Weaknesses:
1. Background on contextual bandit and interaction grounded learning should be strengthened.
2. The authors claimed that this method could handle both implicit and explicit feedback, while only implicit feedback is considered in experiments.
3. The empirical performance improvements seem to be barely marginal, and only three types of contexts are considered in the facebook news recommendation scenario. What is the number of contexts on production datasets?
4. Many details of experiments and implementations are missing, like the definition of metrics, scale of production dataset, model parameters, etc.
5. The paper is generally well-written, but its clarity can be further improved. For example, the meaning of $a$ / $r$ in recommender systems should be clearly defined.

clarity,_quality,_novelty_and_reproducibility: Please refer to my comments above for clarity issues.

I think the idea of IGL for recommendation is novel and reasonable. The proposed method also solves two domain-specific challenges, though left with many others to be further tackled.

The authors also provide the source code for reproducibility check.

summary_of_the_review: Given the listed strengths and weaknesses, I think this is a borderline paper.

## After rebuttal
The authors' response has somewhat alleviated my concerns. Thus I decide to raise my score from 5 to 6.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: In this paper, the authors propose a new algorithm for Personalized RecSys under Interaction Grounded Learning (IGL) paradigm where a personalized policy is learned to maximum unobservable user satisfaction with only implicit feedbacks. The authors alter the independence assumption used in vanilla IGL (Xie et al 2021/2022) to enable personalized feedback that depends on the context. They also increase the reward state from 2 to 3 to capture non-extreme (neutral) user state. Based on the modified assumptions, the authors propose a new IGL algorithm with Inverse Kinematics (different from contrastive learning algorithm proposed in original IGL in 2022) and a novel reward decoder that handles three states.  Empirical results are shown with both simulation and real-world News recommendation dataset, where better fairness is achieved when feedbacks are synthetically designed to be user dependent. They also AB test in production setting and is comparable with contextual bandit with reward engineering.
strength_and_weaknesses: Strength:

- The authors find a novel application of IGL paradigm on personalized RecSys and propose key modifications that adapt to user modeling use case. The resulting model solves an important problem in RecSys where we usually lack explicit feedback on the objective that need to be optimized, and implicit feedbacks are sensitive to user subgroups such that ad-hoc reward engineering is undesirable.
- The application of Inverse Kinematics on IGL problem is novel, and it is technically non-trivial.
- The introduction of 3-state reward decoder is non-trivial and interesting.
- The result on synthetic feedback scenario shows improved fairness, and the algorithm is comparable to SOTA production candidate with hand-crafted reward engineering.

Weakness:

Despite the novelty and contributions, there are some ambiguities that may need further clarification

- When and how is the DN(…) function learned? It seems PU-learning is used where only negative labels are partially provided, and the labeling probability is somehow known. But in the empirical experiments it is unclear how these set of information is obtained. Moreover, it is non-trivial to estimate labeling probability in real-world, and there is not much explanation on how this hyperparameter is chosen and how it impact the performance.
- The synthetic experiments with Facebook and Covertype datasets are designed such that feedbacks are heavily dependent on user type, making it in favor of the proposed algorithm. And only simple CB are used as baselines for comparison. The real-production result is comparable to reward-engineering, while the significance of the lift is not clear as the confidence lower bound <1 for likes and there is a positive lift in dislike. Comparison to action-inclusive IGL in Xie et al 2022 is also lacking, making it less convincing whether the user dependence assumption is the main reason for the lift.
- The authors propose different assumption and learning algorithm than vanilla IGL. Yet no theoretical analysis is provided to prove whether the new independence assumption and the learning algorithm converges to the optimal solution. The enhancement on fairness is also supported with only empirical evidence, thus might limit the impact of the work.

Minor issues:

- There might be a typo in the last line of equation (2) where the prior on numerator is `p(a,x)` whereas in equation(3) it is `p(a|x)`.
- Missing figure for learning curves for IK, reward and policy learning. 

clarity,_quality,_novelty_and_reproducibility: The quality and originality of the work is high, while the writing can be further improved for clarity. Especially, Algorithm 1 is not very informative. Instead of using ‘.predict’ and ‘.learn’, it would be clearer if the author provide the exact loss function and steps used to train KI and Policy, as for RecSys audience who are less familiar with KI it is not obvious how KI is learned.
summary_of_the_review: The paper proposes novel solution for personalized reward learning with implicit feedback in IGL paradigm, and come up with important modification of original IGL (Xie et al 2022). Despite limited theoretical result and baselines, the contribution is potentially significant to the community.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper posits that users communicates their true preferences differently, thus relying on a fixed, human-designed reward function on implicit user feedbacks to come up with the underlying user preferences is sub-optimal. The authors proposed to combine the new Interaction Grounded Learning (IGL) paradigm with inverse kinematics to learn personalized/context-dependent reward function, and showed promising results in offline simulated and live experiments. Reviewers converged to a recommendation of marginally above acceptance threshold after rebuttal. Overall reviewers appreciate the application of IGL to address the challenge of lacking explicit feedback and the need of reward engineering in recommender systems, and ask for more details on the implementations and experiments. 

justification_for_why_not_higher_score: Missing details on implementations and experiments. Only compare to simple baselines. Might not have a big audience at the conference.
justification_for_why_not_lower_score: Overall reviewers appreciate the application of IGL to address the challenge of lacking explicit feedback and the need of reward engineering in recommender systems.
Decision: Accept: poster