Title: IGD: Token Decisiveness Modeling via Information Gain in LLMs for Personalized Recommendation
Venue: NeurIPS 2025
Accepted: yes
Keywords: Personalization

LLM-based Recommendation
Abstract: Large Language Models (LLMs) have shown strong potential for recommendation by framing item prediction as a token-by-token language generation task. However, existing methods treat all item tokens equally, simply pursuing likelihood maximization during both optimization and decoding. This overlooks crucial token-level differences in decisiveness—many tokens contribute little to item discrimination yet can dominate optimization or decoding.
To quantify token decisiveness, we propose a novel perspective that models item generation as a decision process, measuring token decisiveness by the Information Gain (IG) each token provides in reducing uncertainty about the generated item. Our empirical analysis reveals that most tokens have low IG but often correspond to high logits, disproportionately influencing training loss and decoding, which may impair model performance.
Building on these insights, we introduce an Information Gain-based Decisiveness-aware Token handling (IGD) strategy that integrates token decisiveness into both tuning and decoding. Specifically, IGD downweights low-IG tokens during tuning and rebalances decoding to emphasize tokens with high IG. In this way, IGD moves beyond pure likelihood maximization, effectively prioritizing high-decisiveness tokens. Extensive experiments on four benchmark datasets with two LLM backbones demonstrate that IGD consistently improves recommendation accuracy, achieving significant gains on widely used ranking metrics compared to strong baselines. Our codes are available at \url{https://github.com/ZJLin2oo1/IGD}.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes an Information Gain-based Decisiveness-aware Token Handling (IGD) strategy for Large Language Model-based Recommendation (LLM4Rec), targeting the issue that existing methods usually treat all item tokens equally, ignoring differences in their decisiveness. The authors treat the item generation process as a decision process and use Information Gain (IG) to quantify each token’s decisiveness. They observe that many tokens have low IG but high logits, introducing bias in both training and decoding. IGD addresses this by down-weighting low-IG tokens during tuning and decoding to emphasize high-IG tokens—shifting the focus from pure likelihood maximization to decision-critical tokens. Experiments on four benchmark datasets with two LLM backbones show that IGD consistently improves recommendation accuracy.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\nS1. Measuring the importance of item tokens from the perspective of “decisiveness” is interesting and introduces a novel angle.\n\nS2. The authors treat the item generation process as a decision-making process and define token decisiveness through information gain. This definition is both novel and well-justified.\n\nS3. The proposed method is applicable during both the tuning and inference stages, allowing it to emphasize high-decisiveness tokens throughout the modeling pipeline.\n\nS4. The paper is well-motivated: it begins with a preliminary experiment to illustrate the problem, followed by the introduction of the proposed method.\n\nS5. Extensive experiments on four benchmark datasets, using two LLM backbones (Qwen2.5 and LLaMA3), demonstrate consistent performance improvements.\n\nWeaknesses:\n\nW1. Currently, tokens are divided into only two categories based on information gain. While this binary classification is reasonable, a more fine-grained categorization—guided by the magnitude of information gain—may lead to a better results.\n\nW2. Figure 1 is difficult to follow. Additionally, the layout of Section 3 could be made more compact to improve clarity and readability.\n\nW3. The candidate item distribution is not clearly explained and requires further explanation.\n\nW4. A concurrent work [1] also models item generation as a decision tree, which bears some relevance to this paper. A discussion of this related work should be included to clarify distinctions and highlight contributions.\n\n[1] Gao et al. Process-Supervised LLM Recommenders via Flow-guided Tuning.\n\nQUESTIONS:\nQ1. Has the author considered a more fine-grained token categorization, beyond simply distinguishing between zero and non-zero information gain tokens? If so, what are the results?\n\nQ2. What is the candidate item distribution? Is it determined by the frequency of item occurrences in the interaction dataset?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-06-27"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work improves LLM-based recommendation systems, which traditionally treat all tokens equally and maximize likelihood, by introducing a reweighting approach that prioritizes more informative tokens based on their importance. Experiments were conducted on four datasets and two backbone models, demonstrating substantial improvements in recommendation performance.\n\nSTRENGTHS AND WEAKNESSES:\n- **Quality**: The paper clearly identifies a key limitation of existing LLM-based recommendation systems and proposes an appropriate solution. However, the approach appears computationally intensive and likely to incur significant training and inference time.\n- **Clarity**: The research idea, mathematical formulations, and experimental results are all described clearly and precisely.\n- **Significance**: The paper proposes a practical improvement to the bias issue in token-level generation and convincingly demonstrates strong performance through extensive experiments. However, since the evaluation is limited to Amazon datasets, it would be important to validate the approach in other domains.\n- **Originality**: The token decisiveness-based training and decoding strategy is novel. It highlights the potential need for similar decoding mechanisms in existing LLM-based recommendation models.\n\nQUESTIONS:\n1. Compared to previous models, this approach seems to involve more computation. Could you share approximately how much training and decoding time it requires in practice?\n2. During decoding, as the IG weighting increases, it seems likely that the model would increasingly prioritize informative tokens. Did you observe any issues with reduced diversity or over-concentration on certain items as a result?\n\nLIMITATIONS:\nYes, the authors have discussed the main limitations of their work. However, I think the scalability to real-time applications could also be acknowledged.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces a novel method on using large language models (LLMs) for recommendation. In particular, the authors study the entropy / information gain (IG) of different type of tokens and found that existing methods can cause the model to over-focus on zero-IG tokens while under-emphasizing non-zero-IG tokens during tuning. To address this, the authors model the item-generation process as a sequential decision task and weight the loss of each token in training depending on its IG. In addition, the authors also modify the generation process by revising the scores based on their IG, and therefore mitigating the bias toward high-logit but low-decisiveness tokens. The authors experiment on different datasets and compare to multiple benchmarks, where the proposed method can be applied to different eixsting approaches and show performance improvements.\n\nSTRENGTHS AND WEAKNESSES:\nPros:\n\n1. The proposed method is interesting and studies an important problem from the information theory perspective, i.e., how IG / entropy affect the decoding process and influcenes the downstream task performance.\n\n2. The proposed method is somewhat intuitive and reasonable, i.e., non-zero IG tokens with low logit values can be more effective to optimize in trainning. In addition, this method is simple and can be applied to different existing methods such as BIGRec and D3.\n\n3. The authors experimented on multiple datasets, demonstrating its performance over different baseline methods.\n\nCons:\n\n1. The authors only experimented on the RecSys setting with limited datasets, although this setting should naturally extend to more generaly language modeling tasks.\n\n2. The proposed method intriduced additional hyperparameter complexity in both training and decoding (e.g., beta and alpha), increasing the hyperparameter searching costs while only providing marginal gains in many cases.\n\n3. The selected RecSys datasets are all from amazon and rather small in sizes. I encourage the authors to experiment LLMs on larger datasets, potentially with multimodal inputs to comprehensively evaluate the proposed method. More baseline methods from both traditional sequential methods (LRURec etc.) and LLM methods (TallRec etc.) shouls also improve the quality of the experiments.\n\nQUESTIONS:\n1. How does the proposed method apply to different decoding paradigms like sampling / constrained beam search, or does it work with a more genrealized sampling setting with different logits processing (top-k, top-p, repetition penalty etc.)?\n\n2. Is the proposed method sensitive to differen decoding settings? Especially with key parameters like temperature.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposed a re-weighting method for both tuning and decoding for LLM4Rec setting (directly use LLM as recommender to generate next-item), based on the insight the some tokens doesn’t reduce the entropy of the to-be-generated item but are put on high focus in both tuning and decoding. Specifically, the paper proposed a metric called information gain to determine the importance of the token, and put larger weight on high importance token while lower weight on less important ones. The paper performs experiments on benchmark data and achieves good results.\n\nSTRENGTHS AND WEAKNESSES:\nStrength:\n* The paper has simple intuition and the experiment results are promising on several datasets.\n* The proposed IG metric is simple to calculate. The overall algorithm is easy to be plugged into existing methods.\n\nWeakness:\n* The insight seems great, but the approach is quite heuristic but not mathematically motivated; it’s not clear how to tune the hyperparameters beta and alpha, and there is no empirical studies on the sensitivity of such hyperparameters \n* The IG metric is specifically pre-defined through training dataset, and might be hard to generalize in real recommendation scenarios.\n* The experiment are not convincing regarding to the IG metric, specifically the setup seems biased towards to proposed method (i.e. removing tail item to make sure training and testing distribution are more similar/similar type recommendation(i.e. CD, Games, etc)).\n\nQUESTIONS:\nThe method is simple and follows the intuition. My main concern is the setup and motivation.\nFrom equation 5, during SFT, the model will focus more on the token that can dramatically decrease the entropy for token output, and the entropy is calculated using the train dataset. \nFrom Table 1, the test dataset is very small compared to the train data, it might be hard to generalize the w_t if test data is large (i.e. in practice, testing data is much more compared to SFT data). In this case, the estimation of IG on training data can be very different from the actual potential distribution. Moreover, if during testing, there is a new item with a new title, then this method might not work. As in 5.1 first paragraph mentioned, the dataset is truncated in a way that each user and item has at least 5 interactions. Although this is used by another paper (one of the baseline), this kind of process actually leads to implicit leakage since this ensures the statistics in the training dataset can be leveraged to test the dataset. \n\n\nIn eq (7): instead of introducing a heuristic hyperparameter beta, why not make the weight a continuous and monotonic function of IG? \nHow to tune beta and alpha?\n\n\nwhy are there tuning bias and decoding bias? more in-depth analysis on why this is the case would be helpful\n\nLIMITATIONS:\nThe experiment setup does not reflect the real recommendation scenarios. See above comments.\n\nthe weighting approach seems highly heuristic but not mathematically motivated, and not clear how to tune the hyperparameters and no studies on the sensitivity of those", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-07"}