Fine Tuning Out-of-Vocabulary Item Recommendation with User Sequence Imagination

Authors: ruochen  liu
hao  chen
yuanchen  bei
qijie  shen
fangwei  zhong
senzhang  wang
jianxin  wang
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 06:59:09

Abstract: Recommending out-of-vocabulary (OOV) items is a challenging problem since the in-vocabulary (IV) items have well-trained behavioral embeddings but the OOV items only have content features. Current OOV recommendation models often generate 'makeshift' embeddings for OOV items from content features and then jointly recommend with the `makeshift' OOV item embeddings and the behavioral IV item embeddings. However, merely using the 'makeshift' embedding will result in suboptimal recommendation performance due to the substantial gap between the content feature and the behavioral embeddings. To bridge the gap, we propose a novel **User Sequence IMagination (USIM)** fine-tuning framework, which first imagines the user sequences and then refines the generated OOV embeddings with the user behavioral embeddings. Specifically, we frame the user sequence imagination as a reinforcement learning problem and develop a recommendation-focused reward function to evaluate to what extent a user can help recommend the OOV items. Besides, we propose an embedding-driven transition function to model the embedding transition after imaging a user. USIM has been deployed on a prominent e-commerce platform for months, offering recommendations for millions of OOV items and billions of users. Extensive experiments demonstrate that USIM outperforms traditional generative models in OOV item recommendation performance across traditional collaborative filtering and GNN-based collaborative filtering models.

Main paper: 
 introduction: 
 Recommendation systems, such as collaborative filtering models, learn behavioral embeddings from historical interactions to represent the behavioral characteristics of billions of users and items [1] [2] [3] [4] . For instance, the embeddings of interacted user-item pairs have higher inner products, whereas those of un-interacted pairs have lower inner products. However, besides the items with user interactions, thousands of out-of-vocabulary (OOV) items-such as short videos, photos, and posts-are generated or uploaded every second. In the age of AGI, the generation speed of AI-made OOV content, including text, images, and videos, will far exceed the speed of human consumption. To avoid being overwhelmed by the OOV content, it is essential to replicate how humans handle these items, recommending them without disrupting in-vocabulary (IV) items.Traditional OOV recommendation models usually generate 'makeshift' embeddings from the content features and then use them to recommend OOV items. The research can be classified into two categories. (a) Generative models aim to generate realistic embeddings. GAR [5] uses a generative adversarial structure to ensure the embedding distribution of generated OOV embeddings is similar to IV embeddings. ALDI [6] distills knowledge from IV items to OOV items. (b) Dropout models increase the robustness of recommender systems. Dropout [7] randomly substitutes IV embeddings with "makeshift" ones to enhance system robustness. Heater [8] and CLCRec [9] further utilize a mix of experts and contrastive learning techniques to improve OOV recommendation performance.As shown on the left of Figure 1 , current recommender systems typically learn the embeddings for any given items by initializing/generating the embedding and then optimizing them through usersequence backpropagation. However, prevalent OOV recommendation models primarily concentrate on generating improved or robust embeddings, overlooking the potential for further optimization from imagining user sequences. This oversight limits these models due to the following issues. 1. Content-Behavior Gap. 'Makeshift' embeddings are generated from content features, while behavioral embeddings are trained using backpropagation. The substantial difference between content features and behavioral embeddings may lead to discrepancies between IV and OOV items, impacting IV, OOV, or both. 2. Potential Suboptimality. Focusing only on embedding generation overlooks potential improvements from backpropagation, which could finely tune the embeddings to adapt to user preferences and current recommender systems, possibly leading to suboptimal recommendation performance and a reduction in revenue.While imagining the sequential optimization process for OOV item embeddings shows promise, its implementation presents three challenges. 1. Absence of Historical Interactions. The lack of historical user interactions for OOV item embeddings hinders the definition of a clear backpropagation and optimization process. 2. Ambiguous Imagination Objectives. Formulating objectives, stopping criteria, and user selection for imagining OOV item interactions remains an open challenge. 3 . Navigating the Vast User Space. Efficiently identifying suitable user sequences to imagine for OOV items within the massive user space of recommender systems.To address these challenges, we introduce an User Sequence Imagination (USIM) pipeline that further optimizes the embedding of OOV items. It first imagines potential users who may interact with the OOV item and then refines the item embeddings through backpropagation. Specifically, we propose a RL-based USIM solution, which formulates the sequential optimization as a Markov Decision Process and introduces recommender-oriented PPO (RecPPO) to maximize the final recommendation performance of OOV items. In summary, our contributions are as follows:• We introduce USIM, a novel approach that fundamentally addresses the Out-of-Vocabulary (OOV) problem by imagining user sequences and performing user-sequence backpropagation.• We formally define the Reinforcement Learning (RL) formulation of USIM, including the formulation of the Markov Decision Process (MDP) for user sequence imagination, and provide the state transition function framework for user-sequence backpropagation.• We implement USIM on a major e-commerce platform-Alibaba, successfully optimizing millions of OOV items and recommending them to billions of users. The source code is publicly available at https://github.com/Ruochen1003/USIM.• We validate the effectiveness of our approach on two benchmark datasets using both traditional collaborative filtering and graph-based collaborative filtering backbones. Extensive experiments demonstrate that USIM outperforms existing state-of-the-art OOV recommendation models in terms of OOV recommendation performance and overall recommendation quality. 
preliminaries: 
 Notations. Let U and I denote the sets of users and items, respectively. We partition I into in-vocabulary (IV) items I iv (items with interaction history) and out-of-vocabulary (OOV) items I oov (items without interaction history). We denote the cardinality of these sets as |U|, |I iv |, and |I oov |, respectively. For clarity in the following discussion, we also use U i to represent the set of users who have interacted with item i.The embedding matrices for users, IV items, and OOV items are denoted asE u ∈ R |U |×d , E iv ∈ R |Iiv|×d, and E oov ∈ R |Ioov|×d , respectively, where d represents the embedding dimension. For an individual user u and item i, their corresponding embeddings are denoted as e u ∈ R d and e i ∈ R d . Since OOV items lack behavioral embeddings initially, we leverage content features, denoting the content feature vector of item i as c i .Backpropagated Embedding of IV Items. The embeddings of IV items are initialized using standard initialization techniques such as Xavier initialization [10] . These embeddings are subsequently optimized through backpropagation using historical interaction data [11, 4] ,EQUATIONwhere L represents the loss function (e.g., BPR loss).Makeshift Embedding of OOV Items. Due to the absence of historical interactions for OOV items, various approaches have been proposed to generate makeshift embeddings that enable joint recommendation with IV items. These approaches can be broadly categorized into generative models [6, 5, 12] and dropout models [9, 7] , which transform content features c i into embeddings through a generator function G,EQUATIONwhere G is optimized using various objective functions, including similarity-based losses [12] , adversarial losses [5] , and knowledge distillation losses [6] .MDP Formulation of Back Propagation. We formulate the OOV embedding optimization process as a Markov Decision Process (MDP) to narrow the optimization gap between IV and OOV item embeddings. This formulation enables simultaneous user imagination and embedding optimization through backpropagation.An MDP at time step t is defined by the quintuple (S, A, ρ, R, γ), where:• S represents the state space, with each state s ∈ S capturing the environment configuration• A denotes the action space, where each action a ∈ A represents a possible agent decision• ρ : S × A → S defines the state transition function, with s t+1 = ρ(s t , a t )• R : S × A → R specifies the reward function, where r t = R(s t , a t )• γ ∈ [0, 1] represents the discount factor for future rewards 3 Proposed User Sequence Imagination Model 
framework overview: 
 To bridge the fundamental gap between IV and OOV item embedding generation caused by disparate interaction histories, we propose USIM, which fine-tunes OOV item embedding by imagining appropriate user sequences. Specifically, in each step, USIM simulates a user interaction and optimizes the item embedding accordingly. We formulate this process within a reinforcement learning paradigm, as illustrated in Figure 2 . State Space. At time step t, the state s t encapsulates the essential information required for item embedding optimization, defined as s t = [h t , l t ]. The state representation h t ∈ R d , which is refined through optimization, resides in the same embedding space as e i , and its final representation will serve as the OOV item embedding. To specifically denote the state of item i at time step t, we define s i,t = [h i,t , l i,t ]. In contrast, s t serves as a more general denotation. The initial state presentation is obtained by the generator in Eq. ( 2) as h i,0 = G(c i ). Details can refer to Appendix B.The temporal component l t is used as a countdown mechanism, tracking the remaining optimization steps to encourage efficient convergence [13] . Given a maximum action limit N , at time step t, the countdown value is computed as l t = N -t.Action Space. Given state s t , the agent selects an action a t from the action space A = U ∪ {a end }, where a end denotes the termination action. This selection process entails either imagining a user or terminating the optimization process. The action embedding e a corresponds to the user embedding from E u when a ∈ U, and defaults to 0 for the termination action.Policy Network. The agent's decision-making process is governed by policy π(s t ), which maps the current state s t to a probability distribution over possible actions. To effectively utilize existing embeddings while accommodating the special termination action, the policy distribution is given as,EQUATIONwhereW 1 ∈ R d×(d+1) , W 2 ∈ R d+1, and c are parameters, and σ represents the sigmoid function.Reward and State Transition. The agent receives an immediate reward r t = R(s t , a t ) after each action, guiding the optimization trajectory. To align with the imagination process, we design an efficient state transition function s t+1 = ρ(s t , a t ) that facilitates the progressive refinement from content-based to interaction-based embeddings. The experience tuples (s t , a t , r t , s t+1 ) are collected in a replay buffer for subsequent training iterations.State transition involves modifying h t , which will ultimately be used as the OOV item embedding and each step of the state transition corresponds to optimizing h t using the imagined user(last action a t ). Therefore, to design a state transition function that aligns with this optimization process, we must first determine the objective of this optimization. Considering that most current recommendation algorithms calculate relevance scores between users and items for recommendations [11, 14, 4] , we adopt the following objective as our optimization goal,EQUATIONwhere ŷu,i is predicted score between e i and e u .Based on the above idea, we can view the user imagination process as the solution to the optimization objective (Eq. ( 4)). Specifically, we assume that the users imagined by the agent are the users who have interacted with this item, and by using backpropagation, we can optimize the content-based initialized embeddings. Thus, the transition function ρ h of h t can be written as follows,EQUATIONwhere λ is a hyperparameter and can be understood as the learning rate. If ŷat,i is computed using dot product, the final transition function of h t can be written as follows,EQUATIONAnd l t can be updated as l t+1 = l t -1.The objective of the USIM is to optimize the initialized embeddings by imagining user sequences. To facilitate this process, we design a reward function to guide the reinforcement learning approach. Our reward function consists of three components.Embedding Alignment Reward. We believe that the item embeddings generated by the IV model represent the best solution for our optimization process. Consequently, our objective is to closely align the final state representation with actual item embedding (i.e., the corresponding embedding from the IV model). To achieve this, we calculate the reward based on the concept of similarity,EQUATIONwhere D(•, •) denotes the Euclidean distance between embeddings. This reward represents the change in similarity between the state representation and the actual item embedding, before and after the state transition.Recommendaion Performance Reward. Although the embedding alignment reward encourages the state representation h t to be close to the actual item embedding, it does not differentiate between states when multiple representations are equally distant from the actual embedding. Additionally, it does not fully utilize the insights from existing user interactions. Therefore, we design a reward function based on recommendation performance as follows,EQUATIONwhere f (h i,t , e i ) represents the predictive power of state representation h i,t for users in U i . The final performance reward is derived from the change in f before and after the state transition. This change represents the variation in the embedding's predictive capability.Step Regulation. To encourage the agent to achieve the goal in as few steps as possible, we impose a penalty for each action it takes. Therefore, the final reward function is as follows,EQUATIONwhere p is a hyperparameter that represents the penalty.Given the large user base in the recommendation dataset, sampling actions solely by probability during initial reinforcement learning often results in negative rewards, slowing convergence and reducing performance. To explore actions more efficiently, we construct an exploration set according to state s t , comprising three components.Positive Action. In the USIM framework (Section 3.2), the agent assumes that the imagined users correspond to those who have interacted with the item, so these users are included in the exploration set. While optimizing h i,t to match e i theoretically requires only the interaction set U i , achieving this often demands combining multiple actions a ∈ U i , making exploration complex. To streamline this, we select users most likely to bridge h i,t and e i , accelerating the process. The Positive Action set U pos is constructed as follows,EQUATIONwhere Top k1 ((e i -h i,t ), e u ) represents the set of k 1 users with the highest cosine similarity to e i -h i,t , forming a vector that directly points to the actual item embedding.Random Action. Relying solely on the aforementioned action sets may overly restrict actions, limiting state space coverage and reducing model generalization. Moreover, sampling actions with negative rewards can also benefit training [15] . Thus, we augment the action set by randomly selecting k 2 actions from the remaining pool, denoted as U rad .Termination Action. To enable the agent to learn when to terminate, we also incorporate the termination action a end into the final action set. This inclusion allows the agent to determine the appropriate timing for ending the optimization process.For simplicity, we set k 1 = k 2 = k. And exploration set A samp can be represented as follows,EQUATIONDuring the training phase, as the sampling range is narrowed from the full action set A to a specific action set A samp , we rewrite the Eq. ( 3) as follows,EQUATIONWe incorporate recommendation-specific supervision signals into PPO [16] , referring to this enhanced approach as Recommender-Oriented PPO (RecPPO), to train our USIM. In the scenario of OOV item recommendation, the optimal action following certain states is clear, allowing the cumulative expected rewards for these states to be calculated directly. When the state representation h i,t of a specific item is equal to its item embedding e i , according to our designed reward function, the expected value should be 0 because any subsequent actions, except for termination will lead to negative rewards.So we use these supervision signals to assist in training the value network V ω , the specific loss function of the value network is defined as follows,EQUATIONwhere B denotes tuples sampled from the buffer pool, and random(0, N ) is a random number between 0 and N . The first term of the loss is the Temporal Difference loss used in value network training, while the second term includes our recommendation-oriented supervision signals. Regardless of previous actions, when the state representation h i t matches e i , the agent should terminate. Here, random(0, N ) acts as the timer for each termination state. As for the policy network, we train it using the same method as PPO. The whole training process can be found in Appendix C. 
experiments: 
 We conduct comprehensive experiments on two benchmark datasets aiming to address the following three questions: RQ1: Can USIM achieve superior OOV item recommendation performance compared to state-of-the-art OOV item recommendation models? RQ2: How key components of USIM affect its performance? RQ3: Is the proposed USIM more effective than representative RL methods? RQ4: What is the tendency of performance during USIM's imagination process? RQ5: How does USIM perform in real-world industrial recommendations? RQ6: How does USIM achieve efficiency compared to other baselines? Datesets. We evaluate the performance of USIM on OOV items using the widely used datasets: CiteULike [8] and MovieLens [17] . Specifically, CiteULike contains 5,551 users, 16,980 articles (items), and 204,986 interactions. MovieLens comprises 6,040 users, 3,883 movies (items), and 1,000,210 interactions. Details about these datasets are shown in Appendix D.1.Baselines.To assess the effectiveness and universality of USIM, we conduct a comparative analysis with 8 leading-edge models in OOV item recommendations across two distinct datasets. These models include two main groups. (i) Dropout-based methods: DropoutNet [7] , MTPR [18] , Heater [8] , and CLCRec [9] . (ii) Generative-based methods: DeepMusic [12] , MetaEmb [19] , GAR [5] , and ALDI [6] . Details about these models are shown in Appendix D.2. To further verify the generalization ability, we adopted both the widely used collaborative filtering model MF [20] and GNN-based model NGCF [3] as the recommender, respectively.Evaluation Metrics. Following the evaluation of existing OOV recommendation [6, 9] , we conduct three different tasks in our experiments: (1) Overall Recommendation, (2) OOV Recommendation, and (3) IV Recommendation. We employ the full-ranking evaluation approach to assess the performance of overall, IV, and OOV recommendations. Following previous works [4, 3] , we utilize Recall@K and Normalized Discounted Cumulative Gain (NDCG@K) as metrics.Implementation Details. We implement the baselines using their officially provided version. See Appendix E for the detailed implementation. The best hyperparameters are found for each dataset.For fairness, we use the same options and follow the designs in their articles for all baselines. The main comparison results of overall, IV, and OOV item recommendation results can be found in Table 1 . From the results, we can have the following observations. USIM can generally achieve significant improvements over state-of-the-art methods on both overall and OOV item recommendations while keeping the IV item recommendation. From the tables, we observe that the USIM achieves the highest average Recall and NDCG performance across both MF and GNN recommenders. These comparison results verify the superiority of the imagined embeddings over traditional one-step generated embeddings.Dropout-based baselines have the performance drop in the IV item recommendation. We find that the USIM and the generative-based models can keep the IV item recommendation. However, dropout-based models will lead to a performance drop on IV items. This suggests that there is a difference between OOV items and IV items. Pre-training representations of IV items in advance to generate representations for OOV items may be better for retaining information for IV items.To validate the effectiveness of the individual components in our model, we compared the full model against four variants: (i) w/o ct removes the cosine similarity-based top-k user selection when constructing the positive action set. (ii) w/o ra removes the randomly sampled actions in the exploration set. (iii) w/o es does not construct an exploration set and directly explores the entire action set. (iv) w/o pr removes the performance reward, only using the similarity reward in the RL processing. From the results in Table 2 , we make the following observations. Effectiveness of components in the exploration set construction. The w/o ct result indicates that the cosine similarity-based selection of the top-k users can simplify the exploration process by quickly identifying users related to the item embedding. Then, the performance of w/o ra reveals the importance of including negative samples to provide a more comprehensive exploration signal, preventing the agent from overestimating action values by only considering positive feedback. Further, the poor performance of the w/o es approach highlights the challenge of effectively exploring an extremely large action space, emphasizing the need for a well-designed exploration strategy to guide the agent towards promising actions.Effectiveness of components in the reward function, Removing the performance reward component leads to inferior results. This suggests that the reward function should not only consider the distance between the generated embedding and the target embedding but also explicitly incorporate the downstream recommendation performance of the generated embedding.In Figure 3 , we compared our proposed model with our initialized MLP and two other traditional reinforcement learning methods: Wolpertinger Policy [21] (WP) and Hierarchical Reinforcement Learning [22] (HRL). WP utilizes the similarity of action representations and value estimation to solve the large discrete action space problem in reinforcement learning, while HRL employs a hierarchical decomposition of action space and sub-task solution to improve exploration efficiency.Our model significantly outperforms the other two methods in both OOV recommendation and overall recommendation. Moreover, it can be observed that the other two methods do not show apparent improvements compared to their initial states. This supports the effectiveness of our tailored exploration method in improving performance in OOV recommendations.To investigate the performance trend of our proposed method during the optimization process, and to validate the importance of the way of sampling in the embedding optimization process, we compare USIM with the following two user selection strategies: randomly sampling users at each step, and randomly selecting from the top 20 users with the highest relevance scores at each step. The results are shown in Figure 4 , and according to the result, we can draw the following conclusions.(i) Our method outperforms the random selection of top-20 users and the entire user set in both OOV and overall scenarios, indicating it can effectively identify users beneficial for optimization. However, performance declines as more users are imagined, likely due to reduced exploration by the agent.(ii) When randomly selecting users from the top 20, we can observe that the performance in OOV recommendation increases after the first step. This suggests that sampling from high-scoring users is more likely to select users who are beneficial to the optimization process. However, the performance then continuously declines, indicating that this approach is not suitable for all states and has limited help for the optimization process.(iii) Randomly selecting a user at each step leads to a drastic and non-recoverable decline in performance in both the OOV and overall scenarios after the first step. This suggests that in recommendation scenarios, it is extremely difficult to sample users from the massive user set who are beneficial to the optimization, and the majority of users are highly detrimental to the optimization process.More experimental hyperparameter analysis can be found in Appendix F.To evaluate the performance of USIM in an industrial setting, we conducted a two-week online A/B test on a major e-commerce platform with 5% of users in each group. USIM was compared against three baselines: Random, MetaEmb [19] , and ALDI [6] . Details about our test platform and evaluation metrics are provided in Appendix G. Table 3 presents the results of these online A/B tests.These remarkable improvements across all metrics underscore the effectiveness of the USIM in addressing the OOV item recommendation problem in real-world recommender systems. The consistent and substantial performance gains, particularly in OOV item GMV, highlight the practical impact of our approach on business outcomes in e-commerce settings.To More experiments about online recommendation efficiency can be found in Appendix H. 
conclusion: 
 Recommending out-of-vocabulary (OOV) items is challenging due to the lack of well-trained behavioral embeddings. Current models use "makeshift" embeddings from content features, leading to suboptimal performance. We introduced the User Sequence Imagination (USIM) framework to refine OOV embeddings by imagining user sequences and incorporating behavioral embeddings. By framing this as a reinforcement learning problem and creating a recommendation-focused reward function, USIM effectively enhances OOV recommendations. Extensive experiments demonstrate its superior performance and the ablation study further illustrates the effectiveness of USIM. 
experimental setting/details: 
 Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes] Justification: The experimental settings are presented in the core of the paper. And full details are provided appendix.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in the appendix, or as supplemental material. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [Yes]Justification: We have statistical significance in the main experiment.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).• The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. Justification: We checked and ensured that our paper conforms with the NeurlPS Code of Ethics in every respect.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [NA] there is no societal impact of the work performed Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.• Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] the paper poses no such risks.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes]We cite the original paper that produced the code package or dataset.We state which version of the asset is used Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.• If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Answer: [NA]Justification: the paper does not release new assets Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 
crowdsourcing and research with human subjects: 
 Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?Answer: [NA] Justification: the paper does not involve crowdsourcing nor research with human subjects Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector. 
institutional review board (irb) approvals or equivalent for research with human subjects: 
 Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?Answer: [NA]Justification: the paper does not involve crowdsourcing nor research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.• We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: General Recommendation. General recommendation systems typically predict which items users will prefer by leveraging collaborative information. Based on how collaborative information utilized, general recommendation methods typically include approaches such as Matrix Factorization-based, Graph-based, and Sequential recommendation. Matrix Factorization-based recommendation methods derive user and item embeddings by decomposing the interaction matrix into two feature matrices using Matrix Factorization techniques [11, 23, 24] . Graph-based recommendations incorporate graph techniques [25] to model high-order relationships between users and items [4, 3, 26] . Sequential recommendation methods focus on capturing the temporal patterns in user interactions, modeling the order of users' actions to predict their future preferences [27, 28, 14, [29] [30] [31] [32] . However, these methods are generally ineffective in addressing the OOV item,Out-of-Vocabulary Item Recommendation. Out-of-vocabulary (OOV) item recommendation aims to address the problem of recommending a completely new item that has no prior user interactions with users [33, 34] . Its core idea lies in how to map the content information of the cold item into the space defined by the warm item embeddings trained from interactions with users. To achieve this goal, existing methods can be mainly categorized into two approaches. One category is the dropout model, which learns a content map function using the dropout approach [18, 35, 36, 8, 7, 9] . The other category is the generative model, which directly learns the relationship between the content of the OOV item and the IV behavior embeddings to obtain the mapping function [6, 12, 37, 5] .Reinforcement Learning-based Recommendation. Reinforcement learning (RL) is a famous type of learning strategy where an agent learns to make decisions by taking actions in an environment to maximize some notion of cumulative reward, which is a widely used technique for modeling human behavior [38] [39] [40] [41] [42] . Many studies also focus on simulating user behavior within the recommendation community. Cascading DQN [43] uses GAN to learn and simulate real users from historical interactions to obtain the reward function. In [15, 44] , the simulator is trained on user historical data to simulate user feedback. In Pseudo Dyna-Q [45] , a world model (user simulator) is trained by minimizing the error between online and offline rewards. Another common simulation approach is based on collaborative filtering. LIRD [46] builds a memory with (s, a, r) tuples seen in the log dataset and uses a similarity method based on cosine similarity to find the closest state-action pair to the current state and recommended action. DRR [47] and DRGR [48] use the same intuition but based on different factorizations, respectively.Loss function Our initial model G is an MLP trained to minimize the Euclidean distance between the output of G and the item embedding. Specifically, the loss function for G is defined as follows:EQUATION)Training details The initial model shares the same training data as USIM. We use the Adam optimizer with a learning rate of 0.001 and apply early stopping by monitoring NDCG@K on the validation set. The batch size and regularization weight are set to 1024 and 0.001, respectively.Our complete training process is shown in Algorithm 1.Datasets. We evaluate the USIM 's performance on cold-start items using the CiteULike and MovieLens datasets. The goal is to leverage these libraries to recommend relevant new articles to each user. The articles are represented by 300-dimensional vectors as item content features.• MovieLens3 [17] MovieLens comprises 6,040 users, 3,883 items, and 1,000,210 interactions. The content features of items are represented using 200-dimensional vectors.In this paper, the content features of items are represented using 200-dimensional vectors. For each dataset, 20% of items are designated as OOV items, with interactions split into a OOV validation set and testing set (1:1 ratio). Records of the remaining 80% of items are divided into training, validation, and testing sets, using an 8:1:1 ratio.Baselines.To assess the effectiveness and universality of USIM, we conducted a comparative analysis with 8 leading-edge models in the domain of cold-start recommendations. This comparison was carried out across two distinct datasets. The models we benchmarked against include two main groups: (i) Dropout-based models: DropoutNet [7] , MTPR [18] , Heater [8] , and CLCRec [9] . (ii) Generative models: DeepMusic [12] , MetaEmb [19] , GAR [5] , and ALDI [6] .• DeepMusic utilizes deep neural networks to model the mean squared error (MSE) difference between generated and warm embeddings.• MetaEmb trains a meta-learning-based generator for rapid convergence.• GAR generates embeddings through a generative adversarial relationship with the warm recommendation model.• ALDI employs distillation, using warm items as "teachers" to transfer behavioral information to cold items, referred to as "students".• DropoutNet enhances cold-start robustness by randomly discarding embeddings.• MTPR generates counterfactual cold embeddings considering dropout and Bayesian Personalized Ranking (BPR). • Heater improves DropoutNet by using a mix-of-experts network and considering embedding similarity.• CLCRec models cold-start recommendation using contrastive learning from an informationtheoretic perspective.We implement the baselines using their official implementations. Specifically, for GAR, we use the updated version provided in the official repository, which is evaluated under the same CLCRec settings as used in our paper 4 . The embedding dimension is set to 200 for all models. We employ the Adam optimizer with learning rates of 0.001 for the critic and 0.0005 for the actor. Early stopping is applied by monitoring NDCG@K on the validation set. The training batch size and regularization weight are set to 1024 and 0.001, respectively.The experiment was conducted on an NVIDIA GeForce RTX 3090 with 24GB of memory. Hyperparameters are tuned using grid search, and the optimal parameters are identified for each dataset. For fairness, we use the same settings and follow the design choices in their respective articles for all baselines.We conducted a parameter analysis on the CiteULike dataset, and the results are shown in Figures 5 and 6 . Except for the parameters, all other parameters exhibited the same trends on both the MF and GNN backbones.For the parameter k, the performance first increased and then decreased as k increased, reaching the optimal values at 30 and 20, respectively.For the parameter N , the OOV recommendation performance first increased and then decreased, reaching the maximum at N = 9, while the overall recommendation performance consistently decreased.For the parameter p, the OOV performance first increased and then decreased, while the IV performance first decreased and then increased.For the parameter λ, on the MF backbone, the OOV performance consistently increased, while the overall performance first increased and then decreased, reaching the optimal value at λ = 0.07. On the GNN backbone, both the OOV and IV performances exhibited a trend of first increasing and then decreasing.Platform. We have implemented our USIM on the homepage of one of the largest e-commerce platforms, which boasts hundreds of millions of users and billions of items. The homepage features a feed recommendation system that recommends items to users. Thousands of new items are uploaded every hour.Framework. Our online implementation consists of two core components: 1. Online Recommendation; and 2. USIM Imagination. We present our framework in Figure 7 . When an OOV item is uploaded, we utilize the Large Language Model to embed the content features, including the product name and description. We then employ the USIM structure to predict the most suitable user sequence and optimize the embedding accordingly. Finally, we use the USIM-produced as the IV embeddings in the online recommendation model.Evaluation Metrics. We employed three tailored metrics to assess the performance of USIM against existing baselines:• Page Views (OOV item PV): The number of user clicks during the OOV period.• Page Click-Through Rate (OOV item PCTR): The ratio of clicks to impressions during the OOV period. • Gross Merchandise Value (OOV item GMV): The total value of user purchases during the OOV period. LanguageTo evaluate the online recommendation efficiency of USIM, we recorded the LLM-based content feature extraction time (LLM Content Feature Extraction Time) and the inference time for OOV item embeddings (OOV Time) of MetaEmb, ALDI and USIM. The results are shown in Table 5 .The results of the online recommendation efficiency indicate that USIM does not serve as a bottleneck for online recommendation, as the LLM is the main time consumer while USIM's speed remains comparable to that of MetaEmb and ALDI. Additionally, the OOV recommendation is an offline, one-time process that has no impact on online recommendations, as shown in Figure 7 . And given that the platform supports parallel processing, USIM computations for OOV items can be managed efficiently.One limitation of our proposed model is the large number of hyperparameters that need to be tuned. Our model involves 4 key hyperparameters. Tuning these hyperparameters can be a time-consuming Another limitation is the slow generation speed of our model due to its autoregressive generation approach. In our model, the output is generated sequentially, with each token being predicted conditioned on the previously generated tokens. This autoregressive generation process can be computationally intensive.Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?Answer: [Yes] Justification: Source code is available at https://anonymous.4open.science/r/USIM-D776.Guidelines:• The answer NA means that paper does not include experiments requiring code. • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.https://github.com/Zziwei/Heater--Cold-Start-Recommendation/tree/master/datahttps://grouplens.org/datasets/movielens/1mhttps://github.com/zfnWong/GAR 


Decision: Solid reviews, all reviewers recommend acceptance (even if some more enthusiastically than others). Reviewers find the problem important, the enhancements made here to be innovative / novel, and the experiments are convincing. Minor comments are made about experiments, but these seem easy enough to deal with.

Reviews: review: paper_summary: paper_summary: This submission proposes a reinforcement learning framework, termed as USIM, to deal with the issue of out-of-vocabulary items in recommendation system, which is also known as cold start issue in the community. The proposed framework considers a fine-grained user sequence imagining process. Specifically, USIM formulates the imagination process of user sequence as a RL problem, and then designs recommendation-oriented reward functions for user selection. Experiments on two public datasets show that USIM achieves better performance than several baselines.
strength_weakness: strength_weakness: 1. It is mentioned in the abstract that 'USIM has been deployed on a prominent e-commerce platform for months, offering recommendations for millions of OOV items and billions of users'. However, there seems no online comparison involved in the experiment. More details about online A/B test should be discussed to support the claims in the abstract.

2. There exist some statements lacking literature or empirical evidence, and some statements lacking clear description. For example:
    - In Line 47 : 'The substantial distinction between the content features and the behavioral embeddings may result in substantial discrepancies between the IV and OOV items', which is not verified with evidence.
    - Even though mentioned a lot, there is no clear definition about 'imagine user sequence', which seems not a commonly-used term.

3. The writing of the submission is too poor. The presentation of the submission is hard to follow. For example, in section 3.3.1, while the definition of MDP is described, how the MDP is related to the OOV issue is not discussed. The correspondence relation between (state, environment, action, reward) and OOV should be presented, otherwise it is quite challenging for readers to get through the proposed algorithm.

---After reading the author rebuttal---
The author rebuttal has address partial concerns. The results of A/B test have been added, which should be included in the camera ready (if accept). I have rasied my score to 5.
questions: questions: Please refer to the weakness part.
limitations: limitations: NA
paper_summary: paper_summary: The authors propose a novel User Sequence Imagination (USIM) fine-tuning framework. This framework can imagine the user sequences and then refine the generated OOV embeddings with user behavioral embeddings. Specifically, the authors frame the user sequence imagination as a reinforcement learning problem and develop a custom recommendation-focused reward function to evaluate the extent to which a user can help recommend the OOV items. Furthermore, they propose an embedding-driven transition function to model the embedding transition after imagining a user.
strength_weakness: strength_weakness: 1. The experiments conducted in this study used a limited number of datasets, and their scale was not very large. As a result, it is difficult to clearly observe the performance of USIM on large-scale datasets.

2. It is noticed that USIM does not show significant improvements over SOTA methods in certain metrics on the Movielen dataset. Additionally, the author mentions that USIM heavily relies on the selection of hyperparameters, suggesting that the performance advantage of USIM may not be significant.

3. There is a lack of theoretical analysis or experimental comparisons regarding the time efficiency of USIM. Since it is based on RL methods and processed in the form of MDP, it suggests that USIM may have lower time efficiency. Considering that it does not bring significant improvements on MovieLen, the question arises whether the high time complexity it entails is justified.

4. The author mentions that USIM has been deployed in real-world scenarios and has shown improvements. However, the experimental section of the article lacks the representation of relevant results and does not include offline evaluations on large-scale industrial datasets.
questions: questions: 1. Can the author provide the results of significance tests for the main experimental results?

2. Can the author provide a practical comparison of the experimental time between USIM and some SOTA baselines to better evaluate its efficiency?

3. The author mentions that the efficiency of USIM may not be high, while also claiming that it has been deployed in real-world scenarios. However, the deployment of recommendation systems in practical settings demands high efficiency, which contradicts the characteristics of USIM.
limitations: limitations: The authors well addressed the limitations. One limitation of USIM is the large number of hyperparameters that need to be tuned, which is time-consuming and computationally expensive. Another limitation is the slow generation speed due to its autoregressive generation paradigm.
paper_summary: paper_summary: The authors propose the USIM (user sequence imagination) framework to tackle the out-of-vocabulary recommendation problem. The authors point out that existing OOV recommendation models such as generative and dropout models will import significant gaps between the IV items and the OOV items, since the OOV items do not have user sequence. Then USIM can imagine user sequence for OOV items to bridge the gap. Generally, I agree USIM can provide better OOV item recommendation performance. 

In terms of technique details:
- The authors formulate the user sequence imagination process as a MDP, and propose a rec-PPO to optimize the imagination. 
- The authors provide the State Transition Function to describe the embedding optimization process.
- The authors claim they provide RecPPO to optimize the USIM.
- The authors already implement USIM on commercial platforms to verify the effectiveness.

In summary, this paper provide a new prospective of OOV recommendation models, I recommend an accept for this paper. However, there are still some remaining issues as listed in weakness.
strength_weakness: strength_weakness: - The RecPPO, which is a crucial component of the proposed framework, is not clearly defined in Section 3.3.2. Providing more detailed information about RecPPO would help readers better understand its role and implementation within the USIM framework.

- Figure 2, which aims to illustrate the overall USIM process, could be further refined. Enhancing this figure to more clearly and comprehensively present the USIM process would improve the paper’s clarity and help readers grasp the proposed framework's intricacies.

- Although the authors mention that USIM has been deployed on commercial platforms, there is a lack of detailed discussion on its industrial implementation and experiments. Providing insights into the deployment process, challenges faced, and specific industrial experiment results would add valuable context and strengthen the paper’s practical relevance.
questions: questions: See weakness
limitations: limitations: Yes.
paper_summary: paper_summary: This paper presents a novel Out-of-Vocabulary item recommendation or so-called cold-start recommendation model.

The OOV recommendation problem especially item recommendation is very important, since in short video recommendation platforms there are thousands of new post videos or new AGI videos being published every second. OOV Item recommendation are so important in dealing with the flourish of the new videos. User Sequence Imagination (USIM) provides a new paradigm for OOV recommendation to imagine behaviors of OOV items. I do like this paper, since this paper provides a totally novel and promising solution in solving OOV recommendations, of which the paradigm is quite different from traditional works. Traditional models such as generative models and dropout models do not solve the OOV recommendation fundamentally. USIM enable the OOV items to be recommended the same as In-Vocabulary items. I think this approach has the potential to fundamentally resolve the OOV issue.
strength_weakness: strength_weakness: 1. Figure 2 can be further improved. The presentation of Figure 1 is excellent, why not keep Figure 2 in the same style?
2. There may be some typos in Equation 4. The optimization objective should be to maximize the score between e_i and e_u.
3. The fonts in Figure 4 are quite small. Besides, it’s better to modify the Table 1 to be around the main results, rather than the experimental setup.
questions: questions: Q1. The paper mentioned that USIM has been deployed on prominent e-commerce platforms. Are there any AB-Test results or some results from the industrial side?

Q2. Could you discuss whether this model can be used for short video recommendation platforms?
limitations: limitations: Yes

Comments: title: Thank you for your reply
Thank you for your reply
Thank you for your reply
Rasied my score to 5 after reading the author rebuttal.
Thanks for the rebuttal
Thanks for your reply
comment: Thank you for your thoughtful review and feedback. We're glad our response addressed most of your concerns, and we appreciate your continued support for our paper.
We greatly value your thorough review and kind feedback. If any additional concerns arise, please feel free to reach out to us. We’re ready to assist with more information.
We sincerely appreciate your valuable review. If there are any further concerns, please don't hesitate to contact us. We're more than willing to offer additional information.
I will keep my score after reading the response.
I have rasied my score to 5 after reading the author rebuttal.
Thanks for the authors' response. It addressed most of my concerns. I am happy to keep my rating for this paper.
Thank you for your helpful review and kind support! If you have any further concerns, please feel free to reach out to us. We would be happy to provide additional details.
Thank you for the detailed response, the author effectively addressed my issue, I will increase my grade.