LLM-BRec: Personalizing Session-based Social Recommendation with LLM-BERT Fusion Framework

Authors: raksha  jalan
tushar  prakash
Venue: ACM_org_SIGIR_2024_Workshop_Gen-IR
Published: 2024-05-02 20:01:22

Abstract: Recommendation models enhance online user engagement by suggesting personalized content, boosting satisfaction and retention. Session-based Recommender systems (SR) have become a significant approach, focusing on capturing users' short-term preferences for more accurate recommendations. Recently, Session-based Social Recommendation (SSR) has emerged as a new paradigm that extends SR by incorporating users' social networks and historical sessions, aiming to offer more personalized recommendations.  However, current SSR models have two significant limitations: First, they have not efficiently explored user's personalized information, as they focus only on current session information. Second, they use computationally heavy graph-based algorithms for session representations, which significantly hampers the model's efficiency, especially during inference. \\
To address the aforementioned problems, this paper proposes a novel fusion framework, "LLM-BRec," which incorporates  Large Language Models (LLMs)  and  Bidirectional Encoder Representations from Transformer (BERT) to personalize SSR.  Here,  For session modelling,  BERT's transformer architecture and self-attention mechanism are utilized to enhance its computational efficiency by emphasizing relevant contextual information.  Additionally, we leverage LLM  for user-profile generation to further enhance representation at the inference stage.  LLM-BRec has significantly reduced SSR's training and inference time and consistently outperformed the SOTA methods. Experiments on two social datasets and two non-social datasets demonstrate the effectiveness and efficiency of LLM-BRec.

Main paper: 
 introduction: 
 With the rapid growth of e-commerce and online entertainment platforms, users expect personalized items and information. The success of these platforms relies heavily on recommendation systems designed to filter irrelevant information and offer personalized recommendations. Considering sessions in Recommendation Systems is crucial for understanding the temporal dynamics of user preferences, leading to more accurate and personalized recommendations. With this intent, Session-based Recommendation(SR) exploits the commonality of objectives within close temporal proximity to predict the next user interaction within the same session.In the early stages of SR, Anonymous Session-based Recommendations(ASR) [14] have emerged to address scenarios where user-IDs are unavailable or user-tracking is challenging, focusing on learning user preferences from sequential transition patterns in anonymous sessions. Later, Personalized Session-based Recommendations(PSR) [21] have emerged, which leverage user-IDs to enable cross-session information transfer for users with diverse preferences, thereby enhancing the effectiveness of Sessionbased Recommendation models. The rising popularity of social media has transformed online spaces and digital platforms into social-centric landscapes. Popular platforms like Spotify, Instagram, and Amazon Prime enable users not only to consume content but also to share reactions with friends. Consequently, developing recommendation systems that integrate social relationships becomes crucial to adapt to the evolving dynamics of user engagement. The growing prominence of such socialcentric digital landscapes, coupled with session-based recommendations (SR), has led to the development of Session-based Social Recommendation (SSR).The general field of social recommendation has been extensively researched. However, there is limited existing literature explicitly focusing on SSR due to its relatively recent emergence. The concept of SSR was first introduced within DGRec [25] , where Graph Neural Networks (GNN) are employed to aggregate neighbors' preferences for each user. Subsequently, SERec [2] proposes the utilization of heterogeneous GNNs to learn user and item representations, integrating knowledge from social networks.Even though the aforementioned methods have demonstrated promising performance compared to traditional session-based methods [16, 35] , we contend that existing session-based social recommendation methods have two significant drawbacks: (a) Existing SSR approaches have fallen short in leveraging the richness of users' personalized information, as they primarily rely on current session data and neglect exploring personalized interest trends, hindering the model's performance. For instance, SERec only concatenates user representations at the last stage of model inference, while DGRec considers only session-level interaction history, limiting the expressiveness of user interests. (b) Recent advancements in SSR rely mostly on computationally intensive graph-based algorithms for session modeling. Though a longer training time can be acceptable, longer inference times are less tolerable due to the need for low latency. Notably, SEFrame's employment of GNNs and DGRec's Graph Attention Network(GAT)+LSTM model at session levels, significantly hamper their efficiency, especially during inference time. However, adopting lighter models for session modeling can improve model's efficiency without compromising SSR performance.To overcome these challenges, this work proposes a novel framework, "LLM-BRec" for the Session-based Social Recommendation. LLM-BRec employs Social-aware Heterogeneous Graph(SHG) for learning enhanced user/item representations and BERT for efficient session modeling, enhancing overall performance. Additionally, It leverages the strengths of the Language Models(LLM) for user profiling.First, a diverse knowledge graph is built to encompass universal knowledge from all interactions,item's side information and the complete social network. Recognizing the importance of leveraging social connections for modeling user preferences, this Social-aware Heterogeneous Graph is employed to learn comprehensive representations of users and items. Next for the current session, the learned user and item representations from SHG are passed to a BERT for session modeling. This aims to predict the next item with which the user will interact in the current session. Here, we devised BERT-based model, as the self-attention mechanism in BERT calculates attentions within input sequences in parallel, making it more computationally efficient and faster for learning better session representations, especially during inference.Finally, we have found that leveraging the contextual awareness and adaptability of LLMs can serve as a powerful tool for generating user-profile. However, we emphasizes the importance of efficiently using LLMs. Given the computational expense of recurrently updating user-profiles across sessions through LLMs by incorporating current session's information and preventing data leakage during training, we advocate for generating user-profiles based on all sessions post-training and utilizing them solely during inference. This approach optimizes the utilization of LLMs' capabilities without adding computational complexity. The proposed framework is named as "LLM-BERT Fusion Framework for Recommendation(LLM-BRec)". The key contributions of this work are summarized as follows:• Proposed an efficient method for session modeling with BERT, which allows faster processing and reduces the computational time. Specifically, our model has reduced training time by 50% and inference time by 80% compared to SOTA SEFrame. • First to demostrate the importance of user profiling with LLMs post-training to enhance recommendation system performance. • For SSR, the proposed LLM-BRec has consistently outperformed SOTA models. Additionally, to show the effectiveness of LLM-BRec, we have compared model's performance on two non-social SR datasets. 
related work: 
 In this section, we briefly cover relevant work on session-based recommendation systems, particularly session-based social recommendation models.Session-based Recommendation (SR) can be broadly categorized into Anonymous Session-based Recommendation (ASR) and Personalized Session-based Recommendation (PSR), depending on the availability of user information. Early studies in ASR, like FPMC [24] , established the foundation by focusing on sequence extraction through Markov chains. GRU4Rec [14] was the first to formalize ASR, introducing a multi-layered GRU model. NARM [16] is another notable method with attention mechanisms within GRU. Poppular approach, SR-GNN [35] represented sessions as directed subgraphs, leveraging GNNs for item transitions. Yet, these methods catered to anonymous sessions, overlooking user preferences and social network information. Contrary to ASR, Personalized Session-based Recommendation (PSR) involves non-anonymous interactions with associated user information. This enables the linking of different sessions generated by the same user at different times. Popular model HRNN [21] proposed a hierarchical RNN model to capture users' evolving interests. A-PGNN [36] extended SR-GNN [35] for PSR and employed the attention mechanism to capture the influence of the user's past interests on the current session. PSR enables learning of users' long-term preferences and their evolution across sessions but faces challenges in incorporating inter-user influences within the dynamic context of sessions.Recently, Social Recommendations have received a lot of attention since social relations give an additional perspective on user preferences along with interaction history. Early developments include MF-based approaches like SocialMF [15] , TrustSVD [11] , and [39] incorporated social relations with matrix factorization. The landscape of Social Recommendation evolved significantly with the emergence of deep learning technique systems [7, 9, 13, 32] . Additionally, Models such as GraphRec [8] , and SocialGCN [34] emerged, incorporating GNNs and GCNs to enhance Social recommendation systems. Popular social recommendation methods, DeepSoR [32] , and DiffNet++ [33] have demonstrated the effectiveness of graphs and attention-based networks for capturing the user-item interaction and social relations. Later, to capture users' evolving interests, various sequential recommendation algorithms [10, 18, 25] have emerged, which extract features from social relations and user behavior sequences. The approaches outlined above are coarsegrained and struggle to capture users' dynamic preferences since they represent user-level interactions while ignoring the relevance of user profiling and temporal segmentation of interaction.Session-based Social Recommendation (SSR ) aims to combine the advantages of session-based recommendation and social recommendation and provide more accurate and personalized recommendations. It predicts users' next click in the current short-term session-based on social networks and historical sessions. The first SSR model is DGRec [25] , which uses a graph attention network to model the social influence of the user However, this method is inefficient because it needs to process many additional sessions to predict the next item of the current sessions. Then, SERec [2] proposes to use a heterogeneous graph to process related users and items when making predictions for the current session. However, compared with our model in this paper, the common limitations of existing studies are: Despite their commendable efforts to integrate social networks into session-based recommendations, they still fail to take full advantage of the personalized information of the user and item when modeling the current session. Additionally, they lack potential efficiency in session modeling as they used computationally intensive graph-based methods for session modeling during inference, but lighter models can perform more efficiently without compromising SSR performance.Recently, there has been rising enthusiasm for using self-attentionbased models like Transformer [28] , GPT [22] , LLAMA [27] and BERT [6] in the representation learning since they have produced impressive results in text sequence modeling. Inspired by these advancements, Popular recommendation models like Bert4Rec [26] and Transformers4rec [4] have adopted transformers for building recommendation systems. In pursuit of enhancing the recommendation model's performance, recent efforts like [31] involve integrating pre-trained LLMs at various modeling stages. However, due to their computational cost and inefficiency for domain-specific tasks, LLMs may not always be the optimal choice. Motivated by these constraints, our objective is to illustrate the most effective approach for harnessing LLMs to enhance SSR.  
llm-brec framework: 
 The proposed model has two main components: the Social-aware Heterogeneous Graph (SHG), which integrates knowledge from users' historical interactions H , item category, and social network N to learn the enhanced user and item representations, and the BERT-based session-modeling module, which is designed to efficiently capture the temporal dynamics of user preferences in a session-aware manner, ultimately resulting in more personalised recommendations.In the following sections, various components of the model are discussed in detail.This component learns the global context using users' historical interaction H consists of all the sessions, item's side information and user social network N . Our SHG improves upon Knowledge graph presented by [2] . Specifically, In contrast to [2] , we have utilized item's side-information such as items's categories or genre to enrich the item representation from SHG V 𝐻𝐺 during training. The impact of this enhancement is studied in section 5.5.4.1.1 Heterogeneous Graph. Formally, let K = {N, E} represent the heterogeneous graph. The node set N = 𝑈 ∪ 𝑉 encompasses all users and items involved in H and N . Note that here items are already infused with side information as shown in figure 1 . The edge set E comprises three types of directed edges: user-user edges (UU), user-item edges (UI), and item-item edges (II). Specifically, a user-user edge (𝑢 𝑖 , 𝑢 𝑗 ) ∈ E exists if user 𝑢 𝑖 is followed by user 𝑢 𝑗 . A user-item edge (𝑢 𝑖 , 𝑣 𝑗 ) ∈ E signifies that user 𝑢 𝑖 interacted with item 𝑣 𝑗 in any session. Additionally, a item-item edge (𝑣 𝑖 , 𝑣 𝑗 ) ∈ E indicates a transition from item 𝑣 𝑖 to 𝑣 𝑗 observed in any session.Learning User/ Item Representations using Heterogeneous Graph. In order to learn enhanced user and item representations, we have applied a Heterogeneous Graph Neural Network (HGNN) [38] to graph K. Let's consider that the representation of a node 𝑣 at layer 𝑙 is denoted as R 𝑙 [𝑣], where 𝑣 represents either a user or an item. The initial embedding for user/item 𝑣 is given by R 0 [𝑣] ∈ R 𝑑 , with 𝑑 representing the embedding size. In order to get the node representation of a node 𝑢 at 𝑙 𝑡ℎ layer from node 𝑣 at layer 𝑙 -1, a message passing technique is used as both the nodes are neighbours:EQUATIONwhere, 𝑊 𝑇 and 𝑏 𝑙 are the trainable weights. To generate an updated representation for node 𝑢 as we move from layer 𝑙 -1 to layer 𝑙, we find the significance of each connection to node 𝑣 within the graph K as the number of neighbouring users and items might be different, making the neighbouring representation less reliable.EQUATIONwhere, 𝜎 refers to sigmoid activation function and 𝑊 𝑇 are the trainable weights of linear projection. 𝜙 ∈ R 𝑑 are also trainable weights, R 𝑙 -1 [𝑣] and R 𝑙 -1 [𝑢] are the 𝑙 -1 layer representation of node 𝑣 and 𝑢 respectively. Finally, 𝑒 𝑙 ∈ R 𝑑 is the edge feature vector of nodes 𝑣 and 𝑢. Furthermore, these attention scores are normalized for all the neighbouring nodes. Next, the contributions from all adjacent nodes are calculated as the weighted sum of all messages.EQUATIONIn the final step, the updated node representation is computed using a straightforward node-specific linear transformation, followed by the activation function ReLU, which is applied to the aggregated * * EQUATIONWhere 𝑊 𝑇 and 𝑏 𝑙 are the trainable parameters. In this way enhanced social-aware user and item representations are learned.Existing recommendation systems primarily emphasize behavior data, whereas LLMs can enhance incomplete and low-quality side information by tapping into their extensive world knowledge. When a user engages with a diverse array of items and demonstrates varied affinities across sessions, accurately recommending content solely based on user's latest session's behaviors becomes challenging.However, using LLMs to update user-profile at each session while training is computationaly expensive as LLMs requires extensive computational resources. To address this issue, First we have tried to mimic the purpose of LLMs by getting top three categories of interacted item for each user during training. As session-based recommendation restrict access to all interactions during the training period, we dynamically identified the top three interaction categories as the session progressed. But, this strategy let to four-fold increase in training time without any significant improvement in performance.Therefore, We propose to leverage LLMs post-training, only during inference stage. This approach for user-profiling, optimizes the utilization of LLMs' capabilities without adding computational complexity. Using LLMs for this purpose involves analyzing user interactions, such as check-in and TV viewing histories, to generate concise summaries of their preferences. These summaries, ranging from a user description of one or more sentences to a set of keywords defining user interests, are referred to as the user profile in this paper. The LLM model utilized to acquire the user profile summary is Facebook's "Llama-2-7b-chat" [27] .Prompts for User Profiles. To generate the user profile from LLM, two different prompt styles have been proposed. 1) User Description ((UP) 1 ): In order to get the user's interest summary in N sentences where N ∈ {1,2,3}, item attributes of a user 𝑢 are being passed to LLM. Let for a user 𝑢, 𝑇 𝑢 = [𝑡 1 , 𝑡 2 ..., 𝑡 |𝑇 𝑢 | ] be the comma-separated list of item attribute of all the items that user 𝑢 have interacted in H 𝑢 . These attributes might be session titles, movie titles, book titles, etc. The list is sent to the LLM with prompt 1, which requests it to summarize the user's interests in N sentences using previously interacted item attributes. The LLM outputs an N-sentence summary, identifying the user's long-term interest. Like,for Foursquare data, LLM identifies users' frequent "Based on the user's history of visiting a variety of locations, including grocery stores, restaurants, theme parks, and entertainment venues, it is likely that this user is a social and adventurous individual who enjoys trying new experiences and exploring different parts of their city or region."Topics: Gift Shop, Theme Park, ResortStep 1: Prepare item attributes list from side information of items in historical interaction dataStep 2: Using item attribute list to prepare prompts for LLMStep 3: Getting natural language output from LLMs for both the prompts Step 4: Creating user profile embeddings from LLM's outputs 2) Topic of Interest ((UP) 2 ): Similarly, To get the user 𝑁 of interest, N ∈ {1,2,3}, item attribute list 𝑇 𝑢 of user 𝑢 is passed to LLM with prompt 2. This prompt asks LLM to generate the top 𝑁 general topic of interest of user 𝑢 by analyzing the item attributes that the users have interacted with in the past. In our case, the output from LLM for this prompt is a comma-separated list containing the user's top 𝑁 (N=3) topic of interest, as shown in Figure 2 .User profile Embedding Generation. The user generated by LLMs are in language and require conversion into embeddings to improve user representations. For embedding generation, we have explored the following methods:1) Word2Vec (𝑊 𝑉 ): Word2Vec [20] is a prevalent word embedding technique. The user-profile embeddings were created by averaging the pre-trained embeddings of words in LLM generated user-profile 4.2.1. This embedding, denoted as E 𝑊 𝑉 𝐿𝐿𝑀 , shares the same dimensions as the user embeddings (E 𝑢 ∈ R 𝑑 , where 𝑑 = 128), eliminating the need for linear transformation.2) Transformer (𝑆𝑇 The Sentence Transformer, particularly the MiniLM-L6-v2 variant [30] , can understand subtleties and generate paraphrased content. Here, we have used the encoder part to get the embeddings from Sentence Transformer (MiniLM-L6-v2 variant), which is a pre-trained model.These embedding from 𝑆𝑇 , denoted as E 𝑆𝑇 𝐿𝐿𝑀 has a dimension of 𝑑 = 354, which is adjusted using a linear projection to make it equal to the user embeddingEQUATIONHere, the Linear operation the embedding from E 𝑆𝑇 𝐿𝐿𝑀 to (E 𝑆𝑇 𝐿𝐿𝑀 ) 𝑢 ∈ R 𝑑 , where the dimension 𝑑 =3) OpenAI Embedding Model (𝑂𝐸): OpenAI's ada-002" [17] is a language model that excels in generating contextually rich and coherent representations, levertransformer architecture and extensive pre-training.Compared to Sentence transformer 𝑆𝑇 the 𝑂𝐸 model is a large corpus, providing more representations. These embedding from 𝑂𝐸,denoted as E 𝑂𝐸 𝐿𝐿𝑀 has a default dimension of 𝑑 = 1534, is adjusted a linear projection to make it to the user embeddingEQUATIONHere, the Linear operation transforms the embedding from E 𝑂𝐸 𝐿𝐿𝑀 to (E 𝑂𝐸 𝐿𝐿𝑀 ) 𝑢 ∈ R 𝑑 , where the output dimension is 𝑑 = 128.These steps for user profile generation using LLMs are also presented in figure 2 . More information on the pre-trained models is presented in section 5.1.3. For ease, we have referred to user profile embeddings as E 𝑢 𝐿𝐿𝑀 . In the next section, we will discuss how E 𝑢 𝐿𝐿𝑀 embedding is used to enhance the user embedding.The KG embeddings encapsulate universal knowledge from all sessions and the complete social network, yet they do not encompass session-specific contextual information. Conversely, understanding user behaviors in the current session 𝑆 is crucial for capturing dynamic user interests. Hence, BERT [6] is employed to capture the user's present preferences and the contextual properties of items of the current session. BERT's self-attention mechanism calculates attention within input sequences in parallel, making it more computationally efficient and faster for learning better session representations.The representations for users and items are fetched from the Social-Aware Heterogeneous Graph(SHG), denoted by U 𝐻𝐺 and V 𝐻𝐺 respectively. These representations serve as the user and item embeddings, which are eventually passed to BERT for sessionmodelling.Specifically, for 𝑖 𝑡ℎ session of user 𝑢 denoted as 𝑆 𝑢 𝑖 , the input sequence would be 𝑆 𝑢 𝑖 = {𝑣 1 , 𝑣 2 , ..., 𝑣 𝑚 }. In order to train the BERT for the next-item prediction task, the last item of the input sequence is masked. So, new item sequence will be 𝑚𝑎𝑠𝑘𝑒𝑑 (𝑆 𝑢 𝑖 ) = {𝑣 1 , 𝑣 2 ..., [𝑚𝑎𝑠𝑘]}. Now, the embedding E 𝑣 of the item sequence and user embedding E 𝑢 are looked-up from V 𝐻𝐺 and U 𝐻𝐺 respectively:EQUATIONEQUATIONAt training-stage, The final user embeddings is computed as mentioned in equation 8.While at inference stage, In addition to the E 𝑢 𝑓 embedding, a user profile embedding E 𝑢 𝐿𝐿𝑀 generated through LLMs are also used to enhance the user representation, the process of generation of E 𝑢 𝐿𝐿𝑀 embedding is present in the section 4.2. Here, 𝛼 is the weight given to the user profile embedding E 𝑢 𝐿𝐿𝑀 . Therefore final user embeddings at inference would be:EQUATIONFinally, in order to capture the user's evolving preferences, the user embedding E 𝑢 𝑓 is concatenated with item embedding E 𝑣 to get a personalized representation of current session E.EQUATIONThe embedding E is then passed to the BERT to predict the masked item. More details regarding the training are present in the section 4.4.The main objective of our proposed model is to predict the next item in a user's current session 𝑢. Formally, Let 𝜃 be the trainable function of BERT and 𝜓 be the classification layer to predict the next item. For each iteration, embedding E ∈ R 𝑑 , 𝑑 = 128 is generated as mentioned in section 4.3. This embedding is being passed to BERT to get the rich representation of the current session, followed by the classification layer 𝜓 to predict the masked item. The loss function L of our proposed model is defined below:EQUATION)Where 𝐶𝐸_𝑙𝑜𝑠𝑠 is the cross entropy loss and 𝑣 𝑜 is the masked item that the model has to predict. The objective to minimise the loss function L is mentioned below: arg minEQUATIONThe whole training process is illustrated in Figure 1 and mentioned in Algorithm 1.At the inference-stage, we enhance user-embeddings E 𝑢 by adding LLM generated user-profile encodings to it. 𝑚𝑎𝑠𝑘𝑒𝑑 (S 𝑢 𝑖 ) = last_item_masking(S 𝑢 𝑖 )6:E 𝑢 = U 𝐻𝐺 [𝑢]⊲ user emb. look-up from SHG 7:E 𝑣 = V 𝐻𝐺 [𝑚𝑎𝑠𝑘 (S 𝑢 𝑖 )]⊲ Item emb. look-up from SHG 8:E 𝑢 𝑓 = E 𝑢 + 𝛼 * (E 𝐿𝐿𝑀 ) 𝑢 ⊲ (E 𝐿𝐿𝑀 ) 𝑢 used only at inference 9: E = E 𝑢 𝑓 ⊕ E 𝑣 10: v = 𝜓 (𝜃 (E))11: L = 𝐶𝐸_𝑙𝑜𝑠𝑠 (𝑣 𝑜 , v) ⊲ compute 
experiments and evaluations: 
 This section aims to get answers to the following research questions:• RQ1: How does LLM-BRec's performance compare to other stateof-the-art (SOTA) models? • RQ2: How do variations in user-profiling method with LLM enhance the model's performance? • RQ3: How does LLM-BRec training/inference efficiency to SOTA • RQ4: Evaluation of the quantitative impact of essential components on the overall performance of the model.5.1.1 Datasets. We have our proposed framework using publicly available datasets commonly employed in for Session-based Social Recommendations (SSR): (1) Delicious1 : It is a collection of bookmarks from an online bookmarking system that allows users to classify bookmarks into several semantic categories. Following [2] , we see a tag sequence with timestamps on a bookmark as a session, aiming to provide personalized tag suggestions. Here, we have the tag's URLs as side information, along with the session title. As mentioned in section 4.2, these session titles are used to generate user profiles with LLMs.(2) Foursquare2 : It is a large-scale check-in dataset. The social network is built using an external social-media site. Following [2] , the check-in records have been divided into sessions based on one-day intervals. Here, we have a category of checked-in places as side information.To evaluate the adaptability of our model in non-social recommendation scenarios, we tested its performance on widely used datasets such as MovieLens-1M3 and Amazon-Book4 datasets. Since they originally used only interaction history, we modified our model to adapt to this use case. For user profiling in ML-1M, we used item attributes such as movie titles and genre. In Amazon Books, we utilized book titles as side information.For all the datasets, we used the same data partitioning and preprocessing as mentioned in [1, 2, 19, 35] to maintain consistency in results. Some statistics of the datasets after pre-processing are shown in Table 1 .To demonstrate the effectiveness of our proposed model, we compare its performance with various strong baselines from Session-based Recommendation(SR) and SSR.• Anonymous Session-based Recommendations (ASR): Item KNN [3] model is inspired by the classic KNN model.FPMC [24] is a Markov-chain-based approach. NextItNet [37] is a CNN-based approach with dilated convolution. NARM [16] is an RNN-based approach that incorporates attention into GRU. STAMP [19] is a model that uses the attention mechanism to better understand users' short-term interests. SR-GNN [35] is another technique with gated GNN to learn about complex item transitions within sessions. • Personalized Session-based Recommendations (PSR): SSRM [12] is the advanced SOTA approach for streaming SR. • Session-based Social Recommendations (SSR): (1) DGRec [25] is the first proposed method for SSR that captures users' dynamic interests using RNNs and a graph attention network. ( 2) SERec [2] is most recent SOTA for SSR. It utilizes a heterogeneous graphs to learn embeddings and weighted GNN for current session modeling. • LLM-SSRM,LLM-DGRec, LLM-SERec: We investigated the effectiveness of user profiling with LLMs in existing SSR. During inference, we enhanced their user embeddings with best performing LLM-based user-profile embeddings. More details in section 5.3. • LLM-RNNRec, and LLM-LSTMRec: To investigate the impact of BERT, These are variants of our model built by replacing the BERT module with RNN and bidirectional LSTM, resp. • BPR [23] , NCF [13] , GRU4Rec [14] , BERT4Rec [26] , HRNN [21] : These are popular baselines from non-social Session-based recommendation models. First two models are based on Bayesian probability and collaborative filtering respectively. GRU4Rec and HRNN use RNNs while BERT4Rec uses transformer-based architecture, BERT for current session modeling.Metrics. Each dataset is split into 60% training, 20% validation, and 20% testing. At the inference stage, we need to perform only the prediction task for the next session of LLM-BRec to generate predictions on the test dataset. We have adopted standard evaluation metrics Hit Ratio (HR) [5] and Mean Reciprocal Rank (MRR) [29] to evaluate the recommendation performance. To ensure consistency with SERec [2] , all results are multiplied by 100. Each experiment is performed five times to eliminate inconsistency in this process.All experiments are performed on GPU:NVIDIA RTX 3090, 24GB.For hyperparameter tuning, we used a grid search method. The number of transformer heads, transformer layers, and embedding size affected the performance. After testing, the optimal configuration was determined to be 2 transformer heads and 4 transformer layers, with an embedding size of 128 across all datasets. The user profile embedding weight, represented by 𝛼, had the most impact when set at 0.5, a value obtained from a range of 0.1 to 1.0. All experiments implemented early stopping based on the primary metric HR@10 to prevent overfitting.We have used the gensim5 library to get pre-trained Word2Vec model. Sentence Transformer Library6 is used to get pre-trained MiniLM-L6-v2 model and OpenAI API 7 been used to get embeddings from "text-embedding-ada-002" model.The results of all experiments on the Social dataset are shown in Table 2 . Our proposed model, LLM-BRec, demonstrates superior performance compared to existing baselines on both datasets.The notable improvement of LLM-BRec over ASR, PSR, and SSR models highlights the significance of incorporating user profiling in learning better user preferences. LLM-BRec also outperforms recent PSR and SSR models, which utilize computationally heavy attention-based and graph-based models for session modeling. This signifies the effectiveness of considering the bidirectional context of user-item interactions for session-level modeling. Existing SOTA models for SSR, DGRec, and SERec perform better than non-socialaware models by leveraging social networks to learn more accurate user preferences. However, they do not outperform LLM-BRec, indicating the potency of user profiling when combined with the proposed SHG and BERT.LLM-BRec's session modeling efficiently captures contextualized sequential information, adeptly adapting to dynamic user interactions. This is in contrast to DGRec and SERec, which rely on the current session's information and complex predefined graph-based algorithms for session modeling. The performance improvement of LLM-SSRM, LLM-DGRec, and LLM-SERec, i.e., adapted LLM-integrated methods over their original SOTA methods, proves the effectiveness of the proposed user profiling technique.To validate the significance of BERT for our task, we replaced it with RNN (LLM-RNNRec) and LSTM(LLM-LSTMRec) in the proposed model. This comparison in Table 2 reveals that BERT is more capable of creating rich representations of users' session-level behavior sequences through its bi-directional context learning compared to LSTM and RNN. In order to demonstrate the adaptability of our model for non-social SR scenarios, the results on two well-known datasets are reported in Table 3 . For non-social SR systems, LLM-Brec has significantly outperformed BERT4Rec, which also uses BERT, emphasizing the importance of efficient embeddings and architecture. BPR and NCF have shown the worst performance. Our model's performance has beaten GRU4Rec and HRNN as these models utilize only current sessions and ignore the user's long-term interest trends, which we capture with user profiling.Hence, the overall performance of LLM-BRec highlights the improvement gained by SHG for embeddings in conjunction with efficient session modeling using BERT and user profile with LLMs post-training.Table 2 : Performance Evaluation of the proposed model as compared to other methods on Delicious and Foursquare datasets. Improvement has been shown against the best-performing SOTA method, i.e., SERec( underlined). Improvements are statistically significant with p < 0.05.Delicious Foursquare HR@10 MRR@10 HR@20 MRR@20 HR@10 MRR@10 HR@20 MRR@20 ItemKNN [3] 20 Amazon Books HR@10 MRR@10 HR@20 MRR@20 HR@10 MRR@10 HR@20 MRR@20 BPR [23] 5 To examine the effects of various prompts and embedding generation techniques on user profiles created via LLMs during inference, we present two prompt variants and three embedding generation methods, as shown in Figure 2 .Multiple experiments were conducted with varying values of "N" for both prompt styles. Due to space limitations, only the performances with the optimal values are reported: Prompt 1 excels with N=3 (3 keywords), while Prompt 2 achieves optimal performance when the number of sentences is 1. Their performance is mentioned in Table 4 . For this study, we've limited our data to the Delicious and Foursquare datasets, as this paper is focused on SSR. The results in Table 4 indicate that our model exhibits inferior performance when generating user profiles from Prompt 1 compared to Prompt 2. This is mainly because Prompt 2 generates a more context-rich output that naturally gives more robust user profile embedding.Here, The choice of method for user profile embedding generation significantly impacts recommendation performance. The ta-ble4 demonstrates that the user profile embedding created from Word2Vec ((E 𝑊 𝑉 𝐿𝐿𝑀 ) 𝑢 ) performs worse than the embedding generated from Sentence Transformer ((E 𝑆𝑇 𝐿𝐿𝑀 ) 𝑢 ) and OpenAI's 'textembedding-ada-002' ((E 𝑂𝐸 𝐿𝐿𝑀 ) 𝑢 ). This is primarily because both the Sentence Transformer and OpenAI's model can learn enhanced representations as they are using transformer architecture. Additionally, OpenAI's method outperforms the Sentence Transformer because it's trained on a significantly larger data corpus. This naturally leads to a richer contextual representation, thus improving performance.To demonstrate the effectiveness of our proposed model for optimizing training/inference time, we compared the models' processing times for training and inference per 1000 batches using the Foursquare dataset as presented in Table 5 . The proposed framework is exceptionally efficient as the LLM-generated user-profiles and social-aware user/ item representations are pre-computed before inference. Therefore, during inference, the model just needs to process the current session with BERT architecture and predict the next interaction within the session. Table 5 clearly shows that the inference time for our LLM-BRec model is more than 80% less than Additionally, we have performed a computational complexity analysis of our model in terms of floating-point operations (FLOPs). From Table 6 , it is evident that SERec executes approximately ten times more FLOPs compared to our proposed model, LLM-BRec. As BERT processes sequences with a fixed length, resulting in a computationally efficient operation, while SERec, being graphbased, involves variable-sized graphs with higher computational overhead. This proves the superior efficiency of our model. Table 7 illustrates the influence of item side information on learning item representations from the SHG graph V 𝐻𝐺 . Incorporating item side information enhances V 𝐻𝐺 by capturing correlations between items. In Table 7 , "SHG without side info" refers to the model where item side information isn't utilized for learning V 𝐻𝐺 . For instance, In the MovieLens dataset comprising 3083 movie IDs and 298 genres, leveraging genres enhances item representation learning by discerning correlations among diverse movies based on their genre categorizations. Note: We've also tried adding genres as nodes in the SHG setup. These genre nodes connect with similar movies, improving how items are represented. But, doing this makes computations take longer without noticeably improving performance compared to the method mentioned earlier. To understand the impact of various components on the LLM-BRec model's performance, we present four model variants in Table 8 .The variant Only-SHG model represents the current session by averaging the embeddings from the SHG for all items within the session. However, SHG lacks session-based information, resulting in the worst performance as it fails to capture the user's evolving preferences. The Only-BERT model utilizes the BERT module for session representation with randomly initialized user/item embeddings. Despite not incorporating SHG and user profile embeddings, this variant outperforms the previous one, highlighting the importance of efficient session modeling. The Only-SHG+BERT model demonstrates that combining global knowledge from SHG with efficient session modeling using BERT can lead to better performance.To evaluate the influence of user-profiling embeddings on LLM-BRec, user profiles are fed to the model, shown as LLM-BRec, the model demonstrates a significant performance improvement. This highlights the capability of (LLMs) to impact the model's overall performance positively.  
conclusion: 
 This paper introduces LLM-BRec, a fusion framework addressing limitations in Session-based Social Recommendation (SSR) systems. By utilizing LLMs for personalized user profiles and BERT for session modeling, LLM-BRec enhances recommendation performance. Experimental results on diverse datasets demonstrate its superiority over state-of-the-art methods, showcasing reduced computational costs while delivering more accurate and personalized recommendations. LLM-BRec efficiently explores user profiling by leveraging contextual awareness from LLMs, minimizing computation by performing it only once per user post-training. BERT's transformer architecture and self-attention mechanism contribute to computational efficiency in session modeling, contrasting with graph-based algorithms. Extensive ablation studies validate the significance of each component, highlighting LLMs and BERT's roles in enhancing user interests' expressiveness and computational efficiency. Thus, LLM-BRec emerges as a promising framework for personalizing SSR systems efficiently. 
 
 Appendix: https://grouplens.org/datasets/hetrec-2011/https://sites.google.com/site/yangdingqi/home/foursquare-datasethttps://grouplens.org/datasets/movielens/1m/https://jmcauley.ucsd.edu/data/amazon/https://radimrehurek.com/gensim/models/word2vec.htmlhttps://huggingface.co/sentence-transformers/paraphrase-MiniLM-L6-v2https://platform.openai.com/docs/guides/embeddings 


Decision: The paper proposes an approach to integrate LLM and BERT to enhance social session recommendations. The reviewers acknowledge the novelty of the approach, the well-designed experiments and ablations, and the good performance gains. Some possible improvements for the camera-ready version include fixing typos in the paper and providing a computational complexity and scalability analysis to ensure the proposed approach can be used in more practical settings.

Reviews: review: title: title: Review
main_review: main_review: The paper presents an innovative approach to personalized recommendation using large language models (LLMs) and BERT. The proposed method, called LLM-BRec, combines the strengths of LLMs and BERT to deliver high-quality personalized recommendations. The main contributions of this work are:

1. The use of LLMs for personalized recommendation, which improves the performance by capturing the user's interests more effectively.
2. The use of BERT for capturing the user's preferences, which is more efficient than traditional methods.
3. The proposed method has been evaluated on several datasets, and the results show that it outperforms the state-of-the-art methods in terms of accuracy and efficiency.

Strengths:

1. The use of LLMs and BERT for personalized recommendation is a novel and innovative approach. The authors have demonstrated the effectiveness of this approach through extensive experiments on various datasets.

2. The proposed method addresses a critical challenge in personalized recommendation, which is to capture the user's interests accurately and efficiently. The use of LLMs and BERT enables the method to achieve this goal, resulting in improved recommendation performance.

3. The paper provides a comprehensive evaluation of the proposed method, including comparisons with state-of-the-art methods and detailed analysis of the results. The authors have also provided insights into the effectiveness of the proposed method through various experiments and analyses.

Weaknesses:

1. The paper does not provide a detailed analysis of the computational complexity and scalability of the proposed model, especially concerning the use of LLMs and BERT, which can be computationally demanding for large-scale datasets.

2. The evaluation is limited to a few datasets, and the performance of the model on diverse domains or datasets with different characteristics (e.g., sparse data, cold-start scenarios) is not extensively explored.

3. While the paper provides insights into the effectiveness of the proposed method, it lacks a detailed discussion on the interpretability of the model and the impact of the different components on the overall performance.

Overall, the paper presents a novel and effective approach to personalized recommendation using LLMs and BERT. The results demonstrate the potential of the proposed method and its ability to outperform state-of-the-art methods. However, there is room for improvement in terms of clarity, benchmarking, and interpretability.
title: title: Acceptance!
main_review: main_review: Summary:
The paper proposes a novel architecture, LLM-BRec, which integrates LLM and Bidirectional Encoder Representations from BERT to enhance SSR. The framework aims to address limitations in existing SSR models by incorporating user profile generation via LLMs and efficient session modeling through BERT. Experimental results on multiple datasets demonstrate the effectiveness of LLM-BRec, showing significant improvements in recommendation accuracy.

Strengths:
- Good performance gain.
- Good competing model coverage
- The authors released the code
- Statistical significance tests

Weakness:
- Efficiency: Although the paper analyzes the proposed architecture's efficiency, it overlooks the time required to retrieve an answer from ChatGPT. The authors assume they have a complete dump of the user representation from ChatGPT available during the inference stage
- Missing Bonferroni correction
- Missing the name of the statistical test


Minor:
There are some typos, even in the abstract. ("... to personalize SSR. Here, For session modeling, ...")
title: title: Simple use of LLMs to generate user summary in standard recommendation model
main_review: main_review: ## **Summary**

In this paper, the authors propose a new approach for social session recommendation which consists of 3 components: 
- User and item embeddings using a HGCN network
- user interest and summary generation using LLM
- transformer model to utilize these embeddings to make the final prediction.

The authors show through extensive experiment results that their method  outperforms many previous approaches.

## **Strengths**
1. The paper presents an interesting way to utilize the world knowledge of LLMs to augment features used in different recommendation models and show it work for small scale datasets.
2. The experiment section is well designed and several ablations are included to show the impact of different design choices.
3. The results show impressive gain over different baselines and each component contributes to the performance.

## **Weaknesses**

1. The paper is not that relevant to the workshop as the focus on not on any of the topics mentioned in te CFP
2. The writing quality of the paper can be improved. There are several typos, inconsistent case, etc that makes it harder to read the paper.
3. The paper could have benefited from some quantitative analysis of why the proposed approach works and if it would scale to larger datasets in more practical settings.