Deeply Fusing Semantics and Interactions for Item Representation Learning via Topology-driven Pre-training

Authors: shiqin  liu
chaozhuo  li
xi  zhang
minjun  zhao
yuanbo  xu
jiajun  bu
Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-09 04:45:32

Abstract: Learning item representation is crucial for a myriad of on-line e-commerce applications. 
The nucleus of retail item representation learning is how to properly fuse the semantics within a single item, and the interactions across different items generated by user behaviors (e.g., co-click or co-view). 
Product semantics depict the intrinsic characteristics of the item, while the interactions describe the relationships between items from the perspective of human perception. 
Existing approaches either solely rely on a single type of information or loosely couple them together, leading to hindered representations. 
In this work, we propose a novel model named TESPA to reinforce semantic modeling and interaction modeling mutually. 
Specifically, collaborative filtering signals in the interaction graph are encoded into the language models through fine-grained topological pre-training, and the interaction graph is further enriched based on semantic similarities. 
After that, a novel multi-channel co-training paradigm is proposed to deeply fuse the semantics and interactions under a unified framework. 
In a nutshell, TESPA is capable of enjoying the merits of both sides to facilitate item representation learning. 
Experimental results of on-line and off-line evaluations demonstrate the superiority of our proposal.

Main paper: 
 introduction: 
 With the accumulation of an overwhelming amount of information on the Internet, it becomes obligatory for web applications to proactively serve users items based on personalized preferences. To this end, it is crucial to learn high-quality representations for retail items, which are capable of capturing the items' intrinsic semantics precisely. The desirable item representations facilitate a myriad of real-life applications such as retail recommendation [19, 28] .Conventional approaches usually focus on attribute modeling to capture the inner semantics of the items (e.g., title, category, and price) [25, 33, 40, 43, 44] as shown in Figure 1 . A common approach is to design delicate and sophisticated representation learning models to encode such attributes into low-dimensional embeddings. In addition to the item attributes, there exists another type of information: user behaviors (e.g., click and view), providing valuable cross-item interactions from human knowledge and preferences as shown in Figure 1 . These two types of information mutually reinforce each other to learn quality item representations. For example, as the famous example on Walmart, users tend to co-purchase semantically different items such as beer and diapers, which reveals that user behaviors provide valuable collaborative filtering information. Meanwhile, if a user likes to eat cherries and interacts with a lot of cherry-related foods, then he/she also prefers the beer with the flavor of cherry based on semantic similarity. In a nutshell, the core of item representation learning is how to effectively model and fuse the intra-item attributes and inter-item interactions (e.g., co-click or co-view) defined by user behaviors. Traditional works only use the inter-item interactions to define the objective function, which is insufficient to fully uncover the valuable user behaviors [33, 43] . Recently, graph neural networks (GNNs) [17] are introduced to incorporate the rich interactions into the modeling process [7, 48] . Each item is viewed as a node with rich attributes, and the inter-item interactions are viewed as edges. The motivation lies in aggregating the neighborhood as the complementary to enrich the information within a single item. By fusing the item attributes and user behaviors through the message-passing mechanism, GNN-based methods generally achieve superior performance [7] . The fundamental components of the GNN-based methods are the NLU module (e.g., BERT) to model the item semantics and the GNN module to capture the topological interactions. As introduced by the recent works [41] , each item can be represented based on its textual feature in the first place and then aggregated with its co-clicked items' embeddings for the final representation. However, it is worth noting that the modeling of textual and topological features are loosely coupled within the above workflow: the item cannot refer to its neighborhood while encoding its own textual feature. Therefore, the fusion of multi-view information might be insufficient.In addition to the simple combination of GNN and NLU modules, we are going further to investigate how to effectively reinforce each other. On the one hand, from the perspective of NLU models, the pre-trained models are usually learned on the general corpus (e.g., Wikipedia) to capture the common semantic knowledge. However, the semantic similarities between items might be different from common sense. For example, a general pre-trained BERT model would treat the beer and diaper with distinct semantic meanings and learn distant embeddings for them. However, based on the signals from user behaviors, these two items are often co-purchased, and thus their embeddings should be close and similar. Thus, if we could teach language models to understand such interaction signals through continuous pre-training, it has great potential to boost model performance. On the other hand, the semantics learned by NLU models can also provide complementary information to facilitate GNNs. The graph topology usually suffers from the severe challenge of sparsity as users tend to buy hot and popular items, while the rest long-tail items cannot benefit from the behavior graph. The semantic similarities between the items are capable of building another type of correlation to enhance the cross-item message passing, which contributes to tackling the sparsity.In this paper, we investigate the problem of item representation learning by mutually reinforcing semantic modeling and interaction modeling. A novel training paradigm TESPA is proposed to reinforce the NLU and GNN modules reciprocally. TESPA is composed of three major components: topology-driven pre-training of language models, semantic-based graph enrichment, and multichannel co-training module. In the topology-driven pre-training phase, TESPA continues to pre-train the NLU model based on the multi-grained topological signals. Then, the original graph topology is enriched based on semantic similarity. Finally, a novel tightly coupled co-training paradigm is proposed to deeply fuse the semantics and interactions in a multi-channel manner. Contrastive learning techniques are further introduced into both intra-channel and inter-channel contexts to reinforce collaborative correlations. We select the item-to-item (I2I) recommendation as the experimental scenario, and the results on four datasets demonstrate the superior of TESPA. TESPA is further evaluated on a famous on-line advertising platform and achieves significant gains (e.g., +9.25% CTR). Our contributions are summarized as follows:• We propose a novel mutually reinforced training paradigm for item representation learning, which is capable of enjoying the merits of the item semantics and interactions. • We design a novel continuous pre-training phase to teach language models to understand the interactions, and a graph enrichment module to alleviate the challenge of topological sparsity based on semantic similarity. • We conduct extensive experiments over four datasets, and our proposal consistently outperforms SOTA baselines over all the datasets. 
problem definition: 
 In this section, we formally define the studied problem. The item dataset is defined as D = {P, E}, in which P = {𝑝 1 , 𝑝 the semantic similarities. Finally, the item attributes and interaction graph are co-modeled under a multi-channel training paradigm. 
topology-driven pre-training of lms: 
 Traditional pre-trained language models are typically trained on broad text datasets like Wikipedia, aiming to encompass general semantic knowledge. However, as highlighted in the introduction, interaction graphs may encompass distinctive information, exemplified by unique connections like the co-purchase correlations between beer and diapers. Integrating such topology-specific knowledge into language models could enhance item representations. The schematic representation in Figure 3 illustrates our approach, wherein we propose three pre-training tasks to tailor general pretrained language models to interaction graphs.Topological Masked Language Model (TMLM): Previous work [5] demonstrates that the continuous pre-training to incorporate domain-specific knowledge is capable of boosting the expressiveness of language models. Here we modify the original Masked Language Model (MLM) [2] by incorporating the topological connections, dubbed Topological Masked Language Model (TMLM). Given a center item 𝑐 and one of its neighbors 𝑛, their descriptive text is formally defined as Φ denotes the set of masked tokens in T (𝑐 ) , and T (𝑐 ) -Φ denotes the set of observed (unmasked) tokens. The objective of TMLM is formally defined as: (1) in which 𝜃 denotes the learnable parameters in the pre-trained models. TMLM empowers the language model with the capability of predicting the missing tokens in the neighborhoods, which smoothly encodes the token correlations across different items.EQUATIONL tmlm (𝑻 (𝑐 ) Φ |𝑻 (𝑐 ) -Φ , 𝑻 (𝒏) ) = 1 𝑚 𝑚 ∑︁ 𝑖=1 log 𝑝 (𝑡 𝜙 𝑖 |𝑻 (𝑐 ) -Φ , 𝑻 (𝒏) ; 𝜃 ),The previous TMLM task focuses on the token-level correlations. Here we propose a Masked Neighbor Prediction (MNP) task to encode the topological signals at the item level. MNP aims to predict whether the two items will be co-clicked. MNP is similar to the Next Sentence Prediction (NSP) task [2] , but in MNP, part of the item's textual descriptions are randomly masked, which increases the prediction difficulty due to the corrupted semantics. Therefore, it forces the model to seek complementary information from the co-clicked items, which will further strengthen the model's capability of leveraging the neighborhood. Given a center item 𝑐 and one of its co-clicked neighbors 𝑛, we first randomly mask a portion of tokens from the original text T (c) and T (n) . Then, their representations are defined as the [CLS] embeddings learned by the language models as ĥ𝑐 and ĥ𝑛 . We adopt the link prediction as the training task, and the objective function is formally defined as follows:L 𝑚𝑛𝑝 = -log 𝑒 ⟨ ĥ𝑛, ĥ𝑐 ⟩ 𝑒 ⟨ ĥ𝑛, ĥ𝑐 ⟩ + 𝑟 ∈𝑅 𝑒 ⟨ ĥ𝑛, ĥ𝑟 ⟩ ,in which ⟨•⟩ denotes the inner-item operator. Set 𝑅 contains the negative samples. Conventional randomly sampled negative samples might be easily filtered and further hinder the discriminative capability. Here we design a hard negative sampling method based on the topology distances. Given a center item, we view its neighbors within a fixed distance window (e.g., 3) but are not directly connected as the hard negative samples.Neighbor Reordering Task (NRT): MNP task captures the firstorder relations between items. However, one of the striking characteristics of the graph is its nature of high-order connections. The high-order relations provide the relative positions of the neighboring nodes. Hence, we designed another task, dubbed neighbor Reordering task, to capture the high-order closeness. Given a center item 𝑐, we first randomly select a set of neighbors from its topological context based on the shortest path distance (SPD). Specifically, a single neighbor 𝑛 (𝑠 ) is randomly selected from the 𝑠-hop neighbors with SPD(𝑐, 𝑛 (𝑠 ) ) = 𝑠. After collecting neighbors from all the 𝑢 hops, we can achieve a set of neighbors from different hops: 𝑺 = {𝑛 (1) , 𝑛 (2) , . . . , 𝑛 (𝑢 ) }, where 𝑢 is a hyper-parameter deciding the range of receptive field. Based on the center node 𝑐 and the neighbor set 𝑺, we formalize the NRT task as a classification task to classify each pair < 𝑐, 𝑛 (𝑠 ) > to the correct category 𝑠 denoting their relative distance. The representations of 𝑐 and 𝑛 (𝑠 ) are concatenated together and fed into a linear classifier. The training objective is defined as the cross-entropy loss:EQUATIONwhere 𝑾 𝑛𝑟𝑡 is the weight matrix to project the embeddings to the 𝑢-dimensional vectors. 𝜎 denotes the softmax activation function.𝒚 is the ground truth label. NRT task enables the pre-trained model to learn relative high-order relationships among the neighbors.In order to incorporate topological information into language models, we introduce three pretraining tasks: TMLM, MNP, and NRT. offered by these diverse tasks. While a straightforward strategy involves training these tasks sequentially, this approach may encounter the challenge of Knowledge Forgetting, as highlighted in previous research [22] . In such a scenario, the learned model tends to prioritize the final task, potentially forgetting knowledge acquired from preceding tasks. To address this issue, we propose a multi-task training strategy that iteratively trains batches associated with different tasks. This involves offline generation of training samples for each task, with the inclusion of a special token to identify task affiliation. Subsequently, batches corresponding to different tasks are sequentially presented to the model during the training process. The classification layer and objective function are adaptively selected based on the input task identity. Essentially, the proposed strategy ensures the equitable training of the three tasks at the batch level, mitigating the risk of knowledge forgetting.While the preceding section delves into the incorporation of topological information into language models, our focus extends to enhancing graph topology through semantic similarities. User behavior, characterized by a preference for trending and popular items, results in a power-law distribution of node degrees [45] . This distribution implies that a majority of items have limited connections (e.g., on a prominent advertising platform, over 64% of online items receive fewer than three clicks in a month). Consequently, these tail nodes do not fully benefit from topological connections. To address this limitation, our objective is to augment the original graph topology by introducing new edges based on semantic similarities. Metric learning is employed to generate the semantic graph:EQUATIONwhere E s ∈ R 𝐾 ×𝐾 is the learned semantic graph, and h i denotes the embedding of item 𝑝 𝑖 learned by the pre-trained models. 𝜖 is a hyperparameter to control the sparsity of the learned semantic graph. The function F is the similarity calculation function. To ensure the model efficiency, here F is defined as the cosine similarity, and thus the nearest neighbor search could be solved by the efficient ANN search algorithms [35] .In addition, based on the motivation that items with similar semantics tend to share similar neighbors, the hybrid graph is further proposed to generate new edges by deeply fusing the semantics and graph connections. Based on the original graph E and the semantic graph E 𝑠 , the hybrid graph E ℎ is formulated as: E h = E 𝑠 E. The semantic closeness is propagated through the topology connections to provide rich hybrid correlations.Overall, we can achieve three types of graphs: the original graph E, the semantic graph E 𝑠 and the hybrid graph E ℎ , which are combined together as the final enriched topological structure.Based on the pre-trained language models and the enriched interaction graphs, we further introduce the multi-channel co-training module, as shown in Figure 4 , to aggregate the item attributes and the interaction graph. Traditional GNNs [4, 13, 37] usually encode the node attributes into static embeddings, which are frozen in the GNN training process. Such pre-learned static embeddings cannot be ensured to align with the downstream tasks, which may hinder the recommendation performance. Thus, we propose to co-train the item attribute modeling module and the topology aggregation module. The major challenge lies in the heterogeneity of the interaction graph. First, the item has a set of attributes such as title, description, category, and price. These attributes have distinct data formats and various characteristics. Conventional GNN models usually encode all the attributes into a single item embedding, and aggregate such representations from the neighborhood. However, the amount of information in different attributes might be different. For example, compared to the long title and description, the information within the price and category has a larger chance of being overwhelmed in the aggregation process. Thus, we propose a multi-channel aggregation paradigm to preserve the fine-grained correlations. Namely, each type of item attribute is viewed as an independent channel to conduct modeling and aggregation. The outputs from the final layer of different channels are combined as the final representation. For the sake of clarity, we use the semantic channel to describe the channels of title and description, and the categorical channel contains the price and category.In the semantic channel, we aim to colearn the NLU and GNN models in a deeply coupled manner. Similar to [20] , we adopt the tightly-coupled learning paradigm in which multiple layers of graph encoder and textual encoder are alternately stacked. A [CLS] token is padded in the front of each text as the sentence representation.For the graph encoder, the input interaction graph contains three types of edges generated by the graph topology enrichment. Thus, we design the heterogeneous graph aggregations to capture these fine-grained correlations. In the 𝑙-th layer of graph encoder, the [CLS] token embeddings of the center node and its neighbors are collected as a matrix M (𝑙 -1) . The homogeneous graph encoder would be formalized as follows:EQUATIONwhereEQUATIONin which matrices W (𝑙 -1) 𝑄, W(𝑙 -1) 𝐾, W(𝑙 -1) 𝑉 ∈ R 𝑑 ×𝑑 denote the trainable variables. Here we further modify its architecture to fit the heterogeneous setting. Assume item 𝑝 𝑖 and 𝑝 𝑗 is connected by the semantic edge in E 𝑠 with the weight 𝑐 𝑖 𝑗 . The original attention is modified as follows:EQUATIONin which 𝑠 denotes the type of connected edge. We utilize different attention parameters for different types of edges, which contributes to capturing the heterogeneity. In addition, the input graph is essentially a weighted graph, and the weights of edges denote the degrees of closeness. Thus, the graph encoder incorporates the weights via a bias term to the attention module. In the graph encoder, the center node and the neighbor nodes can reciprocally exchange messages, which embed the topological interaction information into the learned [CLS] embeddings. Then in the textual encoder, the topological [CLS] embeddings are dispatched to the related nodes. For each node, we can achieve a matrix M (𝑙 -1) 𝑠 ∈ R (𝑚+1) ×𝑑 by consolidating the [CLS] embedding and the embeddings of 𝑚 original tokens. Matrix M (𝑙 -1) 𝑠 would be fed into the textual encoder similar to Equation (4) to convey the neighborhood information from the [CLS] token to the original textual tokens. By these alternately stacked textual and topological encoders, the semantics within a single item and the interactions are deeply fused to learn powerful item representations.For the categories and prices of the items, we view them as categorical features. The embeddings of the categories are randomly initialized as the look-up table and will be updated to fit the training signals. For the prices, we segment and sort these float values into bins. The identities of the belonged bins are viewed as categorical features, and thus the continuous features are converted to categorical ones. The topological aggregation process of the categorical channels is the same as Equation ( 6) by incorporating the types and weights of the edges.Learning. We can achieve an embedding vector for each channel that preserves semantics and interactions. Considering that the informativeness of various channels might be different, the attention mechanism is further employed to learn the distinct importance of channels. Assume h 𝑡 𝑝 , h 𝑑 𝑝 , h 𝑝 𝑝 and h 𝑐 𝑝 denote the representations learned from the title channel, description channel, price channel and category channel, respectively. For instance, the attention score of the title channel is defined as follows: EQUATIONEQUATIONThe enhanced efficacy of the multi-channel mechanism arises from the presence of distinct types of information within each channel. This allows the model to discern the intrinsic characteristics of individual channels and glean valuable insights. Nevertheless, the intricate relationships and collaborative correlations among different channels have been underexplored. To address this gap, we introduce contrastive learning techniques into both intra-channel and inter-channel contexts to reinforce collaborative correlations between various channels.Contrastive learning is systematically incorporated both intraand inter-channel, specifically within the semantic and categorical channels. The objective is to reinforce collaborative associations within each channel and foster connectivity between the two channels. To capture the collaborative information within each channel, taking the semantic channel as an example, we design an intrachannel contrastive loss with contrastive instance (h 𝑡 𝑝 , h 𝑑 𝑝 ). The corresponding contrastive learning loss is formally defined as:EQUATIONwhere 𝜏 denotes the temperature in softmax. 𝑠 (•) denotes a cosine similarity function that quantifies the similarity between two representations. L 𝑆 𝑖𝑛𝑡𝑟𝑎 denotes the contrastive loss within the semantic channel. Turning attention to the categorical channel, contrastive instances are obtained from both the category and price channels. The analogous contrastive learning loss for the categorical channel, denoted as L 𝐶 𝑖𝑛𝑡𝑟𝑎 , is derived in a similar manner. In addition, to capture the correlations between the semantic channel and the categorical, we further design an inter-channel contrastive learning loss. Contrastive instances between the semantic channel and the categorical channel are obtained by aggregating the corresponding embeddings:EQUATIONEQUATIONwhere 𝛼 represents the normalized importance of the corresponding channel. Subsequently, the obtained embeddings h 𝑠𝑐 𝑝 and h 𝑐𝑐 𝑝 serve as contrast instances, and the corresponding contrastive learning loss is defined as follows:EQUATION)where L 𝑖𝑛𝑡𝑒𝑟 denotes the contrastive loss across the semantic and categorical channels.Training Objective Function. The BPR loss [29] is selected as the training objective function, which measures the relative order of the positive pairs and negative pairs. BPR assumes that the recommendation score of the positive node pairs should be higher than the corresponding negative ones. The objective function L B P R of our model is formulated as:EQUATIONwhereO = 𝑚 {(𝑝, 𝑝 + , 𝑝 -)|(𝑝, 𝑝 + ) ∈ E, (𝑝, 𝑝 -) ∈ E -} denotes thetraining samples. In our implementation, we take advantage of "in-batch negative samples" [16, 24] to reduce the computation cost. 
experiments 4.1 datasets: 
 We use four datasets to evaluate the performance of the proposed TESPA model. Datasets of ProductAds and ProductReco are collected from different niche markets of a popular on-line advertising platform. We first collect the users' click behaviors within a fixed time interval (e.g., six months) and then construct the co-click graph. Each node in the co-click graph denotes an item, while the weights on the edges indicate the co-click frequency. To reduce the influence of user behaviors' uncertainty, we remove the edges with weights less than 3. After that, the edge weights are normalized into (0,1). Each item is associated with four types of attributes: title, description, category, and price. To split the training and test sets, we randomly select 10% edges from the co-clicked graph as the test set, and the rest edges are viewed as the training set. A fixed number of edges are randomly selected as the validation set. In addition, two popular datasets, Yelp1 and Amazon-sports2 are also incorporated. Table 1 presents the detailed data statistics.To extensively evaluate the performance of our proposal, we select four types of methods as the baselines. Firstly, we select three NLU models to conduct item recommendation based on the pure semantic information: DSSM [12] , LSTM [14] and BERT [2] . Besides, four popular recommendation models MF [11] , NCF [10] , NGCF [36] and LightGCN [7] are selected as the baselines. In addition, we also adopt a set of popular GNN models (GCN [17] , GAT [32] ,GraphSAGE [6] ) to verify the importance of the co-training paradigm. Finally, recently several works have been proposed to jointly train the GNNs and NLUs, such as TextGNN [50] , AdsGNN [20] , HashCODE [27] and Heterformer [15] , which are also introduced as the baselines.The evaluation task is defined as a retrieval problem, i.e., to predict whether two input items would be co-clicked based on the historical interactions and their intrinsic attributes. We adopt three typical ranking metrics, including Precision@1, NDCG, and MRR. For each testing case, an item will be associated with 300 candidates: 1 positive plus 299 randomly sampled negatives for evaluation.The fundamental checkpoint of the pre-trained model in all the methods is set to the "Bert-base-uncased" in the huggingface 3 . For the semantic-based models, the prices and categories are viewed as learnable embedding in the lookup tables, and directly concatenated with the outputs from the semantic models. For the GNNs, we use the pre-trained language models to encode the textual information into the low-dimensional attribute vectors. Other item attributes are incorporated into the baseline models in a straight-forward manner, such as concatenation. For the proposed TESPA model, the training batch size is set to 32, the learning rate is set to 1𝑒-5, the number of training epochs is 3, and the number of layers in graph transformers is set to 3. For the GNN baselines, the number of GNN layers is set to 3. Other hyper-parameters are carefully tuned on the validation dataset. The training is on 16× Nvidia V100-16GB GPUs. We use PyTorch 1.6.0 for implementation.Table 2 presents the off-line performance of different models. We conduct each method five times and report the average performance.From the results, we can easily achieve the following observations:(1) Traditional pure semantic models (LSTM and DSSM) achieve inferior performance compared to other methods, which reveals that user behaviors are crucial to learning desirable item representations.(2) Heterformer and HashCODE are two recent works to co-train the GNN and NLU models under a unified learning framework. Their performance significantly outperforms their counterparts, verifying the effectiveness of the co-training paradigm. (3) Our proposal consistently beats baselines by a large margin over all the metrics. Compared to the best baseline, TESPA improves the performance by 3% on average, which passes the significance testing.The performance improvements are owed to the mutual reinforcement between interactions and item semantics and the appropriate multi-channel graph aggregations.4.5.1 Topological Pre-training Tasks. In section 3.1, three pre-training tasks are proposed to encode the topological connections into the language models, including the TMLM, MNP and NRT. Here we aim to investigate the importance of different tasks. The traditional MLM and NSP tasks are also adopted as baselines. The NSP task is modified to predict the neighbors as the next sentences. Please note that, the graph enrichment and the co-training module are 3 presents the performance of BERT with different pre-training tasks. The following conclusions could be easily obtained: (1) Topology-aware pre-training tasks (e.g., NP, TMLM and MNP) consistently outperform the pure semanticbased task (e.g., MLM), which verifies the importance of interaction graphs in item representation learning. (2) TMLM and MNP consistently beat their counterparts MLM and NP. The performance gains are reasonable because TMLP can capture the topology information while MNP advances the model expressiveness via the corrupted inputs. (3) Model performance is further improved after combining all the pre-training tasks, which reveals that different Besides, we also investigate the effectiveness of the multi-channel aggregation function. The single-channel aggregation is selected as the baseline, which first encodes all the attributes into an embedding and then propagates this embedding to the neighborhood. As shown in Table 6 , the multi-channel method surpasses the single channel model by nearly 1.5% on average. For the single channel approach, attributes with rich semantics might overwhelm the comparatively uninformative ones, leading to information loss and inferior performance.To assess the efficacy of the contrastive learning components within both inter-channel and intra-channel contexts, we conducted experiments by individually removing these two losses. The resultant model performances are presented in Table 6 . Notably, a conspicuous decline in model performance is observed after the removal of these losses. This decline underscores the pivotal role played by the contrastive learning losses in enhancing the model's capability to capture nuanced relationships within and between channels. This deterioration in performance suggests that the effectiveness of the model is intricately tied to the integration of contrastive learning mechanisms. The removal of these losses appears to compromise the model's ability to leverage the inherent relationships and collaborative correlations among different channels, underscoring the importance of the proposed contrastive learning framework in our approach.Here we study the performance sensitivity of our proposal on two core parameters: the sparsity weight 𝜖 in Equation ( 3) and the training batch size 𝑏 of Equation (13) . As the performance trends on 0.5 0.6 0.7 0.8 0.9 the two datasets of ProductAds and ProductReco are similar, here we only report the results on ProductAds dataset. The NDCG and MRR scores under different settings on two datasets are recorded. Figure 5 presents the experimental results. From the left sub-figure, one can see that with the increase of 𝜖, the performance over all the datasets first increases and then decreases. A larger 𝜖 leads to a sparser graph with limited topology information, while noise may be introduced if 𝜖 is too small. Thus, 𝜖 should be carefully tuned to achieve desirable performance. From the right sub-figure, we can see that with the increase of 𝑏, model performance first significantly increases and then slightly increases or keeps steady. This is reasonable as at the beginning, a larger batch size provides more in-batch negative samples to boost the discriminative capability. However, model performance would keep steady when the information in the negative instances is sufficient to provide accurate training signals.The on-line performance analysis can be found in Appendix A, and the case study of online services can be found in Appendix B. 
related work: 
 The item representation is one of the crucial issues for intelligent web services. The historical works can be divided into three major categories based on their utilized information: topological-based methods [1, 3, 8, 9, 18, 30, 31, 34, 38, 39, 46] , textual-based methods [21, 25, 33, [42] [43] [44] , and combined-based methods [15, 23, 26, 27, 41, 47, 49, 50] . Topological-based methods rely on topological features, and textual-based methods rely on the textual feature for item representation. Combined-based methods leverage the combination of topological and textual features for better representation quality. Further details on related work can be found in Appendix C. 
conclusion: 
 In this paper, we extensively study the critical task of item representation learning. Our insight lies in mutually reinforcing the semantic modeling within a single item and the interaction modeling across different items. Unlike previous loosely coupled methods, we propose a novel TESPA model to deeply fuse the item semantics and interaction graph. TESPA encodes the knowledge from the user behaviors into the language models and then enriches the graph topology based on the semantics. Finally, the NLU and GNN models are co-trained under a unified learning framework in a novel multi-channel manner. TESPA achieves significant performance improvements on both on-line and off-line scenarios, demonstrating the superiority of our proposal. 
 
 Appendix: https://www.yelp.com/datasethttps://nijianmo.github.io/amazon/#subsetshttps://github.com/huggingface/transformers/ 


Decision: The reviewers agree that the paper is relevant to the scope of ACM Multimedia. They are also unanimous in the opinion that it should be included in the program. Among other, they were positive about the novelty of the proposed approach and the incorporation of the interaction graph. However, they have also pointed to some drawbacks, such as a lack of in-depth motivation for the proposed approach.

Taking into account the initial reviews and the rebuttal process, I suggest acceptance of the paper in line with the reviewer recommendations.

Reviews: review: paper_summary: paper_summary: The paper presents an approach for learning better item representations by blending semantic item information with interaction data and then training it under a unified "multi-channel co-training" framework. The author demonstrate SOTA performance on retrieval tasks.
strength_weakness: strength_weakness: 1. The proposed approach appears to be novel, and the recipe for "Topology-Driven Pre-Training of Language Models" is an
interesting contribution.
2. Incorporation of interactions via a graph is an interesting contribution that does appear to lead to better performance.
limitations: limitations: 1. The motivation for this work is fairly weak. The core contribution is "an improved item representation," but the paper
immediately delves into the technical part without formally motivating how existing representations are constructed and how
they're deficient on a technical level.

2. The figures, especially Figure 2, do not provide much information about the overall construction of the model. Furthermore, the
paper is quite dense and introduces its core contributions without formal motivation.
paper_summary: paper_summary: The work discusses the importance of learning effective item representations for e-commerce applications. It proposes a novel model called TESPA that aims to improve item representation learning by mutually reinforcing semantic modeling (capturing the intrinsic characteristics of an item) and interaction modeling (capturing relationships between items based on user behaviors like co-clicks or co-views). The key aspects of TESPA are:Encoding Interaction Signals: It encodes collaborative filtering signals from the interaction graph into language models through fine-grained topological pre-training. (i) Enriching Interaction Graph: The interaction graph is further enriched based on semantic similarities between items. (ii) Multi-Channel Co-Training: A novel multi-channel co-training paradigm is proposed to deeply fuse the semantics and interactions under a unified framework.
The authors claim that TESPA can leverage the strengths of both semantic and interaction information to facilitate better item representation learning. Experimental results from online and offline evaluations demonstrate the superiority of the proposed TESPA model.

The work is relevant and timely
The work is based two ideas: (i) to teach language models to understand interaction signals between items through continuous pre-training. (ii) exploiting the semantic similarities between the items to enhance the cross-item message passing, which addresses the data sparsity issues. Authors proposes to have a  tightly coupled co-training paradigm to deeply fuse the semantics and interactions in a multi-channel manner.
strength_weakness: strength_weakness: The work integrates a number of features such as to understand interaction signals between items and also exploits the semantic similarities

the work integrates graph features and NLU methods
limitations: limitations: evaluated on four datasets

ignored a  set of other metrics Recall@5,10,20, Precision@5,10,20, which are used widely used in experiments
paper_summary: paper_summary: This paper explores methods to enhance the joint training of GNN and NLU modules for item representation learning in recommendation scenarios. Accordingly, a new model named TESPA is proposed. TESPA begins by pre-training the NLU model using multi-grained topological signals. It then further refines the original graph topology through semantic closeness. Finally, a multi-channel training module is employed to aggregate the final item embeddings. Extensive experimental results validate the effectiveness of TESPA. Overall, the quality of this paper is good and I am positive about it.
strength_weakness: strength_weakness: 1. The paper is well-organized and well-written. 
2. This work investigates the problem of item representation learning by mutually reinforcing semantic modeling and interaction modeling. 
3. The proposed TESPA consists of three parts: topology-driven pre-training of language models, semantic-based graph enrichment, and multi-channel co-training module.  Overall, the proposed solution is sound and reasonable.
4. Table 2 shows that the proposed method consistently outperforms all compared baselines, achieving significant performance improvements.
5. Comprehensive ablation experiments are provided.
limitations: limitations: 1. The novelty of this paper is incremental. The overall framework of the method is based on the joint training of GNN and NLU modules for item representation learning. However, the relationship with recent similar works [15, 20, 27, 50] is not well explained, making it hard to capture the unique aspects of this paper.
2. Although this paper demonstrates the effectiveness of the designed modules through extensive experiments, it provides an insufficient explanation of the motivation behind the method. While the method's performance improvements are noteworthy, insights into why the method works should be more intriguing.
3. This paper lacks a comparative analysis between the proposed TESPA and HashCODE and Heterformer. Since these methods have similar designs (say, jointly train the GNNs and NLUs), the reasons for the performance improvements should be further elaborated.
4. It’s better to analyze the time complexity of the TESPA or efficiency performance.