Title: Enhancing User Intent Capture in Session-Based Recommendation with Attribute Patterns
Venue: NeurIPS 2023
Accepted: yes
Keywords: session-based recommendation

representation learning

pattern mining
Abstract: The goal of session-based recommendation in E-commerce is to predict the next item that an anonymous user will purchase based on the browsing and purchase history. However, constructing global or local transition graphs to supplement session data can lead to noisy correlations and user intent vanishing. In this work, we propose the Frequent Attribute Pattern Augmented Transformer (FAPAT) that characterizes user intents by building attribute transition graphs and matching attribute patterns. Specifically, the frequent and compact attribute patterns are served as memory to augment session representations, followed by a gate and a transformer block to fuse the whole session information. Through extensive experiments on two public benchmarks and 100 million industrial data in three domains, we demonstrate that FAPAT consistently outperforms state-of-the-art methods by an average of 4.5% across various evaluation metrics (Hits, NDCG, MRR). Besides evaluating the next-item prediction, we estimate the models' capabilities to capture user intents via predicting items' attributes and period-item recommendations.

Enhancing User Intent Capture in Session-Based Recommendation with Attribute Patterns

Abstract

The goal of session-based recommendation in E-commerce is to predict the next item that an anonymous user will purchase based on the browsing and purchase history. However, constructing global or local transition graphs to supplement session data can lead to noisy correlations and user intent vanishing. In this work, we propose the Frequent Attribute Pattern Augmented Transformer (FAPAT) that characterizes user intents by building attribute transition graphs and matching attribute patterns. Specifically, the frequent and compact attribute patterns are served as memory to augment session representations, followed by a gate and a transformer block to fuse the whole session information. Through extensive experiments on two public benchmarks and 100 million industrial data in three domains, we demonstrate that FAPAT consistently outperforms state-of-the-art methods by an average of 4.5% across various evaluation metrics (Hits, NDCG, MRR). Besides evaluating the next-item prediction, we estimate the models' capabilities to capture user intents via predicting items' attributes and period-item recommendations.

1 Introduction

With the explosive demand for E-commerce services [44, 12, 23, 11, 43], numerous user behaviors are emerging. Understanding these historical action records is critical in comprehending users' interests and intent evolution, particularly in a cold-start regime that lacks sufficient context. This has spurred research on session-based recommendations (SBR) [8, 35, 37, 13] that capture user-side dynamics from a short-time period (namely a session) using temporally historical information. Numerous SBR algorithms have been proposed, ranging from sequence-based methods [27, 8, 16, 19, 31, 28, 9] to graph-based methods [35, 38, 34, 37, 17, 30] for learning dynamic user characterization. However, both lines of methods have their limitations. Specifically, sequence-based methods treat each user behaviors in a session as an action sequence and model the local dependencies inside. This can only capture users' preference evolution via chronological order while failing to identify the complex non-adjacent item correlation, especially when the session length is insufficient to support temporal prediction [33]. To address this issue, graph-based methods adopt a higher perspective by introducing a global item transition graph, which aggregates local session graphs constructed from historical session sequences. Thus, a newly-emerging short session sequence can benefit from the global topology (e.g., global co-occurrences) and representations (e.g., item semantics) [34]. Unfortunately, existing session graph construction ignores temporal signals. Figure 1 shows that two different sessions result in the same session graph, leading to vanishing of user intent variation during sequence-to-graph conversion. And such global graphs are fragile due to noise from random clicks.

Timeline

Session graph

(Item correlation)

Timeline

Session sequence

(Temporal information)

brand

color

type

tablet→notebook→cell phone→cell phone→cell phone→cell phone→tablet→

?

silver→silver→blue→blue→burgundy→blue→silver→

?

Apple→Apple→Apple→Apple→Samsung→Apple→Apple→

?

Attribute Sequences

Timeline

Session Sequence

(Temporal Information)

(Intent Information)

?

Apple

Samsung

?

blue

burgundy

silver

?

cell phone

tablet

notebook

Attribute Graphs

↑

notebook

Ccellphone

f

tablet

1

=

Apple

Samsung

blue

burgundy

11

silver

Besides the global item transition graph, there are other possible solutions for item correlations. One feasible solution involves item-side knowledge. Items can be connected through shared attributes, such as being manufactured by the same company [28]. We argue that the current use of item-side metadata provides little assistance in SBR models as user intent may change over time. However, such meta-data are still useful from the view of graphs. As illustrated in Figure 2, session attributes can be organized into attribute graphs and anchored to the local session graph to create multiplexes. Such multiplexes provide several intent clues, in addition to item-side correlations. For example, the color pattern silver ↔ silver ↔ blue ↔ blue reveals the user's color intent, while the brand pattern Apple ↔ Apple ↔ Samsung implies a potential change in intent. Different sessions can benefit from shared attribute topologies and representations, which can ultimately entail implicit high-order correlations. However, the session graph is essentially a general conditional random field, and optimization becomes intractable due to the large candidate size. Graph neural networks (GNNs) also face challenges due to the possibility of over-smoothing and data noise [46, 18, 45, 32].

To alleviate the aforementioned issues, we propose a novel framework called Frequent Attribute Pattern Augmented Transformer ( FAPAT ) that considers highly frequent attribute patterns as supplementary instead of directly learning multiplex graphs. In traditional graph learning, graphlets (such as triangles, triangular pyramids, etc.) have been proven to be useful features in graph classification and representation learning [29, 15]. Therefore, we employ frequent graph pattern mining algorithms to find consequential graphlets and view them as compact hyper-edges (e.g., cell phone , tablet , notebook in Figure 2). We then use these attribute patterns as accessible memory to augment session sequence encoding. Before encoding patterns, we use Jaccard similarities to rank and retrieve the most highly correlated patterns, which significantly reduces the graph density and computational cost. It has been shown that GNNs can estimate the isomorphism and frequency of substructures [39, 21, 20]. Thus, we leverage multi-head graph attention to learn pattern and local session graph representations in the aligned space [10, 22]. To incorporate temporal signals and capture user intents, we distribute graph representations back to session sequences and use external pattern memory to augment sequence representations via memory attention with relative position bias. Finally, the sequence is fully fused by a transformer block. In other words, graph information is used to aggregate attribute patterns, while temporal actions are used to encode items.

To validate the effectiveness of FAPAT, we conduct extensive experiments on two public benchmark datasets and three real-world large-scale industrial datasets with around 100 million clicks, and experimental results demonstrate significant improvement with an average boost of 4.5% across various evaluation metrics (Hits, NDCG, MRR). Compared with baselines, the attribute pattern density can significantly relieve over-smoothing. Besides, we also extend evaluation to attribute estimations and sequential recommendations to measure the model capability to capture user intents. Code and data are availiable at https://github.com/HKUST-KnowComp/FAPAT .

2 Related Work

Neural Methods for Session-based Recommendation Systems . Table 1 presents a summary of the distinctions between current neural techniques and our novel FAPAT. The concept of Temporal Information implies Markov decision processes with previous histories. The technique of History Attention employs attention for learning long-distance sequences. The approach of Local Session

Topology involves modeling session sequences from the view of session graphs. Lastly, Global Item Correlation and Attribute Association place emphasis on capturing item-side and attribute knowledge.

Sequence-based Models . FPMC [27] uses first-order Markov chain and matrix factorization to identify sequential patterns of long-term dependencies. However, the Markov-based method usually has difficulty exploring complicated temporal patterns beyond first-order relationships. Recently, neural networks have shown power in exploiting sequential data in SBR tasks, such as GRU4Rec [8]. NARM[16]extends GRUs with attention to emphasize the user's primary purchase purpose. Similarly, STAMP [19] uses an attention-based memory network to capture the user's current interest. These attention-based models separately deal with the user's last behaviors and the whole session history to detect the general and latest interests. But they mainly focus on the user's preference from a temporal view but ignore the item correlations. Pre-training techniques [47] and multi-task learning [28] also demonstrate effectiveness in injecting item metadata to embeddings and predicting item attributes. Besides, some recent sequence-based approaches leverage generative pretrained language models to provide explicit explanations for recommendation systems [4, 6].

Graph-based Models . Graph neural networks (GNNs) have recently been explored in SBRs due to the substantial implications behind natural transition topologies. SR-GNN [35] adopts a gate GNN to obtain item embeddings over the local session graph and predict the next item with weighted sum pooling, showing impressive results on benchmark data. Some advanced variants have further boosted performance, such as GC-SAN [38] with self-attention mechanism and FGNN [24] with weighted attention graph layers. To acquire further collaborative information, S2-DHCN [31] constructs line graphs to capture correlations among neighbor sessions, and GCE-GNN [34] directly applies a graph convolution over the global transitions to aggregate more relevant items for local sessions. However, GNN-based methods still face challenges in capturing temporal signals, filtering noise, and leveraging implicit high-order collaborative information. Other methods, such as LESSR [1] and MSGIFSR [7], have also shown significant improvement by building multigraphs and shortcut graphs for session representation learning and user intents from different granularities, respectively.

Pattern Mining for Recommendation Systems . Pattern mining is an important data mining technique with board applications. In recommendation systems, sequential pattern mining assists in analyzing customer purchase behaviors through frequent sequential patterns. Such mining focuses on item patterns with frequencies above a threshold in all sessions, which reduces the diversity of the recommended items [2]. Personalized sequential pattern mining [42] effectively learns user-specific sequence importance knowledge and improves the accuracy of recommendations for target users. It can be challenging to generalize to SBR systems when neural networks have already implicitly captured such behavior patterns. But attribute graph patterns still need to be explored in SBR.

3 Background and Motivations

3.1 Problem Definition

Session-Based Recommendation (SBR) assumes that users' historical behaviors outside the current session are inaccessible, in contrast to general recommendations. For example, users do not log in due to user privacy and security reasons. SBR predicts the next item that an anonymous user is most

likely to click on or purchase based on historical behaviors within a short period. Despite the lack of personal profiles, this universal setting can better reflect the quality of item-side recommendations. Suppose that there are N unique items in the database, and each session is represented as a repeatable sequence of items S = [ v 1 , v 2 , . . . , v L ] , v i ∈ V (1 ≤ i ≤ L ) represents the i -th behavioral item of the anonymous user within session S , where V is the item set collected from overall sessions, and L is the length of the session. Given a session S , the goal is to recommend the topK items (1 ≤ K ≤ N ) that have the highest probabilities of being clicked by the anonymous user.

3.2 Session Graphs and Transition Graphs

Session sequence modeling is not always sufficient for SBR as it only reflects transitions from the user side. To account for item correlations, SR-GNN [35] converts session sequences to session graphs. Each session graph G S = ( V S , E S ) is a directed graph with node set V S ⊆ V , consisting of unique items in the session, and edge set E S , recording adjacent relations between two items in session S . Edge weights can be normalized by indegrees or outdegrees to model transition probabilities. GCE-GNN [34] extends this graph modeling by merging all session graphs as a global transition graph, which aggregates more item correlations but faces over-smoothing and data noise [46, 18].

Instead of modeling global item transitions, we enrich session graphs with attributes and patterns. Assume there are M different kinds of attributes, and the m -th attribute type A ( m ) (1 ≤ m ≤ M ) consists of |A ( m ) | possible values. Thus, each item v ∈ V S has attribute list [ a (1) v , a (2) v , · · · , a ( M ) v ] , where a ( m ) v ∈ A ( m ) denotes the m -th attribute value of v . Each session sequence S corresponds to M attribute histories, with the m -th attribute sequence denoted as S ( m ) = [ a ( m ) v 1 , a ( m ) v 2 , · · · , a ( m ) v L ] . Using the sequence-to-graph transform, we convert S ( m ) to G ( m ) S , whi is usually denser than the session graph G S . Finally, the M attribute sequences are separately transformed into M attribute session graphs in different property-specific channels, anchored in items V S . Finally, we add edges for item v i and attribute a ( m ) v i to construct a multiplex session graph, preserving attribute values and transitions. We represent the session graph with attributes as the multiplex G A S (as Figure 2).

3.3 Frequent Pattern Mining

Frequent pattern mining aims to extract inductive clues from data to comprehend data distributions, which includes two main categories: sequential pattern mining and graph pattern mining. The former is concerned with sequence databases composed of ordered elements, while the latter statistics the important graph structures. For two sequences S ′ = [ v ′ 1 , v ′ 2 , · · · , v ′ L ′ ] and S = [ v 1 , v 2 , · · · , v L ] , we refer to S ′ as the pattern of S if S ′ is a subsequence of S . Similarly, a graph G S ′ = ( V S ′ , E S ′ ) is a subgraph of G S = ( V S , E S ) if V S ′ ⊆ V S and E S ′ ⊆ E S . Compared with sequence pattern mining, graph pattern mining is more general since it involves the structural topology and attribute information. For instance, as depicted in Figure 2, { cell phone , tablet , notebook } corresponds to a triangle. It may be challenging to discover the triangle from the temporal sequence, but it is a vital clue from the graph view. Thus, we stick on frequent graph patterns rather than sequence patterns.

4 Methodology

We present the Frequent Attribute Pattern Augmented Transformer (FAPAT), a novel framework that captures user intents and item correlations. Our method is built upon session sequences and corresponding attribute graphs. Initially, we mine frequent attribute patterns from the attribute graphs to explore coarse-grained item correlations. These patterns are then used as memory to enhance the session encoder, which consists of graph-nested transformer layers. Figure 3 illustrates the overview.

4.1 Frequent Attribute Pattern Acquisition

In this subsection, we describe how to extract frequent patterns from training recommendation sessions. The aim of frequent pattern mining is to minimize the impact of random clicks in the global transition graph and avoid over-smoothing when learning multiplex attribute graphs. To achieve this, we design a mining-filtering paradigm to ensure representativeness.

…

Multi-Head

Self Attention

Add & Norm

Multi-Head Graph

Attention

…

Multi-Head

Memory Attention

Gate

Graph2Sequence

Multi-Head

Memory Attention

Predictor

Multi-Head

Memory Attention

Multi-Head

Graph Attention

FF & Add & Norm

Pattern Structures

Pattern Mining

Training

Attribute

Graphs

…

Attribute Pattern

Retrieval

Multi-Head

Memory Attention

Multi-Head

Memory Attention

Multi-Head Graph

Attention

Multi-Head

Graph Attention

𝒫 (1)

𝒫 (2)

𝒫 (𝑀)

𝒑 2

𝒑 𝑀

𝒑 1

Sequence2Graph

Attribute Pattern

Retrieval

Attribute Pattern

Retrieval

ഥ

𝒉𝑖

2

ഥ

𝒉𝑖

𝑀

ഥ

𝒉𝑖

1

ഥ

𝒉𝑖

ഥ

𝒉𝑣𝑖

Pattern Filtering

Session Sequence

𝑆

𝒢𝑆

𝒜

4.1.1 Graph Pattern Mining

Small graph patterns, also called motifs or graphlets, are valuable features in graph learning and property prediction [29, 15], exhibiting strong statistical correlations between graph structures and node semantics. In this study, we focus on similar graphlet structures but extend to SBR scenarios. We collect patterns consisting of no more than four nodes (representing different attribute values), and further restrict them to those containing either a circle or a triangle to significantly reduce the number of candidates. We adopt gSpan [40] to acquire undirected patterns from attribute session graphs and keep patterns belonging to one of twenty types shown in Figure 3.

4.1.2 Loose Pattern Filtering

However, complex patterns may contain smaller ones. For instance, the first pattern in Figure 3 is a subgraph of the second. While each subgraph has an equal or higher frequency than its supergraph, such loose patterns do not convey much information. To eliminate them, we employ VF2 [3] subgraph isomorphism algorithm to filter. If a pattern P ′ is a subgraph of another pattern P , then P ′ is excluded from the pattern candidates. Finally, we retain compact and frequent patterns for session encoding.

4.2 Intent-aware Sequence Encoding

To recommend items of high interest to users, we use a GAT-based encoder to learn pattern representations from the item side, which are then served as memory to augment session encoding.

4.2.1 Relevant Graph Pattern Retrieval

The input item sequence is converted to a multiplex session graph representing the transitional information of different item attributes, as depicted in Figure 1. To improve graph representations by utilizing frequent attribute subgraphs, we retrieve relevant patterns from those mined in §4.1. For the m -th attribute type A ( m ) (1 ≤ m ≤ M ) , we denote an arbitrary subgraph mined from the previous step as G ( m ) P = ( V ( m ) P , E ( m ) P ) . Then we retrieve at most I subgraph patterns that have the most considerable Jaccard similarities to the transition graph G ( m ) S as Eq. (1). This can be done within O ( |V ( m ) S | × |V ( m ) P | ) , but this is always linear because of |V ( m ) P | ≤ 4 .

4.2.2 Attribute Pattern Representation

After retrieving the relevant patterns for the corresponding multiplex session graph, we encode them using multi-head relational graph attention for further memory augmentation. For a pattern G ( m ) P for the m -th attribute, we compute the attention weight for two arbitrary nodes a ( m ) i and a ( m ) j by:

where e a ( m ) i denotes the embedding of a ( m ) i , ◦ indicates element-wise multiplication, N ( a ( m ) i ) represents the neighbors of a ( m ) i (including itself), and r ( m ) ij corresponds to the relation-specific vector. Then the representation of h a ( m ) i for node a ( m ) i in the pattern is aggregated by:

After computing the representation of each node in pattern subgraph G ( m ) P , a pooling layer (e.g, average pooling) over all node representations is to aggregate the pattern presentation for G ( m ) P :

4.2.3 Attribute Memory Augmentation

Meanwhile, we also employ graph attention to obtain the native graph representations of the multiplex session graph G ( m ) S . We apply this to each m -th attribute transition graph G ( m ) S , computing node representations a ( m ) i using Eq. (2) and Eq. (3) and denote the resulting node representation as h ( m ) v i . To compute the aggregated attribute representation of each node v i in the original session graph G S , we combine its different representations from all attribute transition graphs by:

where e v i is the representation of v i by the item embedding lookup.

Suppose all retrieved attribute patterns associated with A ( m ) are P ( m ) , whose representations from Eq. (4) are { p ( m ) | p ( m ) ∈ P ( m ) } . To preserve temporal information when utilizing these patterns, we map graph representations { h ( m ) v i | v i ∈ V S } to sequence representations { h ( m ) i | 1 ≤ i ≤ L } with length L . Next, we concatenate all pattern representations { p ( m ) | p ( m ) ∈ P ( m ) } , all session representations { h ( m ) i | 1 ≤ i ≤ L } , and two special embeddings e CLS and e MASK to separate the two parts and indicate the item to predict:

Then, we augment sequence encoding with memory, taking inspiration from TransformerXL [5] and Memorizing Transformer [36]. As shown in Figure 4, we introduce T5 relative position bias [25] to distinct short and long-term histories. Furthermore, we utilize unidirectional attention to focus solely on the user's current intent instead of global intent, which may become increasingly noisy over time.

4.2.4 User Intent Aggregation

To aggregate memory-augmented representations from M different attributes and pattern sets, we employ gating mechanism:

𝒑 𝑚 |𝑝 𝑚

∈ 𝒫 (𝑚)

𝒆CLS

+

𝒕 -𝑖

ഥ

𝒉 𝑗

(𝑚)

+𝒕𝑗-𝑖|𝑗 < 𝑖

ഥ

𝒉 𝑖

(𝑚)

+𝒕0

ഥ

𝒉 𝑘

(𝑚)

+𝒕𝑘-𝑖|𝑘 > 𝑖

𝒆MASK + 𝒕𝐿+1-𝑖

…

…

where W ( m ) β ∈ R d × d and b ( m ) β ∈ R d are trainable parameters ( d is the dimension of hidden states), ˆ h ( m ) i is the memory-augmented results, and β ( m ) i controls the importance of attribute patterns. While ˆ h i combines attribute information, it disregards long-range histories as h ( m ) i solely accesses one-hop neighbors. Thus, we incorporate a transformer block to aggregate global session information:

where ˆ h CLS and ˆ h MASK are directly taken from Eq. (6) since they do not belong to the original session but still observe the memory patterns and relative temporal signals.

4.3 Next-item Recommendation

Once the sequence representations are obtained, the next step is to predict the next item that may interest the user for clicking or purchasing. We adopt the approach used in previous work [35, 34, 37] to concatenate additional reversed positional embeddings as follows:

where h i is the i -th item representation H [ i ] from Eq. (7), and W z ∈ R 2 d × d and b z ∈ R d are trainable parameters. This concatenation of positional embeddings is intended to prioritize nearest intents over long-distance historical purposes. Figure 2 demonstrates this idea through two cases: the male user is more likely to be interested in Android cell phones since his last click was in that category, while the female user sticks to Apple products as she reviews iPhone and iPad once again.

We utilize the representation of the MASK to compute soft attention and then represent the session and the user's latest intent through a weighted average:

where W γ ∈ R d × d and b γ ∈ R d are trainable parameters. We compute the prediction of the next item using a similarity-based approach rather than a linear layer. This approach is similar to the optimization of Bayesian Personalized Ranking (BPR) [26]. Alternatively, we can train the model using cross-entropy minimization. Finally, the prediction is obtained by ˆ y = argmax v ( u ⊤ e v ) .

5 Experiment

5.1 Setup

Datasets . We first evaluate our method on public benchmarks. diginetica contains the browser logs and anonymized transactions, Tmall collects anonymous users' shopping logs on the Tmall online website. We also acquire sessions from the browse and purchase logs from our E-commerce platform. We target at beauty , books , and electronics and gather 20-minute interactions within last successful purchases into one session after removing long-tail items. Appendix C provides more details.

Baselines . We compare our method with seven sequence-based baselines (FPMC [27], GRU4Rec [8], NARM [16], STAMP [19], CSRM [31], S3-Rec [47], and M2TRec [28]) and six graph-based baselines (SR-GNN [35], GC-SAN [38], S2-DHCN [37], GCE-GNN [34], LESSR [1], and MSGIFSR [7]). Model details are given in Appendix D. Each model is aligned with the official code implementation.

Evaluation . We evaluate SBRs as a ranking problem and employ Hits@ K , NDCG@ K ,MRR@ K as standard metrics. Hits@ K measures the percentage of ranks up to and including K , while NDCG@ K

assigns higher scores to hits at the top of the list. MRR@ K is the average of reciprocal ranks, with ranks above K assigned 0. For public benchmarking, we report the average performance across seeds { 2020 , · · · , 2024 } , while for industrial data, we use a fixed seed (see Appendix E for more details).

5.2 Next-item Prediction Evaluation

Experimental Results. We first compare our models with selected baselines on two public datasets in Table 2. Overall, there is no huge difference between sequence and graph models when the session historical information is limited (e.g., in diginetica ). However, we observe a performance boost with the history attention mechanism and session topology. FPMC that utilizes first-order Markov chains and matrix factorization is the worst. This reveals the difference between traditional recommendation and session-based recommendation. In contrast, RNN-based methods (GRU4Rec, NARM, STAMP, and CSRM) show better generalizability, along with the benefits of attention and memory. But we do not see further gains from pretraining and multi-task learning in S3-Rec and M2TRec. On the other hand, GNN-based algorithms for local sessions also achieve comparable results even with the absence of temporal signals. Moreover, the heterogeneous graphs of MSGIFSR beat the shortcut graphs from LESSR, indicating over-smoothing and potential noise. Although explicit global collaborative graphs enhance GCE-GNN, the implicit collaborative information from neurons and patterns in FAPAT easily surpasses other baselines significantly. At the same time, we evaluate algorithms on our 100 million industrial data. Table 3 demonstrates that most sequence models become unstable except S3-Rec. It employs incomplete data to predict masked items and attributes during pretraining and identifies contextualized collaborations by determining whether two incomplete sequences belong to the same session. But the multi-tasking learning for predicting attributes makes M2TRec hard to recommend items. While most GNN methods are competitive, S3-Rec and FAPAT are still better.

Over-smoothing Relief. We examine different graph topologies in baselines and FAPAT by comparing the graph density of different methods: the density of local session graphs (in SR-GNN), global collaborative filtering graphs(in GCE-GNN), shortcut graphs (in LESSR), heterogeneous graphs (in MSGIFSR), attribute patterns (in FAPAT) on E-commerce data are 3.658, 114.910, 25.447, 2.598, and 1.117, respectively. The density decrease significantly relieves the over-smoothing. Detailed analyses and expectations are provided in Appendix B.





















































































(a) Tmall



















































































(b) Books

5.3 Ablation Study

Attribute Pattern Augmentation. To evaluate the effect of attribute patterns, we conduct experiments on variants with single attributes or without any attribute. The same soft attention strategy from Eq. (6) is employed to fuse attribute embeddings for competitive baselines. Results in Figure 5 show that attribute pattern augmentation is more stable than attribute soft attention. FAPAT benefits from graph-nested attention, where the graph attention aligns the hidden space, and the memory attention captures item correlations and user intents. But attribute embeddings may have side effects on optimization in baselines, especially in graph neural networks. We also discover that not all attributes have a positive impact. Comparing among attribute pattern numbers, attribute patterns with significant frequencies (slightly lower than or similar to the item number) can have adverse effects.

Graph-nested Attention. The graph-nested attention is one of our contributions, distinguishing it from GraphFormer [41]. Unlike GraphFormer, our graph attention is integrated inside the blocks, which allows for direct benefit from the broader attention in the following self-attention via backpropagation. To ensure fairness, we replace the encoding module of FAPAT with GraphFormer and vanilla Transformer. Results in Table 4 demonstrate the advantages of our proposed graph-nested attention. Our experiments also show that even a simple vanilla Transformer can outperform previous state-of-the-art models by a significant margin, indicating the importance of emphasizing temporal information in SBRs and the appropriateness of attention for capturing long-distance dependencies.

5.4 Intent Capture Inspection

Attribute Estimation. Beyond item predictions, we also estimate the awareness of user intents from the product attribute side. We do not require models (except M2TRec) to predict attributes but to retrieve attributes from predicted items instead. We consider it a successful estimation if the top-ranked items have the same attribute value as the ground truth. M2TRec performs well on public data with multi-task learning but struggles on industrial E-commerce data with four attributes, as shown in Table 5. Pretraining in S3-Rec is not helpful due to catastrophic forgetting. GNN-based models perform similarly, except on Tmall , where the data are too sparse so that global collaborative information assists. On the contrary, FAPAT achieves robust predictions across four datasets. Even when session data are sufficient, frequent patterns remain effective.



























(a) Tmall





























(b) Books













































(c) Electronics

Period-item Recommendation. In addition to single-step evaluation, we revisit recommenders. A sound and robust SRB system must understand user intents in deep and foresee the possible consistency and the potential change. Autoregressive settings pose challenges for GNNs due to short click histories and error accumulation. Therefore, we evaluate period recommendations like search engines by comparing the top-10 predicted items with the next 3/5/10 clicks. Figure 6 demonstrates period-recommendation performance. Sequence models offer steady results in Recall, indicating that temporal information is one of the prerequisites to analyzing users' latest intents. However, pretraining may hinder models' ability to generalize over long periods, resulting in a severe decline in NDCG. Our FAPAT achieves the best performance in all metrics, indicating the effectiveness of attribute graphlets in capturing deep user intents.

6 Conclusion

Our paper introduces FAPAT, a novel framework that leverages attribute graph patterns to augment anonymous sequence encoding for session-based recommendations. Compared to other GNN-based methods, frequent attribute graphlets can reduce noise and topology densities for enhancing user intent capture. Our sequence encoder can better preserve temporal signals and forecast the user's latest intents. Experimental results clearly illustrate the effectiveness, and extensive ablation studies and intent capture inspections provide additional support. We discuss limitations in Appendix A. One of the future works is to improve ranking priority by combining pretraining and pattern augmentation.

Acknowledgments

The authors of this paper were supported by the NSFC Fund (U20B2053) from the NSFC of China, the RIF (R6020-19 and R6021-20) and the GRF (16211520 and 16205322) from RGC of Hong Kong. We also thank the support from the UGC Research Matching Grants (RMGS20EG01-D, RMGS20CR11, RMGS20CR12, RMGS20EG19, RMGS20EG21, RMGS23CR05, RMGS23EG08).

References

Tianwen Chen and Raymond Chi-Wing Wong. Handling information loss of graph neural networks for session-based recommendation. In SIGKDD , pages 1172-1180, 2020.

Yen-Liang Chen, Mi-Hao Kuo, Shin-yi Wu, and Kwei Tang. Discovering recency, frequency, and monetary (RFM) sequential patterns from customers' purchasing data. Electronic Commerce Research and Applications , 8(5):241-251, 2009.

Luigi P. Cordella, Pasquale Foggia, Carlo Sansone, and Mario Vento. A (sub)graph isomorphism algorithm for matching large graphs. TPAMI , 26(10):1367-1372, 2004.

Zeyu Cui, Jianxin Ma, Chang Zhou, Jingren Zhou, and Hongxia Yang. M6-rec: Generative pretrained language models are open-ended recommender systems. arXiv preprint arXiv:2205.08084 , 2022.

Zihang Dai, Zhilin Yang, Yiming Yang, Jaime G. Carbonell, Quoc Viet Le, and Ruslan Salakhutdinov. Transformer-xl: Attentive language models beyond a fixed-length context. In ACL , pages 2978-2988, 2019.

Shijie Geng, Shuchang Liu, Zuohui Fu, Yingqiang Ge, and Yongfeng Zhang. Recommendation as language processing (RLP): A unified pretrain, personalized prompt & predict paradigm (P5). In RecSys , pages 299-315, 2022.

Jiayan Guo, Yaming Yang, Xiangchen Song, Yuan Zhang, Yujing Wang, Jing Bai, and Yan Zhang. Learning multi-granularity consecutive user intent unit for session-based recommendation. In WSDM , pages 343-352, 2022.

Balázs Hidasi, Alexandros Karatzoglou, Linas Baltrunas, and Domonkos Tikk. Session-based recommendations with recurrent neural networks. In ICLR , 2016.

Yupeng Hou, Binbin Hu, Zhiqiang Zhang, and Wayne Xin Zhao. CORE: simple and effective session-based recommendation within consistent representation space. In SIGIR , pages 17961801. ACM, 2022.

Ziniu Hu, Yuxiao Dong, Kuansan Wang, and Yizhou Sun. Heterogeneous graph transformer. In WWW , pages 2704-2710, 2020.

Jie Huang, Yifan Gao, Zheng Li, Jingfeng Yang, Yangqiu Song, Chao Zhang, Zining Zhu, Haoming Jiang, Kevin Chen-Chuan Chang, and Bing Yin. Ccgen: Explainable complementary concept generation in e-commerce. arXiv preprint arXiv:2305.11480 , 2023.

Haoming Jiang, Tianyu Cao, Zheng Li, Chen Luo, Xianfeng Tang, Qingyu Yin, Danqing Zhang, Rahul Goutam, and Bing Yin. Short text pre-training with extended token classification for e-commerce query understanding. arXiv preprint arXiv:2210.03915 , 2022.

Wei Jin, Haitao Mao, Zheng Li, Haoming Jiang, Chen Luo, Hongzhi Wen, Haoyu Han, Hanqing Lu, Zhengyang Wang, Ruirui Li, Zhen Li, Monica Xiao Cheng, Rahul Goutam, Haiyang Zhang, Karthik Subbian, Suhang Wang, Yizhou Sun, Jiliang Tang, Bing Yin, and Xianfeng Tang. Amazon-m2: A multilingual multi-locale shopping session dataset for recommendation and text generation. In NeurIPS , 2023.

Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR , 2015.

Nils M. Kriege, Fredrik D. Johansson, and Christopher Morris. A survey on graph kernels. Appl. Netw. Sci. , 5(1):6, 2020.

Jing Li, Pengjie Ren, Zhumin Chen, Zhaochun Ren, Tao Lian, and Jun Ma. Neural attentive session-based recommendation. In CIKM , pages 1419-1428, 2017.

Zihan Lin, Changxin Tian, Yupeng Hou, and Wayne Xin Zhao. Improving graph collaborative filtering with neighborhood-enriched contrastive learning. In WWW , pages 2320-2329, 2022.

Fan Liu, Zhiyong Cheng, Lei Zhu, Zan Gao, and Liqiang Nie. Interest-aware message-passing GCN for recommendation. In WWW , pages 1296-1305, 2021.

Qiao Liu, Yifu Zeng, Refuoe Mokhosi, and Haibin Zhang. STAMP: short-term attention/memory priority model for session-based recommendation. In SIGKDD , pages 1831-1839, 2018.

Xin Liu and Yangqiu Song. Graph convolutional networks with dual message passing for subgraph isomorphism counting and matching. In AAAI , pages 7594-7602, 2022.

Xin Liu, Haojie Pan, Mutian He, Yangqiu Song, Xin Jiang, and Lifeng Shang. Neural subgraph isomorphism counting. In SIGKDD , pages 1959-1969. ACM, 2020.

Xin Liu, Jiayang Cheng, Yangqiu Song, and Xin Jiang. Boosting graph structure learning with dummy nodes. In ICML , volume 162, pages 13704-13716. PMLR, 2022.

Chen Luo, William Headden, Neela Avudaiappan, Haoming Jiang, Tianyu Cao, Qingyu Yin, Yifan Gao, Zheng Li, Rahul Goutam, Haiyang Zhang, and Bing Yin. Query attribute recommendation at amazon search. In RecSys , page 506-508, 2022.

Ruihong Qiu, Jingjing Li, Zi Huang, and Hongzhi Yin. Rethinking the item order in sessionbased recommendation with graph neural networks. In CIKM , pages 579-588, 2019.

Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research , 21:140:1-140:67, 2020.

Steffen Rendle, Christoph Freudenthaler, Zeno Gantner, and Lars Schmidt-Thieme. BPR: bayesian personalized ranking from implicit feedback. In UAI , pages 452-461, 2009.

Steffen Rendle, Christoph Freudenthaler, and Lars Schmidt-Thieme. Factorizing personalized markov chains for next-basket recommendation. In WWW , pages 811-820, 2010.

Walid Shalaby, Sejoon Oh, Amir Afsharinejad, Srijan Kumar, and Xiquan Cui. M2trec: Metadata-aware multi-task transformer for large-scale and cold-start free session-based recommendations. In RecSys , pages 573-578, 2022.

Nino Shervashidze, S. V. N. Vishwanathan, Tobias Petri, Kurt Mehlhorn, and Karsten M. Borgwardt. Efficient graphlet kernels for large graph comparison. In AISTATS , volume 5, pages 488-495, 2009.

Zhongwei Wan, Xin Liu, Benyou Wang, Jiezhong Qiu, Boyu Li, Ting Guo, Guangyong Chen, and Yang Wang. Spatio-temporal contrastive learning enhanced gnns for session-based recommendation. Transactions on Information Systems , 2023.

Meirui Wang, Pengjie Ren, Lei Mei, Zhumin Chen, Jun Ma, and Maarten de Rijke. A collaborative session-based recommendation approach with parallel memory modules. In SIGIR , pages 345-354, 2019.

Ruijie Wang, Zheng Li, Dachun Sun, Shengzhong Liu, Jinning Li, Bing Yin, and Tarek Abdelzaher. Learning to sample and aggregate: Few-shot reasoning over temporal knowledge graphs. In NeurIPS , pages 16863-16876, 2022.

Shoujin Wang, Qi Zhang, Liang Hu, Xiuzhen Zhang, Yan Wang, and Charu Aggarwal. Sequential/session-based recommendations: Challenges, approaches, applications and opportunities. In SIGIR , pages 3425-3428, 2022.

Ziyang Wang, Wei Wei, Gao Cong, Xiao-Li Li, Xianling Mao, and Minghui Qiu. Global context enhanced graph neural networks for session-based recommendation. In SIGIR , pages 169-178, 2020.

Shu Wu, Yuyuan Tang, Yanqiao Zhu, Liang Wang, Xing Xie, and Tieniu Tan. Session-based recommendation with graph neural networks. In AAAI , pages 346-353, 2019.

Yuhuai Wu, Markus N. Rabe, DeLesley Hutchins, and Christian Szegedy. Memorizing transformers. In ICLR , 2022.

Xin Xia, Hongzhi Yin, Junliang Yu, Qinyong Wang, Lizhen Cui, and Xiangliang Zhang. Selfsupervised hypergraph convolutional networks for session-based recommendation. In AAAI , pages 4503-4511, 2021.

Chengfeng Xu, Pengpeng Zhao, Yanchi Liu, Victor S. Sheng, Jiajie Xu, Fuzhen Zhuang, Junhua Fang, and Xiaofang Zhou. Graph contextualized self-attention network for session-based recommendation. In IJCAI , pages 3940-3946, 2019.

Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. How powerful are graph neural networks? In ICLR , 2019.

Xifeng Yan and Jiawei Han. gspan: Graph-based substructure pattern mining. In ICDM , pages 721-724, 2002.

Junhan Yang, Zheng Liu, Shitao Xiao, Chaozhuo Li, Defu Lian, Sanjay Agrawal, Amit Singh, Guangzhong Sun, and Xing Xie. Graphformers: Gnn-nested transformers for representation learning on textual graph. In NeurIPS , pages 28798-28810, 2021.

Ghim-Eng Yap, Xiaoli Li, and Philip S. Yu. Effective next-items recommendation via personalized sequential pattern mining. In DASFAA , volume 7239, pages 48-64, 2012.

Changlong Yu, Weiqi Wang, Xin Liu, Jiaxin Bai, Yangqiu Song, Zheng Li, Yifan Gao, Tianyu Cao, and Bing Yin. FolkScope: Intention knowledge graph construction for E-commerce commonsense discovery. In Finding of ACL , pages 1173-1191, 2023.

Danqing Zhang, Zheng Li, Tianyu Cao, Chen Luo, Tony Wu, Hanqing Lu, Yiwei Song, Bing Yin, Tuo Zhao, and Qiang Yang. Queaco: Borrowing treasures from weakly-labeled behavior data for query attribute value extraction. In CIKM , page 4362-4372, 2021.

Peiyan Zhang, Jiayan Guo, Chaozhuo Li, Yueqi Xie, Jaeboum Kim, Yan Zhang, Xing Xie, Haohan Wang, and Sunghun Kim. Efficiently leveraging multi-level user intent for session-based recommendation via atten-mixer network. In WSDM , 2023.

Lingxiao Zhao and Leman Akoglu. Pairnorm: Tackling oversmoothing in gnns. In ICLR , 2020.

Kun Zhou, Hui Wang, Wayne Xin Zhao, Yutao Zhu, Sirui Wang, Fuzheng Zhang, Zhongyuan Wang, and Ji-Rong Wen. S3-rec: Self-supervised learning for sequential recommendation with mutual information maximization. In CIKM , pages 1893-1902, 2020.

𝑆

:

𝑣1

𝑣2

𝑣3

𝑣2

𝑣4

𝑣5

𝑣1

𝑣2

𝑣3

𝑣4

𝑣5

𝑣1

𝑣2

𝑣3

𝑣4

𝑣5

𝑣1

𝑣2

𝑣3

𝑣4

𝑣5

𝑣1,2

𝑣2,3

𝑣3,2

𝑣2,4

𝑣4,5

𝑣1

𝑣2

𝑣3

𝑣4

𝑣5

…

𝑣1

𝑣2

𝑣3

𝑣4

𝑣5

Session Sequence

Local Session Graph      Global Transition Graph     Shortcut Graph       Heterogeneous Graphs            Attribute Patterns

A Limitations

There are three limitations to our current proposed method and evaluation. First, our method separately processes and retrieves patterns for each attribute type. We do not merge all attributes in a candidate pool because we aim for our method to easily generalize to real recommendation systems with hundreds of attribute types and category hierarchies. The current implementation supports adding a new attribute type to a model as long as its embeddings align with the embeddings of other attributes. Second, we conducted experiments based on "clean" session data. Most E-commerce platforms do not have truly clean data on product attributes, so attribute data, in general, is very sparse and full of invalid values. We performed human-centric attribute regularization to drop products without valid attribute values, which may create a gap compared to a real industrial system. Third, the evaluation does not consider the same products with different identifiers. Therefore, evaluating results (especially MRR) cannot accurately reflect the performance. To better reflect the real performance with error tolerance, a larger K is suggested. The current comparison is still fair for all algorithms, and we address this synonym problem in attribute estimation in § 5.4, where we merge attribute values based on semantics and syntax.

B Transition Graph Density

The graph structure is crucial for neural networks to capture explicit transitions and implicit connections. A local session records the history of a user's clicks or purchases, which is usually sparse. In contrast, the global collaborative graph could be extremely dense because each pair of items may have a potential connection. From this perspective, the density indicates the explicit information provided from session data. On the other hand, different graph topologies and densities also present different focuses and challenges. The sparse local transition graph emphasizes current intents, while the global collaboration indicates broader interests and revenues. Graph neural networks excel at capturing local features, but a large number of neighbors can overshadow important connections with less significant ones. Considering that previous methods have focused on different granularities individually, we summarize them in Figure 7 and compare them in terms of optimization interpretations 3 .

Local session graphs correspond to item transitions within a session, where edges are created between two consecutively clicked/purchased items. The density is usually sparse (slightly greater than 1.0), allowing exploration and global collaboration to be learned through model parameters instead of explicit connections. Therefore, generalizing to unseen click patterns becomes challenging.

Global transition graphs record all collaborations. In a real industrial system, the density is usually beyond one hundred or even one thousand. Ideally, any session can benefit from this global collaborative information, including multi-hop connections. However, optimizing graph neural

networks to learn such topologies (due to oversmoothing) and building a large model for real-time inference (due to latency and streaming processing) pose challenges.

Shortcut graphs aim to avoid constructing global graphs and make session learning more efficient. They were proposed by LESSR [1] to address information loss in graph convolutions. Specifically, they allow latent items to be aware of all previous clicks, resembling shortcuts for multi-hop neighbors in the directed local session graph. However, they lack the extensive exploration capabilities of the global transition graph and suffer from oversmoothing issues due to dense connections.

Heterogeneous graphs strike a balance between shortcuts and local adjacency. Nodes with different numbers of items are categorized into different groups, and transition edges capture varying levels of spatial continuity. From a high-level perspective, this graph is sparser than the local session graph, resulting in faster convergence for optimization. However, the propagation of high-order information introduces additional processing costs and the risk of overfitting.

Patterns , especially attribute patterns, should be the most efficient features for recommendations in a large candidate item pool. Each pattern can be considered a higher-grained heterogeneous graph. However, pattern filtering can significantly eliminate noise influence, not to mention the benefits gained from offline indexing. Besides, the partial match of patterns can provide the intent information from other sessions, making the learning and prediction more reliable and steady.

C Experimental Data

C.1 Public Benchmarks

We choose two public benchmarks for session-based recommendation evaluation: diginetica 4 is CIKM Cup 2016 that contains the browser logs and anonymized transactions; Tmall 5 comes from a competition in IJCAI-15 which collects anonymous users' shopping logs on the Tmall online website. We acquire attributes from the original data and drop items without attributes or with invalid values. Therefore, the performance of baselines may not be exactly same as the reported numbers in the original papers.

C.2 E-commerce Data Collection

We collect E-commerce data from our log systems in two months. We follow the same procedure to clean and process session data in beauty , books , and electronics domains 6 :

I We focus on successful purchases so that we only keep sessions ending with 'purchase' actions.

II To make sure previous clicks can reflect the purchase intent, we drop actions 20 minutes ago.

III We filter out items with missing attributes (i.e., books without publishers, authors, or genre, and electronics without colors and brands).

IV We adopt the 20-core setting to finalize the item sets, in which items appear on at least 20 different days.

V Only sessions whose length is no greater than 50 are preserved.

VI We retrieve item attributes in our attribute databases.

VII For GNN models that requires the global transition graph from training data, we maintain 12 neighbors based on the co-occurrence, which is consistent with GCE-GNN [34].

C.3 Data Split

We follow previous settings that split training/validation/testing data based on timestamps. For diginetica , we gather the last 8-14 days as validation, the last 7 days as testing, and remaining as training. For Tmall , we use the last 101-200 seconds as validation, the last 100 seconds as testing, and remaining as training. For our industrial E-commerce data (i.e., Beauty , Books , Electronics ), we select the last 6-10 days as validation, the last 5 days as testing, and remaining as training.

C.4 Data Statistics

Table 6 summarizes the statistics of the experimental datasets based on timestamps. The density is calculated based on undirected graphs, which would be doubled during graph convolution in practice. Local density , as used in SR-GNN and GC-SAN, corresponds to the average density of local session graphs in E-commerce sessions. On the other hand, global density , as used in GCE-GNN, refers to the density of the global collaborative graph obtained by connecting all adjacent items appearing in all sessions. Shortcut density , as used in LESSR, is the density resulting from connecting all items in a single session as a complete graph. Heterogeneous density , as used in MSGIFSR, refers to the average density of the heterogeneous graphs obtained by regarding the consecutive adjacent two nodes as a fine-grained intent unit. Lastly, pattern density , as used in FAPAT, is the density of the acquired frequent and compact patterns. From Table 6, it is evident that leveraging patterns is the most effective way of characterizing user intents because other graph topologies vary with data sources and scales, making it difficult to generalize and provide stable performance. Besides, patterns can be preprocessed as indicies to aid recommendations, making them more practical in industrial scenarios. Moreover, it is easy to update attribute patterns dynamically, whereas other graph structures are more closely coupled with input sessions and are more sensitive to tiny variations.

D Baselines

We compare our method with the following baselines: Sequence-based methods

FPMC [27] learns the representation of session via Markov-chain based methods.

GRU4Rec [8] is the first RNN-based approach that simulates the Markov Decision Process (MDP) but has a better generalization.

NARM [16] is a attention-based RNN model to learn session embeddings.

STAMP [19] adopts attention mechanism between the last item to previous histories to represent users' short-term interests.

CSRM [31] proposes to engage an inner memory encoder and external memory network to capture correlations between neighborhood sessions to enrich the collaborative representations.

S3-Rec [47] is the first pretrained SBR model that predicts items, attributes, and segments during the pretraining stage.

M2TRec [28] is a metadata-aware multi-task Transformer model. In the original paper, the authors ignore item embeddings. For a fair comparison, we also regard the item ids as one of metadata.

Graph-based methods

SR-GNN [35] is the first GNN-based model for the SBR task, which transforms the session data into a direct unweighted graph and learns the representation of the item-transitions graph.

GC-SAN [38] uses gated GNNs to extract local context information and then self-attention to obtain the global representation.

S2-DHCN [37] transforms the session data into hyper-graphs and line-graphs and encodes them via GCNs to enhance the session representations.

GCE-GNN [34] aggregates two levels of item embeddings from session graphs and global graphs with soft attention.

LESSR [1] preserves the edge order and constructs shortcuts to encode sessions for GNNs.

MSGIFSR [7] captures the user intents from multiple granularities to relieve the computational burden of long-dependency. In experiments, we search the best model from the level-1, level-2, and level-3 consecutive intent units.

E Experimental Settings

We fix all embeddings and hidden dimensions as 100, and the batch size is searched among {100, 200, 500} for all methods. We also choose the number of layers/iterations (if applicable) from the validation performance (e.g., MRR@10). A learning scheduler with 10% linear warmup and 90% decay is associated with the Adam optimizer [14]. The initial learning rate is set as 1e-3, and the regularization weight is tuned among {1e-4, 1e-5, 1e-6}. We seek the dropout probability between two modules from {0.0, 0.2, 0.4}, but fix the attention dropout rate as 0.2. The number of attention heads is empirically set as 4. We follow the setting of GCE-GNN that the maximum one-hop neighbor number in GAT is 12. In the interest of fairness, we also set the maximum selected pattern number as 12. Hyper-parameter tuning is time costly on our industrial data so that we use the best combinations obtained from one day transactions. We implement our methods and run experiments with Python and PyTorch over 8 x A100 NVIDIA GPUs.

F Experimental Results

Due to the space limit, we only report some results in the main content. More comprehensive comparisons are shown in Tables 7-11, where standard deviations are enclosed in brackets. The best and second-best results are respectively highlighted in bold and underlined. Methods that use attributes are marked with ‡, and * indicates the p -value < 0.0001 in t-test.

Peer reviews:
comments: SUMMARY:
The paper presents a transformer-based method, namely Frequent Attribute Pattern Augmented Transformer (FAPAT), that considers attribute patterns as supplementary information. More specifically, FAPAT builds attribute transition graphs, mine frequent attribute patterns, and match attribute patterns to better captures user intents. Experiments conducted on two public benchmarks as well as three industrial datasets show that the proposed method deliver noticeable performance boosts (4.5%) averaged over all experiments. The code and datasets are promised to be released after acceptance.

STRENGTHS:
1. Introduction section is written very well to present the problem and the high-level picture of the proposed method.
2. Table 1 in Related Work section is very straight-forward and efficient in comparing a large number of existing methods and demonstrating how they differ from the proposed method. And the categorical items (temporal info, history attention, etc.) are reasonably defined.
3. Methodology section is clear and easy to follow.
4. Experiments are thorough with a good number of baselines.

WEAKNESSES:
1. Some of the thought process is not explained, for example, in section 4.1.1, what are the thoughts process behind adopting gSpan and how were those twenty types of patterns chosen?

QUESTIONS:
1. In introduction, can you elaborate more on “We argue that the current use of item-side metadata provides little assistance in SBR models as user intent may change over time.“?
2. Typo in line 131 - "whi".
3. I am not sure what is the purpose of Section 3 - Background and Motivations, especially 3.2 and 3.3, they read like they should belong to Related Work.

LIMITATIONS:
Yes, the authors discuss limitations in Appendix A.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2023-07-03

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes FAPAT, which augments session-based recommendation models using item attributes. Specifically, FAPAT constructs attribute graphs and models user intent using pattern mining and an improved graph attention mechanism.\n\nSTRENGTHS:\n1. The paper studies an important application task, i.e., session-based recommendation.\n2. Experiments are conducted on 2 public datasets and 3 large-scale industrial datasets.\n\nWEAKNESSES:\n1. Over-complicated method. Session-based recommendation methods that incorporate session graphs have been criticized for being overly complex. The significant increase in algorithm complexity generally yields limited improvements in performance. The method proposed in this paper, involving modules such as motif extraction and graph attention mechanisms, adds further complexity to this paradigm. However, based on the experiments in Table 2, even with such complexity, this method only outperforms the concise baseline GRU4Rec by a limited margin on the classic benchmark dataset diginetica. The results raise doubts about the necessity of such a complex approach. It could be possible to achieve comparable results by introducing attributes using a much simpler method.\n2. Missing baselines. Two important baselines are missed. SASRec [1] is a highly popular baseline method for sequential/session-based recommendation, based on Transformer modules. FDSA [2] is a straightforward method based on SASRec that introduces item attributes. Both of these methods are concise and can be regarded as a basic version of the proposed model. These two methods should be included in experiments and compared against the proposed method.\n3. Code is not available during the reviewing phase, making reproduction of the reported results difficult.\n\n[1] Kang et al. Self-Attentive Sequential Recommendation. ICDM 2018.\n\n[2] Zhang et al. Feature-level Deeper Self-Attention Network for Sequential Recommendation. IJCAI 2019.\n\nQUESTIONS:\nPlease refer to \"Weakness\".\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a novel framework for session-based recommendations. The code idea of the method is to extract highly frequent attribute patterns from graphs to augment session sequence encoding. Specifically, it first leverages frequent graph pattern mining for attribute pattern retrieval. Then it applies GAT-based encoders for item representations and use these representations as memory to faciliate session-based encoding. Finally, it converts item-side graph representations into sequences and aggregates them with session sequences for transformer-based model training. Empirical results compared with sequence-based and graph-based baselines show the advantages of the method.\n\nSTRENGTHS:\nThe paper is well-written wrt the high-level topic and what the high-level intuitions are.  \nGood motivation of mining attribute features for sequence encoding enhancement, which is a significant problem to the community.  \nExperiment is extensive and convincing.  Experimental results on two public datasets and one industrial dataset seem promising compared with sequence-based and graph-based baselines.  \nComprehensive analysis in the ablation studies and the appendix.\n\nWEAKNESSES:\nSome clarifications can be made more clear. For example, it mentions using gSpan to mine patterns and keep patterns in the twenty types as it shows. However, it is not clear why keeping patterns in these twenty types, like it is a common practice or it can faciliate training.   \nMore powerful sequence-based approaches can be discussed.  As far as I know, P5 [1] and M6-rec [2] are more powerful sequence-based baselines. It would be better to discuss these methods. \n\n[1] Geng, Shijie, et al. \"Recommendation as language processing (rlp): A unified pretrain, personalized prompt & predict paradigm (p5).\" RecSys 2022.  \n[2] Cui et al. \"M6-Rec: Generative Pretrained Language Models are Open-Ended Recommender Systems.\" Arxiv 2022.\n\nQUESTIONS:\nBeing not familiar with graph-based transformers, I mainly have concerns about the design choice mentioned in its method.  \n- What is the reason of choosing the GAT-based encoder rather than GCN and GraphSAGE, for learning attribute graph representations?  \n- What is the insights and design intuition of using attribute representations as memory to augment sequence encoding? \n- Why using sequences as the final representations rather than aggregate information using graph representations?\n\nLIMITATIONS:\nDesign intuition and technical details can be explained more clearly.  \nIt would be better to provide discussion and comparision of more state-of-the-art baselines.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies session-based recommendation in E-commerce. The authors propose to enhance user intent identification by constructing attribute transition graphs and using frequent attribute patterns as memory to augment session representations. The study employs frequent graph pattern mining algorithms to find consequential graphlets and uses these attribute patterns as accessible memory to augment session sequence encoding. It leverages multi-head graph attention to learn patterns and local session graph representations in the aligned space. The experiments are conducted on two public benchmark datasets and three large-scale industrial datasets, demonstrating notable improvement across various evaluation metrics.\n\nSTRENGTHS:\n1. Session-based recommendation is an important research topic in data mining.\n2. The proposed method achieves better results.\n3. The authors provide sufficient comparisons with baseline methods.\n\nWEAKNESSES:\n1. The motivations for different model components are not clearly stated nor discussed. The model components are directly introduced, without detailed discussions on their intuitions and big pictures. The experimental analysis also does not well elaborate on the reasons and technical findings.\n\n2. The proposed method is complicated and its computational cost is high. Session-based recommendation has strict latency restrictions and it is unclear whether the proposed method meets the requirements. Further efficiency analysis is highly needed, which is important for supporting the main motivation.\n\n3. The evaluation is purely offline and no online experimental result is reported. It is not clear whether the proposed method can be deployed online with acceptable effort.\n\n4. The proposed framework is a combination of mature techniques. Thus, the technical contributions are not quite salient. More empirical analysis could be made.\n\n5. It would be helpful to compare the results on long-tailed/cold-start attributes and items, since the proposed method seems to be suitable for handling these cases.\n\nQUESTIONS:\nHow about the computational cost of this work? Is there any analysis of the real training and inference time?\n\nLIMITATIONS:\nThe authors mainly discussed the limitations at the technique level, but are not aware of the potential negative societal impact. Some discussions on recommendation fairness and diversity can be added.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

comments: SUMMARY:
The paper proposes a framework that effectively utilizes attribute graph patterns to enhance anonymous sequence encoding for session-based recommendations. Given the lack of personal information in session-based recommendation scenarios, making accurate item suggestions within the session is crucial. The authors performed an extensive experimental evaluation, and it seems there is an improvement. I would appreciate more info on methodology used for experimentation.

Minor comments:

The citations are clickable, which is convenient for further reading.
It would be beneficial to add references to Table 1 for better context.
Providing more explanation in lines 70-75 would enhance clarity and understanding.
While I'm not an expert in the area, it seems that the related work section could benefit from including more recent papers. Currently, there is only one paper from 2023.
Fig 5-6 are clear and easy to read, which is commendable.

STRENGTHS:
1. Extensive experimentation

WEAKNESSES:
1. Not clearly demonstrated novelty of the method 
2. The paper is a bit hard to follow
3. Some examples of the data can be useful in the paper.

QUESTIONS:
Can you please list a number of the current practical applications that would benefit from the proposed method? 

Could you provide more information about the costs associated with the model? Additionally, I'm interested in knowing about the latency, or response time, of the model.

LIMITATIONS:
Yes
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-11