SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Xin Liu $^{1*}$  Zheng Li $^{2}$  Yifan Gao $^{2}$  Jingfeng Yang $^{2}$  Tianyu Cao $^{2}$  Zhengyang Wang $^{2}$  Bing Yin $^{2}$  Yangqiu Song $^{1\dagger}$ $^{1}$ Department of Computer Science and Engineering, HKUST  $^{2}$ Amazon.com Inc {xliucr, yqsong}@cse.ust.hk {amzzhe, yifangao, jingfe, caoty, zhengywa, alexbyin}@amazon.com
The goal of session-based recommendation in E-commerce is to predict the next item that an anonymous user will purchase based on the browsing and purchase history. However, constructing global or local transition graphs to supplement session data can lead to noisy correlations and user intent vanishing. In this work, we propose the Frequent Attribute Pattern Augmented Transformer (FAPAT) that characterizes user intents by building attribute transition graphs and matching attribute patterns. Specifically, the frequent and compact attribute patterns are served as memory to augment session representations, followed by a gate and a transformer block to fuse the whole session information. Through extensive experiments on two public benchmarks and 100 million industrial data in three domains, we demonstrate that FAPAT consistently outperforms state-of-the-art methods by an average of  $4.5\%$  across various evaluation metrics (Hits, NDCG, MRR). Besides evaluating the next-item prediction, we estimate the models' capabilities to capture user intents via predicting items' attributes and period-item recommendations.
With the explosive demand for E-commerce services [44, 12, 23, 11, 43], numerous user behaviors are emerging. Understanding these historical action records is critical in comprehending users' interests and intent evolution, particularly in a cold-start regime that lacks sufficient context. This has spurred research on session-based recommendations (SBR) [8, 35, 37, 13] that capture user-side dynamics from a short-time period (namely a session) using temporally historical information. Numerous SBR algorithms have been proposed, ranging from sequence-based methods [27, 8, 16, 19, 31, 28, 9] to graph-based methods [35, 38, 34, 37, 17, 30] for learning dynamic user characterization. However, both lines of methods have their limitations. Specifically, sequence-based methods treat each user behaviors in a session as an action sequence and model the local dependencies inside. This can only capture users' preference evolution via chronological order while failing to identify the complex non-adjacent item correlation, especially when the session length is insufficient to support temporal prediction [33]. To address this issue, graph-based methods adopt a higher perspective by introducing a global item transition graph, which aggregates local session graphs constructed from historical session sequences. Thus, a newly-emerging short session sequence can benefit from the global topology (e.g., global co-occurrences) and representations (e.g., item semantics) [34]. Unfortunately, existing session graph construction ignores temporal signals. Figure 1 shows that two different sessions result in the same session graph, leading to vanishing of user intent variation during sequence-to-graph conversion. And such global graphs are fragile due to noise from random clicks.
*Work was done during Xin's internship at Amazon. Corresponding author: Xin Liu and Zheng Li.
†Prof. Yangqiu Song is a visiting academic scholar at Amazon.
Figure 1: Local session graph construction.
Figure 2: A session graph enriched by multiplex attribute graphs.
Besides the global item transition graph, there are other possible solutions for item correlations. One feasible solution involves item-side knowledge. Items can be connected through shared attributes, such as being manufactured by the same company [28]. We argue that the current use of item-side metadata provides little assistance in SBR models as user intent may change over time. However, such meta-data are still useful from the view of graphs. As illustrated in Figure 2, session attributes can be organized into attribute graphs and anchored to the local session graph to create multiplexes. Such multiplexes provide several intent clues, in addition to item-side correlations. For example, the color pattern silver  $\leftrightarrow$  silver  $\leftrightarrow$  blue  $\leftrightarrow$  blue reveals the user's color intent, while the brand pattern Apple  $\leftrightarrow$  Apple  $\leftrightarrow$  Samsung implies a potential change in intent. Different sessions can benefit from shared attribute topologies and representations, which can ultimately entail implicit high-order correlations. However, the session graph is essentially a general conditional random field, and optimization becomes intractable due to the large candidate size. Graph neural networks (GNNs) also face challenges due to the possibility of over-smoothing and data noise [46, 18, 45, 32].
To alleviate the aforementioned issues, we propose a novel framework called Frequent Attribute Pattern Augmented Transformer (FAPAT) that considers highly frequent attribute patterns as supplementary instead of directly learning multiplex graphs. In traditional graph learning, graphlets (such as triangles, triangular pyramids, etc.) have been proven to be useful features in graph classification and representation learning [29, 15]. Therefore, we employ frequent graph pattern mining algorithms to find consequential graphlets and view them as compact hyper-edges (e.g., cell phone, tablet, notebook in Figure 2). We then use these attribute patterns as accessible memory to augment session sequence encoding. Before encoding patterns, we use Jaccard similarities to rank and retrieve the most highly correlated patterns, which significantly reduces the graph density and computational cost. It has been shown that GNNs can estimate the isomorphism and frequency of substructures [39, 21, 20]. Thus, we leverage multi-head graph attention to learn pattern and local session graph representations in the aligned space [10, 22]. To incorporate temporal signals and capture user intents, we distribute graph representations back to session sequences and use external pattern memory to augment sequence representations via memory attention with relative position bias. Finally, the sequence is fully fused by a transformer block. In other words, graph information is used to aggregate attribute patterns, while temporal actions are used to encode items.
To validate the effectiveness of FAPAT, we conduct extensive experiments on two public benchmark datasets and three real-world large-scale industrial datasets with around 100 million clicks, and experimental results demonstrate significant improvement with an average boost of  $4.5\%$  across various evaluation metrics (Hits, NDCG, MRR). Compared with baselines, the attribute pattern density can significantly relieve over-smoothing. Besides, we also extend evaluation to attribute estimations and sequential recommendations to measure the model capability to capture user intents. Code and data are available at https://github.com/HKUST-KnowComp/FAPAT.
Neural Methods for Session-based Recommendation Systems. Table 1 presents a summary of the distinctions between current neural techniques and our novel FAPAT. The concept of Temporal Information implies Markov decision processes with previous histories. The technique of History Attention employs attention for learning long-distance sequences. The approach of Local Session
Table 1: Comparison with existing popular methods.
Methods	Temporal Information	History Attention	Local Session Topology	Global Item Correlation	Attribute Association
FPMC	✓	X	X	✓	X
GRU4Rec	✓	X	X	X	X
NARM	✓	✓	X	X	X
STAMP	✓	✓	X	X	X
CSRM	✓	✓	X	✓	X
S3-Rec	✓	✓	X	✓	✓
M2TRec	✓	✓	X	✓	✓
SR-GNN	X	X	✓	X	X
GC-SAN	X	✓	✓	X	X
S2-DHCN	X	X	✓	✓	X
GCE-GNN	X	X	✓	✓	X
LESSR	✓	X	✓	X	X
MSGIFSR	✓	✓	✓	X	X
FAPAT	✓	✓	✓	✓	✓
Topology involves modeling session sequences from the view of session graphs. Lastly, Global Item Correlation and Attribute Association place emphasis on capturing item-side and attribute knowledge.
Sequence-based Models. FPMC [27] uses first-order Markov chain and matrix factorization to identify sequential patterns of long-term dependencies. However, the Markov-based method usually has difficulty exploring complicated temporal patterns beyond first-order relationships. Recently, neural networks have shown power in exploiting sequential data in SBR tasks, such as GRU4Rec [8]. NARM [16] extends GRUs with attention to emphasize the user's primary purchase purpose. Similarly, STAMP [19] uses an attention-based memory network to capture the user's current interest. These attention-based models separately deal with the user's last behaviors and the whole session history to detect the general and latest interests. But they mainly focus on the user's preference from a temporal view but ignore the item correlations. Pre-training techniques [47] and multi-task learning [28] also demonstrate effectiveness in injecting item metadata to embeddings and predicting item attributes. Besides, some recent sequence-based approaches leverage generative pretrained language models to provide explicit explanations for recommendation systems [4, 6].
Graph-based Models. Graph neural networks (GNNs) have recently been explored in SBRs due to the substantial implications behind natural transition topologies. SR-GNN [35] adopts a gate GNN to obtain item embeddings over the local session graph and predict the next item with weighted sum pooling, showing impressive results on benchmark data. Some advanced variants have further boosted performance, such as GC-SAN [38] with self-attention mechanism and FGNN [24] with weighted attention graph layers. To acquire further collaborative information, S2-DHCN [31] constructs line graphs to capture correlations among neighbor sessions, and GCE-GNN [34] directly applies a graph convolution over the global transitions to aggregate more relevant items for local sessions. However, GNN-based methods still face challenges in capturing temporal signals, filtering noise, and leveraging implicit high-order collaborative information. Other methods, such as LESSR [1] and MSGIFSR [7], have also shown significant improvement by building multigraphs and shortcut graphs for session representation learning and user intents from different granularities, respectively.
Pattern Mining for Recommendation Systems. Pattern mining is an important data mining technique with board applications. In recommendation systems, sequential pattern mining assists in analyzing customer purchase behaviors through frequent sequential patterns. Such mining focuses on item patterns with frequencies above a threshold in all sessions, which reduces the diversity of the recommended items [2]. Personalized sequential pattern mining [42] effectively learns user-specific sequence importance knowledge and improves the accuracy of recommendations for target users. It can be challenging to generalize to SBR systems when neural networks have already implicitly captured such behavior patterns. But attribute graph patterns still need to be explored in SBR.
3 Background and Motivations
Session-Based Recommendation (SBR) assumes that users' historical behaviors outside the current session are inaccessible, in contrast to general recommendations. For example, users do not log in due to user privacy and security reasons. SBR predicts the next item that an anonymous user is most
likely to click on or purchase based on historical behaviors within a short period. Despite the lack of personal profiles, this universal setting can better reflect the quality of item-side recommendations. Suppose that there are  $N$  unique items in the database, and each session is represented as a repeatable sequence of items  $S = [v_{1}, v_{2}, \ldots, v_{L}]$ ,  $v_{i} \in \mathcal{V}$  ( $1 \leq i \leq L$ ) represents the  $i$ -th behavioral item of the anonymous user within session  $S$ , where  $\mathcal{V}$  is the item set collected from overall sessions, and  $L$  is the length of the session. Given a session  $S$ , the goal is to recommend the top- $K$  items ( $1 \leq K \leq N$ ) that have the highest probabilities of being clicked by the anonymous user.
3.2 Session Graphs and Transition Graphs
Session sequence modeling is not always sufficient for SBR as it only reflects transitions from the user side. To account for item correlations, SR-GNN [35] converts session sequences to session graphs. Each session graph  $\mathcal{G}_S = (\mathcal{V}_S,\mathcal{E}_S)$  is a directed graph with node set  $\mathcal{V}_S\subseteq \mathcal{V}$ , consisting of unique items in the session, and edge set  $\mathcal{E}_S$ , recording adjacent relations between two items in session  $S$ . Edge weights can be normalized by indegrees or outdegrees to model transition probabilities. GCE-GNN [34] extends this graph modeling by merging all session graphs as a global transition graph, which aggregates more item correlations but faces over-smoothing and data noise [46, 18].
Instead of modeling global item transitions, we enrich session graphs with attributes and patterns. Assume there are  $M$  different kinds of attributes, and the  $m$ -th attribute type  $\mathcal{A}^{(m)}$  ( $1 \leq m \leq M$ ) consists of  $|\mathcal{A}^{(m)}|$  possible values. Thus, each item  $v \in \mathcal{V}_S$  has attribute list  $[a_v^{(1)}, a_v^{(2)}, \dots, a_v^{(M)}]$ , where  $a_v^{(m)} \in \mathcal{A}^{(m)}$  denotes the  $m$ -th attribute value of  $v$ . Each session sequence  $S$  corresponds to  $M$  attribute histories, with the  $m$ -th attribute sequence denoted as  $S^{(m)} = [a_{v_1}^{(m)}, a_{v_2}^{(m)}, \dots, a_{v_L}^{(m)}]$ . Using the sequence-to-graph transform, we convert  $S^{(m)}$  to  $\mathcal{G}_S^{(m)}$ , which is usually denser than the session graph  $\mathcal{G}_S$ . Finally, the  $M$  attribute sequences are separately transformed into  $M$  attribute session graphs in different property-specific channels, anchored in items  $\mathcal{V}_S$ . Finally, we add edges for item  $v_i$  and attribute  $a_{v_i}^{(m)}$  to construct a multiplex session graph, preserving attribute values and transitions. We represent the session graph with attributes as the multiplex  $\mathcal{G}_S^A$  (as Figure 2).
3.3 Frequent Pattern Mining
Frequent pattern mining aims to extract inductive clues from data to comprehend data distributions, which includes two main categories: sequential pattern mining and graph pattern mining. The former is concerned with sequence databases composed of ordered elements, while the latter statistics the important graph structures. For two sequences  $S' = [v'_1, v'_2, \dots, v'_L]$  and  $S = [v_1, v_2, \dots, v_L]$ , we refer to  $S'$  as the pattern of  $S$  if  $S'$  is a subsequence of  $S$ . Similarly, a graph  $\mathcal{G}_{S'} = (\mathcal{V}_{S'}, \mathcal{E}_{S'})$  is a subgraph of  $\mathcal{G}_S = (\mathcal{V}_S, \mathcal{E}_S)$  if  $\mathcal{V}_{S'} \subseteq \mathcal{V}_S$  and  $\mathcal{E}_{S'} \subseteq \mathcal{E}_S$ . Compared with sequence pattern mining, graph pattern mining is more general since it involves the structural topology and attribute information. For instance, as depicted in Figure 2, {cell phone, tablet, notebook} corresponds to a triangle. It may be challenging to discover the triangle from the temporal sequence, but it is a vital clue from the graph view. Thus, we stick on frequent graph patterns rather than sequence patterns.
We present the Frequent Attribute Pattern Augmented Transformer (FAPAT), a novel framework that captures user intents and item correlations. Our method is built upon session sequences and corresponding attribute graphs. Initially, we mine frequent attribute patterns from the attribute graphs to explore coarse-grained item correlations. These patterns are then used as memory to enhance the session encoder, which consists of graph-nested transformer layers. Figure 3 illustrates the overview.
4.1 Frequent Attribute Pattern Acquisition
In this subsection, we describe how to extract frequent patterns from training recommendation sessions. The aim of frequent pattern mining is to minimize the impact of random clicks in the global transition graph and avoid over-smoothing when learning multiplex attribute graphs. To achieve this, we design a mining-filtering paradigm to ensure representativeness.
Figure 3: An overview of FAPAT.
4.1.1 Graph Pattern Mining
Small graph patterns, also called motifs or graphlets, are valuable features in graph learning and property prediction [29, 15], exhibiting strong statistical correlations between graph structures and node semantics. In this study, we focus on similar graphlet structures but extend to SBR scenarios. We collect patterns consisting of no more than four nodes (representing different attribute values), and further restrict them to those containing either a circle or a triangle to significantly reduce the number of candidates. We adopt gSpan [40] to acquire undirected patterns from attribute session graphs and keep patterns belonging to one of twenty types shown in Figure 3.
4.1.2 Loose Pattern Filtering
However, complex patterns may contain smaller ones. For instance, the first pattern in Figure 3 is a subgraph of the second. While each subgraph has an equal or higher frequency than its supergraph, such loose patterns do not convey much information. To eliminate them, we employ VF2 [3] subgraph isomorphism algorithm to filter. If a pattern  $P'$  is a subgraph of another pattern  $P$ , then  $P'$  is excluded from the pattern candidates. Finally, we retain compact and frequent patterns for session encoding.
4.2 Intent-aware Sequence Encoding
To recommend items of high interest to users, we use a GAT-based encoder to learn pattern representations from the item side, which are then served as memory to augment session encoding.
4.2.1 Relevant Graph Pattern Retrieval
The input item sequence is converted to a multiplex session graph representing the transitional information of different item attributes, as depicted in Figure 1. To improve graph representations by utilizing frequent attribute subgraphs, we retrieve relevant patterns from those mined in §4.1. For the  $m$ -th attribute type  $\mathcal{A}^{(m)}$  ( $1 \leq m \leq M$ ), we denote an arbitrary subgraph mined from the previous step as  $\mathcal{G}_P^{(m)} = (\mathcal{V}_P^{(m)}, \mathcal{E}_P^{(m)})$ . Then we retrieve at most  $I$  subgraph patterns that have the most considerable Jaccard similarities to the transition graph  $\mathcal{G}_S^{(m)}$  as Eq. (1). This can be done within  $\mathcal{O}(|\mathcal{V}_S^{(m)}| \times |\mathcal{V}_P^{(m)}|)$ , but this is always linear because of  $|\mathcal{V}_P^{(m)}| \leq 4$ .
$$
\operatorname {J a c c a r d} \left(\mathcal {V} _ {S} ^ {(m)}, \mathcal {V} _ {P} ^ {(m)}\right) = \frac {\left| \mathcal {V} _ {S} ^ {(m)} \cap \mathcal {V} _ {P} ^ {(m)} \right|}{\left| \mathcal {V} _ {S} ^ {(m)} \right| + \left| \mathcal {V} _ {P} ^ {(m)} \right| - \left| \mathcal {V} _ {S} ^ {(m)} \cap \mathcal {V} _ {P} ^ {(m)} \right|}. \tag {1}
$$
4.2.2 Attribute Pattern Representation
After retrieving the relevant patterns for the corresponding multiplex session graph, we encode them using multi-head relational graph attention for further memory augmentation. For a pattern  $\mathcal{G}_P^{(m)}$  for the  $m$ -th attribute, we compute the attention weight for two arbitrary nodes  $a_i^{(m)}$  and  $a_j^{(m)}$  by:
$$
\alpha_ {i j} ^ {(m)} = \operatorname {s o f t m a x} \left(\frac {\text {L e a k y R e L U} \left(\boldsymbol {r} _ {i j} ^ {(m)} ^ {\top} \left(\boldsymbol {e} _ {a _ {j} ^ {(m)}} \circ \boldsymbol {e} _ {a _ {i} ^ {(m)}}\right)\right)}{\sum_ {a _ {k} ^ {(m)}} \in \mathcal {N} \left(a _ {i} ^ {(m)}\right) \text {L e a k y R e L U} \left(\boldsymbol {r} _ {i k} ^ {(m)} ^ {\top} \left(\boldsymbol {e} _ {a _ {k} ^ {(m)}} \circ \boldsymbol {e} _ {a _ {i} ^ {(m)}}\right)\right)}\right), \tag {2}
$$
where  $e_{a_i^{(m)}}$  denotes the embedding of  $a_i^{(m)}$ ,  $\circ$  indicates element-wise multiplication,  $\mathcal{N}(a_i^{(m)})$  represents the neighbors of  $a_i^{(m)}$  (including itself), and  $r_{ij}^{(m)}$  corresponds to the relation-specific vector. Then the representation of  $\overline{h}_{a_i^{(m)}}$  for node  $a_i^{(m)}$  in the pattern is aggregated by:
$$
\bar {\boldsymbol {h}} _ {a _ {i} ^ {(m)}} = \sum_ {a _ {j} ^ {(m)} \in \mathcal {N} \left(a _ {i} ^ {(m)}\right)} \alpha_ {i j} ^ {(m)} \boldsymbol {e} _ {a _ {i} ^ {(m)}}. \tag {3}
$$
After computing the representation of each node in pattern subgraph  $\mathcal{G}_P^{(m)}$ , a pooling layer (e.g., average pooling) over all node representations is to aggregate the pattern presentation for  $\mathcal{G}_P^{(m)}$ :
$$
\boldsymbol {p} ^ {(m)} = \operatorname {P o o l} \left(\left\{\overline {{\boldsymbol {h}}} _ {a _ {i} ^ {(m)}} \mid v _ {i} \in \mathcal {G} _ {P} ^ {(m)} \right\}\right) \tag {4}
$$
4.2.3 Attribute Memory Augmentation
Meanwhile, we also employ graph attention to obtain the native graph representations of the multiplex session graph  $\mathcal{G}_S^{(m)}$ . We apply this to each  $m$ -th attribute transition graph  $\mathcal{G}_S^{(m)}$ , computing node representations  $a_i^{(m)}$  using Eq. (2) and Eq. (3) and denote the resulting node representation as  $\overline{h}_{v_i}^{(m)}$ . To compute the aggregated attribute representation of each node  $v_i$  in the original session graph  $\mathcal{G}_S$ , we combine its different representations from all attribute transition graphs by:
$$
\bar {h} _ {v _ {i}} = \frac {1}{M} \sum_ {m} \bar {h} _ {v _ {i}} ^ {(m)} + e _ {v _ {i}}, \tag {5}
$$
where  $e_{v_i}$  is the representation of  $v_i$  by the item embedding lookup.
Suppose all retrieved attribute patterns associated with  $\mathcal{A}^{(m)}$  are  $\mathcal{P}^{(m)}$ , whose representations from Eq. (4) are  $\{\pmb{p}^{(m)}|p^{(m)}\in \mathcal{P}^{(m)}\}$ . To preserve temporal information when utilizing these patterns, we map graph representations  $\{\overline{\pmb{h}}_{v_i}^{(m)}|v_i\in \mathcal{V}_S\}$  to sequence representations  $\{\overline{\pmb{h}}_i^{(m)}|1\leq i\leq L\}$  with length  $L$ . Next, we concatenate all pattern representations  $\{\pmb{p}^{(m)}|p^{(m)}\in \mathcal{P}^{(m)}\}$ , all session representations  $\{\overline{\pmb{h}}_i^{(m)}|1\leq i\leq L\}$ , and two special embeddings  $e_{\mathrm{CLS}}$  and  $e_{\mathrm{MASK}}$  to separate the two parts and indicate the item to predict:
$$
\left[ \right.\left[ \right.\left. \right.\left\| \right. _ {p ^ {(m)} \in \mathcal {P} ^ {(m)}} \boldsymbol {p} ^ {(m)} \left. \right]\left. \right\| e _ {\mathrm {C L S}} \left. \right\|\left[ \right.\left. \right.\left\| \right. _ {1 \leq i \leq L} \overline {{\boldsymbol {h}}} _ {i} ^ {(m)} \left. \right]\left. \right\| e _ {\mathrm {M A S K}} \left. \right].
$$
Then, we augment sequence encoding with memory, taking inspiration from TransformerXL [5] and Memorizing Transformer [36]. As shown in Figure 4, we introduce T5 relative position bias [25] to distinct short and long-term histories. Furthermore, we utilize unidirectional attention to focus solely on the user's current intent instead of global intent, which may become increasingly noisy over time.
4.2.4 User Intent Aggregation
To aggregate memory-augmented representations from  $M$  different attributes and pattern sets, we employ gating mechanism:
$$
\hat {\boldsymbol {h}} _ {i} = \sum_ {m = 1} ^ {M} \beta_ {i} ^ {(m)} \hat {\boldsymbol {h}} _ {i} ^ {(m)} + \bar {\boldsymbol {h}} _ {i}, \text {s . t .} \beta_ {i} ^ {(m)} = \operatorname {s o f t m a x} \left(\boldsymbol {W} _ {\beta} ^ {(m)} \bar {\boldsymbol {h}} _ {i} ^ {(m)} + \boldsymbol {b} _ {\beta} ^ {(m)}\right), \tag {6}
$$
Figure 4: Schema of pattern augmented attention: relative position bias is added, and one item can only access memory and previous histories.