SlowGuess/NeurIPS_2023_OCR huggingface dataset:
where  $\pmb{W}_{\beta}^{(m)} \in \mathbb{R}^{d \times d}$  and  $\pmb{b}_{\beta}^{(m)} \in \mathbb{R}^d$  are trainable parameters ( $d$  is the dimension of hidden states),  $\hat{\pmb{h}}_i^{(m)}$  is the memory-augmented results, and  $\beta_i^{(m)}$  controls the importance of attribute patterns. While  $\hat{\pmb{h}}_i$  combines attribute information, it disregards long-range histories as  $\overline{\pmb{h}}_i^{(m)}$  solely accesses one-hop neighbors. Thus, we incorporate a transformer block to aggregate global session information:
$$
\boldsymbol {H} = \operatorname {T r a n s f o r m e r} \left(\left[ \hat {\boldsymbol {h}} _ {\mathrm {C L S}} \right\rVert \left[ \begin{array}{l} \| \\ 1 \leq i \leq L \end{array} \hat {\boldsymbol {h}} _ {i} \right] \left\| \hat {\boldsymbol {h}} _ {\mathrm {M A S K}} \right]\right), \tag {7}
$$
where  $\hat{h}_{\mathrm{CLS}}$  and  $\hat{h}_{\mathrm{MASK}}$  are directly taken from Eq. (6) since they do not belong to the original session but still observe the memory patterns and relative temporal signals.
4.3 Next-item Recommendation
Once the sequence representations are obtained, the next step is to predict the next item that may interest the user for clicking or purchasing. We adopt the approach used in previous work [35, 34, 37] to concatenate additional reversed positional embeddings as follows:
$$
\boldsymbol {z} _ {i} = \tanh  \left(\boldsymbol {W} _ {z} \left[ \boldsymbol {h} _ {i} \| \boldsymbol {t} _ {L - i + 1} \right] + \boldsymbol {b} _ {z}\right), 0 \leq i \leq L + 1, \tag {8}
$$
where  $h_i$  is the  $i$ -th item representation  $H[i]$  from Eq. (7), and  $W_z \in \mathbb{R}^{2d \times d}$  and  $b_z \in \mathbb{R}^d$  are trainable parameters. This concatenation of positional embeddings is intended to prioritize nearest intents over long-distance historical purposes. Figure 2 demonstrates this idea through two cases: the male user is more likely to be interested in Android cell phones since his last click was in that category, while the female user sticks to Apple products as she reviews iPhone and iPad once again.
We utilize the representation of the MASK to compute soft attention and then represent the session and the user's latest intent through a weighted average:
$$
\boldsymbol {u} = \sum_ {i = 1} ^ {L} \gamma_ {i} \boldsymbol {h} _ {i}, \text {s . t .} \gamma_ {i} = \boldsymbol {r} _ {\gamma} ^ {\top} \operatorname {s i g m o i d} \left(\boldsymbol {W} _ {\gamma} \boldsymbol {z} _ {i} + \boldsymbol {z} _ {\text {m a s k}} + \boldsymbol {b} _ {\gamma}\right), \tag {9}
$$
where  $W_{\gamma} \in \mathbb{R}^{d \times d}$  and  $b_{\gamma} \in \mathbb{R}^{d}$  are trainable parameters. We compute the prediction of the next item using a similarity-based approach rather than a linear layer. This approach is similar to the optimization of Bayesian Personalized Ranking (BPR) [26]. Alternatively, we can train the model using cross-entropy minimization. Finally, the prediction is obtained by  $\hat{y} = \operatorname{argmax}_{v} (\boldsymbol{u}^{\top} \boldsymbol{e}_{v})$ .
Datasets. We first evaluate our method on public benchmarks. diginetica contains the browser logs and anonymized transactions, Tmall collects anonymous users' shopping logs on the Tmall online website. We also acquire sessions from the browse and purchase logs from our E-commerce platform. We target at beauty, books, and electronics and gather 20-minute interactions within last successful purchases into one session after removing long-tail items. Appendix C provides more details.
Baselines. We compare our method with seven sequence-based baselines (FPMC [27], GRU4Rec [8], NARM [16], STAMP [19], CSRM [31], S3-Rec [47], and M2TRec [28]) and six graph-based baselines (SR-GNN [35], GC-SAN [38], S2-DHCN [37], GCE-GNN [34], LESSR [1], and MSGIFSR [7]). Model details are given in Appendix D. Each model is aligned with the official code implementation.
Evaluation. We evaluate SBRs as a ranking problem and employ Hits@  $K$ , NDCG@  $K$ , MRR@  $K$  as standard metrics. Hits@  $K$  measures the percentage of ranks up to and including  $K$ , while NDCG@  $K$
Table 2: Performance evaluation for next-item prediction, where standard deviations are enclosed in brackets. The best and second-best results are respectively highlighted in bold and underlined. Methods that use attributes are marked with  $\ddagger$ , and * indicates the  $p$ -value  $< 0.0001$  in t-test.
Model	diginetica	Tmall
Hits@10	NDCG@10	MRR@10	Hits@20	NDCG@20	MRR@20	Hits@10	NDCG@10	MRR@10	Hits@20	NDCG@20	MRR@20
FPMC	31.57*	17.40*	13.08*	43.19*	20.33*	13.88*	13.71*	9.02*	7.56*	16.44*	9.71*	7.74*
GRU4Rec	36.77*	20.71*	15.80*	49.68*	23.97*	16.70*	18.82*	12.28*	10.25**	22.68*	13.25*	10.51*
NARM	35.98*	20.18*	15.36*	48.89*	23.44*	16.26*	22.74*	15.46*	13.19*	26.73*	16.47*	13.47*
STAMP	33.59*	18.89*	14.41*	45.87*	22.00*	15.26*	24.32*	16.55*	14.12*	28.40*	17.58*	14.41*
CSRM	33.97*	19.43*	14.98*	45.83*	22.42*	15.80*	25.13*	18.56*	16.48*	27.94*	19.27*	16.68*
S3-Rec‡	33.48*	18.58*	14.04*	45.97*	21.74*	14.90*	18.24*	12.30*	10.46*	22.31*	13.32*	10.74*
M2TRec‡	29.67*	16.30*	12.23*	41.23*	19.22*	13.02*	11.42*	7.56*	6.36*	13.75*	8.15*	6.52*
SR-GNN	35.21*	19.68*	14.94*	47.99*	22.90*	15.82*	18.21*	12.11*	10.20*	21.34*	12.91*	10.42*
GC-SAN	35.25*	19.72*	14.97*	47.87*	22.90*	15.85*	19.29*	12.80*	10.78*	23.18*	13.78*	11.05*
S2-DHCN	30.76*	17.04*	12.86*	42.39*	19.98*	13.66*	22.00*	13.36*	10.68*	27.23*	14.69*	11.05*
GCE-GNN	36.32*	20.77*	16.02*	48.67*	23.89*	16.87*	28.33*	20.01*	17.32*	30.24*	20.50*	17.45*
LESSR	33.68*	18.71*	14.14*	46.23*	21.88*	15.01*	20.99*	14.64*	12.13*	25.92*	13.96*	10.50*
MSGIFSR	34.74*	19.43*	14.76*	46.23*	21.88*	15.01*	23.18*	15.19*	12.69*	27.78*	16.35*	13.01*
FAPAT‡	37.42	21.31	16.39	50.41	24.59	17.29	32.45	22.02	18.72	36.18	22.97	18.99
Improv.	3.03%	2.60%	2.31%	1.46%	2.59%	2.49%	14.19%	10.04%	8.08%	19.64%	12.05%	8.83%
Table 3: Performance evaluation for next-item prediction on our 100 million industrial data.
Model	Beauty	Books	Electronics
Hits@10	NDCG@10	Hits@20	NDCG@20	Hits@10	NDCG@10	Hits@20	NDCG@20	Hits@10	NDCG@10	Hits@20	NDCG@20
CSRMT-S3-Rec‡	89.74	75.28	92.61	76.01	78.69	56.70	82.88	57.77	62.28	44.35	67.47	45.67
89.64	75.56	92.53	76.30	75.00	58.54	79.45	59.67	74.36	56.03	79.63	57.37
M2TRec‡	80.13	65.97	83.66	66.87	32.56	22.58	35.39	25.70	57.32	44.84	61.70	45.95
SR-GNN	88.69	70.42	91.74	71.20	66.55	47.55	69.77	48.37	74.86	54.30	79.66	55.52
GCE-GNN	89.34	73.15	91.29	73.65	77.61	57.60	80.03	58.22	72.93	53.74	78.49	55.15
LESSR	89.95	71.29	92.98	72.06	73.72	53.86	82.31	54.77	72.91	50.46	78.78	51.96
MSGIFSR	90.18	73.62	92.50	74.21	72.93	52.23	76.33	53.09	73.56	53.83	77.45	54.73
FAPAT‡	92.72	76.29	94.10	76.87	81.62	61.08	85.12	61.97	78.36	56.81	82.81	57.94
Improv.	2.82%	0.97%	1.20%	0.75%	3.72%	4.34%	2.70%	3.85%	4.68%	1.39%	3.95%	0.99%
assigns higher scores to hits at the top of the list. MRR@  $K$  is the average of reciprocal ranks, with ranks above  $K$  assigned 0. For public benchmarking, we report the average performance across seeds  $\{2020, \dots, 2024\}$ , while for industrial data, we use a fixed seed (see Appendix E for more details).
5.2 Next-item Prediction Evaluation
Experimental Results. We first compare our models with selected baselines on two public datasets in Table 2. Overall, there is no huge difference between sequence and graph models when the session historical information is limited (e.g., in diginetica). However, we observe a performance boost with the history attention mechanism and session topology. FPMC that utilizes first-order Markov chains and matrix factorization is the worst. This reveals the difference between traditional recommendation and session-based recommendation. In contrast, RNN-based methods (GRU4Rec, NARM, STAMP, and CSRM) show better generalizability, along with the benefits of attention and memory. But we do not see further gains from pretraining and multi-task learning in S3-Rec and M2TRec. On the other hand, GNN-based algorithms for local sessions also achieve comparable results even with the absence of temporal signals. Moreover, the heterogeneous graphs of MSGIFSR beat the shortcut graphs from LESSR, indicating over-smoothing and potential noise. Although explicit global collaborative graphs enhance GCE-GNN, the implicit collaborative information from neurons and patterns in FAPAT easily surpasses other baselines significantly. At the same time, we evaluate algorithms on our 100 million industrial data. Table 3 demonstrates that most sequence models become unstable except S3-Rec. It employs incomplete data to predict masked items and attributes during pretraining and identifies contextualized collaborations by determining whether two incomplete sequences belong to the same session. But the multi-tasking learning for predicting attributes makes M2TRec hard to recommend items. While most GNN methods are competitive, S3-Rec and FAPAT are still better.
Over-smoothing Relief. We examine different graph topologies in baselines and FAPAT by comparing the graph density of different methods: the density of local session graphs (in SR-GNN), global collaborative filtering graphs(in GCE-GNN), shortcut graphs (in LESSR), heterogeneous graphs (in MSGIFSR), attribute patterns (in FAPAT) on E-commerce data are 3.658, 114.910, 25.447, 2.598, and 1.117, respectively. The density decrease significantly relieves the over-smoothing. Detailed analyses and expectations are provided in Appendix B.
Figure 5: Effects of different attribute settings.
Table 4: Results of encoder comparison, where * indicates the p-value  $< {0.05}$  in t-test.
Encoder	diginetica	Tmall	Beauty	Books
Hits@10	MRR@10	Hits@10	MRR@10	Hits@10	MRR@10	Hits@10	MRR@10
FAPAT w/o Attr.	36.82	16.29	31.53	18.86	88.70	67.11	77.18	49.66
GraphFormer	36.05*	16.17	30.05*	18.58	88.48	65.65	77.03	47.86
Transformer	36.30*	16.02*	28.83*	18.30*	88.10	65.90	74.67	46.54
Attribute Pattern Augmentation. To evaluate the effect of attribute patterns, we conduct experiments on variants with single attributes or without any attribute. The same soft attention strategy from Eq. (6) is employed to fuse attribute embeddings for competitive baselines. Results in Figure 5 show that attribute pattern augmentation is more stable than attribute soft attention. FAPAT benefits from graph-nested attention, where the graph attention aligns the hidden space, and the memory attention captures item correlations and user intents. But attribute embeddings may have side effects on optimization in baselines, especially in graph neural networks. We also discover that not all attributes have a positive impact. Comparing among attribute pattern numbers, attribute patterns with significant frequencies (slightly lower than or similar to the item number) can have adverse effects.
Graph-nested Attention. The graph-nested attention is one of our contributions, distinguishing it from GraphFormer [41]. Unlike GraphFormer, our graph attention is integrated inside the blocks, which allows for direct benefit from the broader attention in the following self-attention via backpropagation. To ensure fairness, we replace the encoding module of FAPAT with GraphFormer and vanilla Transformer. Results in Table 4 demonstrate the advantages of our proposed graph-nested attention. Our experiments also show that even a simple vanilla Transformer can outperform previous state-of-the-art models by a significant margin, indicating the importance of emphasizing temporal information in SBRs and the appropriateness of attention for capturing long-distance dependencies.
5.4 Intent Capture Inspection
Attribute Estimation. Beyond item predictions, we also estimate the awareness of user intents from the product attribute side. We do not require models (except M2TRec) to predict attributes but to retrieve attributes from predicted items instead. We consider it a successful estimation if the top-ranked items have the same attribute value as the ground truth. M2TRec performs well on public data with multi-task learning but struggles on industrial E-commerce data with four attributes, as shown in Table 5. Pretraining in S3-Rec is not helpful due to catastrophic forgetting. GNN-based models perform similarly, except on Tmall, where the data are too sparse so that global collaborative information assists. On the contrary, FAPAT achieves robust predictions across four datasets. Even when session data are sufficient, frequent patterns remain effective.
Table 5: Attribute estimation evaluation.
Model	diginetica	Tmall	Beauty	Books
Hits@10	MRR@10	Hits@10	MRR@10	Hits@10	MRR@10	Hits@10	MRR@10
CSRM	89.87*	87.91*	48.03*	34.97*	94.82	83.57	88.93	73.31
M2TRec‡	94.10	89.76	59.44	39.72	95.93	84.95	83.80	71.22
S3-Rec‡	89.66*	88.42*	44.95*	32.60*	95.35	84.79	88.11	75.63
GCE-GNN	89.89*	87.96*	55.88*	38.57*	95.44	84.16	89.78	75.68
LESSR	88.60*	86.24*	49.43*	32.61*	95.02	81.56	89.57	73.73
MSGIFSR	89.48*	87.19*	50.09*	33.95*	95.79	85.42	86.67	71.99
FAPAT‡	89.99*	88.23*	59.49	40.56	95.94	86.87	90.77	76.83
w/o Attr.	89.35*	87.50*	58.22*	40.08	95.07	81.65	89.29	74.12
(a) Tmall
(b) Books
(c) Electronics
Figure 6: Period-item recommendation evaluation.
Period-item Recommendation. In addition to single-step evaluation, we revisit recommenders. A sound and robust SRB system must understand user intents in deep and foresee the possible consistency and the potential change. Autoregressive settings pose challenges for GNNs due to short click histories and error accumulation. Therefore, we evaluate period recommendations like search engines by comparing the top-10 predicted items with the next 3/5/10 clicks. Figure 6 demonstrates period-recommendation performance. Sequence models offer steady results in Recall, indicating that temporal information is one of the prerequisites to analyzing users' latest intents. However, pretraining may hinder models' ability to generalize over long periods, resulting in a severe decline in NDCG. Our FAPAT achieves the best performance in all metrics, indicating the effectiveness of attribute graphlets in capturing deep user intents.
Our paper introduces FAPAT, a novel framework that leverages attribute graph patterns to augment anonymous sequence encoding for session-based recommendations. Compared to other GNN-based methods, frequent attribute graphlets can reduce noise and topology densities for enhancing user intent capture. Our sequence encoder can better preserve temporal signals and forecast the user's latest intents. Experimental results clearly illustrate the effectiveness, and extensive ablation studies and intent capture inspections provide additional support. We discuss limitations in Appendix A. One of the future works is to improve ranking priority by combining pretraining and pattern augmentation.
[37] Xin Xia, Hongzhi Yin, Junliang Yu, Qinyong Wang, Lizhen Cui, and Xiangliang Zhang. Self-supervised hypergraph convolutional networks for session-based recommendation. In AAAI, pages 4503-4511, 2021.
[38] Chengfeng Xu, Pengpeng Zhao, Yanchi Liu, Victor S. Sheng, Jiajie Xu, Fuzhen Zhuang, Junhua Fang, and Xiaofang Zhou. Graph contextualized self-attention network for session-based recommendation. In IJCAI, pages 3940–3946, 2019.
[39] Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. How powerful are graph neural networks? In ICLR, 2019.
[40] Xifeng Yan and Jiawei Han. gspan: Graph-based substructure pattern mining. In ICDM, pages 721-724, 2002.
[41] Junhan Yang, Zheng Liu, Shitao Xiao, Chaozhuo Li, Defu Lian, Sanjay Agrawal, Amit Singh, Guangzhong Sun, and Xing Xie. Graphformers: Gnn-nested transformers for representation learning on textual graph. In NeurIPS, pages 28798-28810, 2021.
[42] Ghim-Eng Yap, Xiaoli Li, and Philip S. Yu. Effective next-items recommendation via personalized sequential pattern mining. In DASFAA, volume 7239, pages 48–64, 2012.
[43] Changlong Yu, Weiqi Wang, Xin Liu, Jiaxin Bai, Yangqiu Song, Zheng Li, Yifan Gao, Tianyu Cao, and Bing Yin. FolkScope: Intention knowledge graph construction for E-commerce commonsense discovery. In Finding of ACL, pages 1173-1191, 2023.
[44] Danqing Zhang, Zheng Li, Tianyu Cao, Chen Luo, Tony Wu, Hanqing Lu, Yiwei Song, Bing Yin, Tuo Zhao, and Qiang Yang. Queaco: Borrowing treasures from weakly-labeled behavior data for query attribute value extraction. In CIKM, page 4362-4372, 2021.
[45] Peiyan Zhang, Jiayan Guo, Chaozhuo Li, Yueqi Xie, Jaeboum Kim, Yan Zhang, Xing Xie, Haohan Wang, and Sunghun Kim. Efficiently leveraging multi-level user intent for session-based recommendation via attenu-mixer network. In WSDM, 2023.
[46] Lingxiao Zhao and Leman Akoglu. Pairnorm: Tackling oversmoothing in gnns. In ICLR, 2020.
[47] Kun Zhou, Hui Wang, Wayne Xin Zhao, Yutao Zhu, Sirui Wang, Fuzheng Zhang, Zhongyuan Wang, and Ji-Rong Wen. S3-rec: Self-supervised learning for sequential recommendation with mutual information maximization. In CIKM, pages 1893–1902, 2020.
Figure 7: Different granularities of graphs derived from a session, where the violet edges, blue edges, and green edges correspond to the global collaborations, shortcuts, and higher-level heterogeneous connections, respectively.
There are three limitations to our current proposed method and evaluation. First, our method separately processes and retrieves patterns for each attribute type. We do not merge all attributes in a candidate pool because we aim for our method to easily generalize to real recommendation systems with hundreds of attribute types and category hierarchies. The current implementation supports adding a new attribute type to a model as long as its embeddings align with the embeddings of other attributes. Second, we conducted experiments based on "clean" session data. Most E-commerce platforms do not have truly clean data on product attributes, so attribute data, in general, is very sparse and full of invalid values. We performed human-centric attribute regularization to drop products without valid attribute values, which may create a gap compared to a real industrial system. Third, the evaluation does not consider the same products with different identifiers. Therefore, evaluating results (especially MRR) cannot accurately reflect the performance. To better reflect the real performance with error tolerance, a larger K is suggested. The current comparison is still fair for all algorithms, and we address this synonym problem in attribute estimation in § 5.4, where we merge attribute values based on semantics and syntax.
B Transition Graph Density
The graph structure is crucial for neural networks to capture explicit transitions and implicit connections. A local session records the history of a user's clicks or purchases, which is usually sparse. In contrast, the global collaborative graph could be extremely dense because each pair of items may have a potential connection. From this perspective, the density indicates the explicit information provided from session data. On the other hand, different graph topologies and densities also present different focuses and challenges. The sparse local transition graph emphasizes current intents, while the global collaboration indicates broader interests and revenues. Graph neural networks excel at capturing local features, but a large number of neighbors can overshadow important connections with less significant ones. Considering that previous methods have focused on different granularities individually, we summarize them in Figure 7 and compare them in terms of optimization interpretations<sup>3</sup>.
- Local session graphs correspond to item transitions within a session, where edges are created between two consecutively clicked/purchased items. The density is usually sparse (slightly greater than 1.0), allowing exploration and global collaboration to be learned through model parameters instead of explicit connections. Therefore, generalizing to unseen click patterns becomes challenging.
- Global transition graphs record all collaborations. In a real industrial system, the density is usually beyond one hundred or even one thousand. Ideally, any session can benefit from this global collaborative information, including multi-hop connections. However, optimizing graph neural
<sup>3</sup>Graphs are typically considered undirected in practical algorithms.
networks to learn such topologies (due to oversmoothing) and building a large model for real-time inference (due to latency and streaming processing) pose challenges.
- Shortcut graphs aim to avoid constructing global graphs and make session learning more efficient. They were proposed by LESSR [1] to address information loss in graph convolutions. Specifically, they allow latent items to be aware of all previous clicks, resembling shortcuts for multi-hop neighbors in the directed local session graph. However, they lack the extensive exploration capabilities of the global transition graph and suffer from oversmoothing issues due to dense connections.
- Heterogeneous graphs strike a balance between shortcuts and local adjacency. Nodes with different numbers of items are categorized into different groups, and transition edges capture varying levels of spatial continuity. From a high-level perspective, this graph is sparser than the local session graph, resulting in faster convergence for optimization. However, the propagation of high-order information introduces additional processing costs and the risk of overfitting.
- Patterns, especially attribute patterns, should be the most efficient features for recommendations in a large candidate item pool. Each pattern can be considered a higher-grained heterogeneous graph. However, pattern filtering can significantly eliminate noise influence, not to mention the benefits gained from offline indexing. Besides, the partial match of patterns can provide the intent information from other sessions, making the learning and prediction more reliable and steady.
C Experimental Data
C.1 Public Benchmarks
We choose two public benchmarks for session-based recommendation evaluation: diginetica  $^{4}$  is CIKM Cup 2016 that contains the browser logs and anonymized transactions; Tmall  $^{5}$  comes from a competition in IJCAI-15 which collects anonymous users' shopping logs on the Tmall online website. We acquire attributes from the original data and drop items without attributes or with invalid values. Therefore, the performance of baselines may not be exactly same as the reported numbers in the original papers.
C.2 E-commerce Data Collection
We collect E-commerce data from our log systems in two months. We follow the same procedure to clean and process session data in beauty, books, and electronics domains:
I We focus on successful purchases so that we only keep sessions ending with "purchase" actions.
II To make sure previous clicks can reflect the purchase intent, we drop actions 20 minutes ago.
III We filter out items with missing attributes (i.e., books without publishers, authors, or genre, and electronics without colors and brands).
IV We adopt the 20-core setting to finalize the item sets, in which items appear on at least 20 different days.
V Only sessions whose length is no greater than 50 are preserved.
VI We retrieve item attributes in our attribute databases.
VII For GNN models that requires the global transition graph from training data, we maintain 12 neighbors based on the co-occurrence, which is consistent with GCE-GNN [34].
4https://competitions.codalab.org/competitions/11161
<sup>5</sup>https://tianchi.aliyun.com/dataset/dataDetail?dataId=42
The sampled data scales and distributions are different in real systems due to out-of-domain items filtering.