SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Wei Jin $^{12*}$ , Haitao Mao $^{3*}$ , Zheng Li $^{1}$ , Haoming Jiang $^{1}$ , Chen Luo $^{1}$ , Hongzhi Wen $^{3}$ , Haoyu Han $^{3}$ , Hanqing Lu $^{1}$ , Zhengyang Wang $^{1}$ , Ruirui Li $^{1}$ , Zhen Li $^{1}$ , Monica Cheng $^{1}$ , Rahul Goutam $^{1}$ , Haiyang Zhang $^{1}$ , Karthik Subbian $^{1}$ , Suhang Wang $^{4}$ , Yizhou Sun $^{5}$ , Jiliang Tang $^{3}$ , Bing Yin $^{1}$ , and Xianfeng Tang $^{1}$
$^{1}$  Amazon.com  $^{2}$  Emory University  $^{3}$  Michigan State University  $^{4}$  The Pennsylvania State University  $^{5}$  University of California, Los Angeles {joewjin,amzzhe,jhaoming,zhengywa,ruiril,uhanqin,cheluo} @amazon.com, {amzzhn,chengxc,rgoutam,hhaiz,ksubbian,alexbyin,xianft} @amazon.com, wei.jin@emory.edu, szw494@psu.edu, yzsun@cs.ucla.edu, {haitaoma,wenhongz,hanhaoy1,tangjili} @msu.edu
Modeling customer shopping intentions is a crucial task for e-commerce, as it directly impacts user experience and engagement. Thus, accurately understanding customer preferences is essential for providing personalized recommendations. Session-based recommendation, which utilizes customer session data to predict their next interaction, has become increasingly popular. However, existing session datasets have limitations in terms of item attributes, user diversity, and dataset scale. As a result, they cannot comprehensively capture the spectrum of user behaviors and preferences. To bridge this gap, we present the Amazon Multilingual Multi-locale Shopping Session Dataset, namely Amazon-M2. It is the first multilingual dataset consisting of millions of user sessions from six different locales, where the major languages of products are English, German, Japanese, French, Italian, and Spanish. Remarkably, the dataset can help us enhance personalization and understanding of user preferences, which can benefit various existing tasks as well as enable new tasks. To test the potential of the dataset, we introduce three tasks in this work: (1) next-product recommendation, (2) next-product recommendation with domain shifts, and (3) next-product title generation. With the above tasks, we benchmark a range of algorithms on our proposed dataset, drawing new insights for further research and practice. In addition, based on the proposed dataset and tasks, we hosted a competition in the KDD CUP  $2023^2$  and have attracted thousands of users and submissions. The winning solutions and the associated workshop can be accessed at our website https://kddcup23.github.io/.
In the era of information explosion, recommender systems have become a prevalent tool for understanding user preferences and reducing information overload [1, 2, 3, 4, 5, 6]. Traditionally, the majority of recommendation algorithms focus on understanding long-term user interests through utilizing user-profiles and behavioral records. However, they tend to overlook the user's current purpose
$^{2}$ https://www.aicrowd.com/challenges/amazon-kdd-cup-23-multilingual-recommendation-challenge
which often has a dominant impact on user's next behavior. Besides, many recommendation algorithms require access to user profiles [7, 8, 9], which can be incomplete or even missing in real-world situations especially when users are browsing in an incognito mode. In these cases, only the most recent user interactions in the current session can be utilized for understanding their preferences. Consequently, the session-based recommendation has emerged as an effective solution for modeling user's short-term interest, focusing on a user's most recent interactions within the current session to predict the next product. Over the past few years, the session-based recommendation has gained significant attention and has prompted the development of numerous models [10, 11, 12, 13, 14, 15, 16, 17].
A critical ingredient for evaluating the efficacy of these methods is the session dataset. While numerous session datasets [18, 19, 20, 11, 21] have been carefully curated to meet the requirements of modeling user intent and are extensively employed for evaluating session-based recommender systems, they have several drawbacks. First, existing datasets only provide limited product attributes, resulting in incomplete product information and obscuring studies that leverage attribute information to advance the recommendation. Second, the user diversity within these datasets is limited and may not adequately represent the diversity of user-profiles and behaviors. Consequently, it can result in biased or less accurate recommendations, as the models may not capture the full range of customer preferences. Third, the dataset scale, particularly in terms of the product set, is limited, which falls short of reflecting real-world recommendation scenarios with vast product and user bases.
To break the aforementioned limitations, we introduce the Amazon Multilingual Multi-Locale Shopping Session Dataset, namely Amazon-M2, a large dataset of anonymized user sessions with their interacted products collected from multiple language sources at Amazon. Specifically, the dataset contains samples constructed from real user session data, where each sample contains a list of user-engaged products in chronological order. In addition, we provide a table of product attributes, which contains all the interacted products with their associated attributes such as title, brand, color, etc. Modeling such session data can help us better understand customers' shopping intentions, which is also the main focus of e-commerce. Particularly, the proposed dataset exhibits the following characteristics that make it unique from existing session datasets.
(a) Rich semantic attributes: Amazon-M2 includes rich product attributes (categorical, textual, and numerical attributes) as product features including title, price, brand, description, etc. These attributes provide a great opportunity to accurately comprehend the user's interests. To our best knowledge, it is the first session dataset to provide textual features.
(b) Large scale: Amazon-M2 is a large-scale dataset with millions of user sessions and products, while existing datasets only contain tens of thousands of products.
(c) Multiple locales: Amazon-M2 collected data from diverse sources, i.e., six different locales including the United Kingdom, Japan, Italian, Spanish, French, and Germany. Thus, it provides a diverse range of user behaviors and preferences, which can facilitate the design of less biased and more accurate recommendation algorithms.
(d) Multiple languages: Given the included locales, Amazon-M2 is special for its multilingual property. Particularly, six different languages (English, Japanese, Italian, Spanish, French, and German) are provided. It enables us to leverage recent advances such as language models [22, 23, 24] to model different languages in user sessions.
By utilizing this dataset, we can perform diverse downstream tasks for evaluating relevant algorithms in recommendation and text generation. Here, we focus on three different tasks, consisting of (1) next-product recommendation, (2) next-product recommendation with domain shifts, and (3) next-product title generation. The first task is the classic session-based recommendation which requires models to predict the ID of the next product, where the training dataset and test dataset are from the same domain. The second task is similar to the first task but requires the models to pre-train on the large dataset from large locales and transfer the knowledge to make predictions on downstream datasets from different domains (i.e., underrepresented locales). The third task is a novel task proposed by us, which asks models to predict the title of the next product which has never been shown in the training set. Based on these tasks, we benchmark representative baselines along with simple heuristic methods. Our empirical observations suggest that the representative baselines fail to outperform simple heuristic methods in certain evaluation metrics in these new settings. Therefore, we believe that Amazon-M2 can inspire novel solutions for session-based recommendation and enable new opportunities for tasks that revolve around large language models and recommender systems.
Table 1: Comparison among popular session datasets. Note that "Diverse products" indicates whether the products are from diverse categories or only specific categories (e.g., fashion clothes).
#Train Sessions	#Test Sessions	#Products	Textual feat.	Multilingual	Multi-locale	Diverse products
Yoochoose [18]	1,956,539	15,324	30,660	X	X	X	✓
Tmall [21]	195,547	4,742	40,728	X	X	X	✓
Diginetica [25]	186,670	15,963	43,097	X	X	X	✓
Dressipi [19]	1,000,000	100,000	23,496	X	X	✓	X
Amazon-M2	3,606,249	361,659	1,410,675	✓	✓	✓	✓
2 Dataset & Task Description
2.1 Dataset Description
Before we elaborate on the details of the proposed Amazon-M2 dataset, we first introduce session-based recommendation. Given a user session, session-based recommendation aims to provide a recommendation on the product that the user might interact with at the next time step. As shown in Figure 1a, each session is represented as a chronological list of products that the user has interacted with. Specifically, we use  $S = \{s_1, s_2, \dots, s_n\}$  to denote the session dataset containing  $n$  sessions, where each session is represented by  $s = \{e_1, e_2, \dots, e_t\}$  with  $e_t$  indicating the product interacted by the user at time step  $t$ . In addition, let  $V = \{v_1, v_2, \dots, v_m\}$  denote a dictionary of unique products that appeared in the sessions, and each product is associated with some attributes.
Designed for session-based recommendation, Amazon-M2 is a large-scale dataset composed of customer shopping sessions with interacted products. Specifically, the dataset consists of two components: (1) user sessions where each session is a list of product IDs interacted by the current user (Figure 1a), and (2) a table of products with each row representing the attributes of one product (Figure 1b). Particularly, the user sessions come from six different locales, i.e., the United Kingdom (UK), Japan (JP), German (DE), Spain (ES), Italian (IT), and France (FR). Given its multi-locale nature, the dataset is also multilingual: the textual attributes (e.g., title and description) of the products in the user sessions are in multiple languages, namely, English, Italian, French, Germany, and Spanish. Based on this dataset, we construct the training/test dataset for each task. A summary of our session dataset is given in Table 2. It includes the number of sessions, the number of interactions, the number of products, and the average session length for six different locales. We can find that UK, DE, and JP have approximately 10 times the number of sessions/products compared to ES, FR, and IT. More details about the collection process of the dataset can be found in Appendix B.
Comparison with Existing Datasets. We summarize the differences between existing session datasets (especially from session-based recommendation) and Amazon-M2 in Table 1. First of all, Amazon-M2 is the first dataset to provide textural information while other datasets majorly focus on the product ID information or categorical attributes. Without comprehensive product attributes, the recommendation models may struggle to capture the nuanced preferences of customers. Second, existing datasets only provide sessions from a single locale (or country) which limits their user diversity. Consequently, it may lead to biased or less accurate recommendations, as the models may not capture the full range of customer preferences. By contrast, our proposed Amazon-M2 is collected from multiple locales and is multilingual in nature. Third, our proposed Amazon-M2 provides a large number of user sessions and is on a much larger product scale, which can better reflect real-world recommendation scenarios with huge product bases.
2.2 Task Description
The primary goal of this dataset is to inspire new recommendation strategies and simultaneously identify interesting products that can be used to improve the customer shopping experience. We introduce the following three different tasks using our shopping session dataset.
(a) Task 1. Next-product recommendation. This task focuses on traditional session-based recommendation task, aiming to identify the next product of interest within a user session. Given a user session, the goal of this task is to predict the next product that the user will interact with. Note that the training/test data are from the same distribution of large locales (JP, UK, and DE).
Figure 1: An illustration of the proposed Amazon-M2 dataset. (a) A user session contains a list of previous products that the user has interacted with and the next product that the user is going to interact with. The user sessions come from multiple locales such as UK, ES, JP, etc. It is important to note that one product can appear in multiple locales. (b) The product attributes are publicly available and can be accessed on Amazon.com. Users can find information about a specific product by its ASIN number.
(b) Task 2. Next-product recommendation with domain shifts. This task is similar to Task 1 but advocates a novel setting of transfer-learning: practitioners are required to perform pretraining on a large pretraining dataset (user sessions from JP, UK, and DE) and then finetune and make predictions on the downstream datasets of underrepresented locales (user sessions from ES, IT, and FR). Thus, there is a domain shift between the pretraining dataset and the downstream dataset, which requires the model to transfer knowledge from large locales to facilitate the recommendation for underrepresented locales. This can address the challenge of data scarcity in these languages and improve the accuracy and relevance of the recommendations. More discussions on distribution shift can be found in Section 3.
(c) Task 3. Next-product title generation. This is a brand-new task designed for session datasets with textual attributes, and it aims to predict the title of the next product that the user will interact with within the current session. Notably, the task is challenging as the products in the test set do not appear in the training set. The generated titles can be used to improve cold-start recommendation and search functionality within the e-commerce platform. By accurately predicting the title of the next product, users can more easily find and discover items of interest, leading to improved user satisfaction and engagement.
The tasks described above span across the fields of recommender systems, transfer learning, and natural language processing. By providing a challenging dataset, this work can facilitate the development of these relevant areas and enable the evaluation of machine learning models in realistic scenarios.
3 Dataset Analysis
In this section, we offer a comprehensive analysis of the Amazon-M2 dataset to uncover valuable insights. Our analysis covers several essential perspectives: long-tail phenomenon, product overlap between locales, session lengths, repeat pattern, and collaborative filtering pattern. Corresponding codes can be found here.
Long-tail phenomenon [26, 27] is a significant challenge in the session recommendation domain. It refers to the situation where only a handful of products enjoy high popularity, while the majority of products receive only a limited number of interactions. To investigate the presence of the long-tail phenomenon in Amazon-M2 dataset, we analyze the distribution of product frequencies, as depicted
Table 2: Statistics of the multilingual shopping session dataset for training and test: the number of sessions, the number of products, the number of interactions, and the average session length.
Train	Test
Language	#Products	#Sessions	#Interactions	Avg. Length	#Sessions	#Interactions	Avg. Length
UK	500,180	1,182,181	4,872,506	4.1	115,936	466,265	4.0
DE	518,327	1,111,416	4,836,983	4.4	104,568	450,090	4.3
JP	395,009	979,119	4,388,790	4.5	96,467	434,777	4.5
ES	42,503	89,047	326,256	3.7	8,176	31,133	3.8
FR	44,577	117,561	416,797	3.5	12,520	48,143	3.9
IT	50,461	126,925	464,851	3.7	13,992	53,414	3.8
Total	1,410,675	3,606,249	15,306,183	4.2	361,659	1,483,822	4.2
(a) Product frequency
(b) Product overlap ratio
(c) Session length
(d) Ratio of sessions with repeat pattern
(e) Distribution of repeated products in the session
(f) Distribution of 10th highest SKNN score
Figure 2: Data analysis on Amazon-M2. (a)(c)(e) illustrate the long-tail phenomenon of product frequency, session length distribution, and the number of repeat products in a single session. (b) shows product overlap ratio between locales. (d) illustrates the proportion of sessions with repeat patterns in different locales. (f) shows most sessions can find relevant products of high SKNN scores.
in Figure 2a. The results clearly demonstrate the existence of a long-tail distribution, where the head of the distribution represents popular items and the tail consists of less popular ones. Furthermore, we observe that the long-tail phenomenon is also evident within each individual locale. For detailed experimental results regarding this phenomenon in each locale, please refer to Appendix B.2. The long-tail distribution makes it difficult to effectively recommend less popular products, as a small number of popular items dominate the recommendations.
Product overlap ratio between locales is the proportion of the same products shared by different locales. A large number of overlapping products indicates a better transferability potential when transferring the knowledge from one locale to the other. For example, cross-domain recommendation algorithms like [28] can then be successfully applied, which directly transfers the learned embedding of the overlapping products from popular locales to the underrepresented locales. We then examine product overlap between locales in Amazon-M2 with the product overlap ratio. It is calculated as  $\frac{|\mathcal{N}_a\cap\mathcal{N}_b|}{|N_a|}$ , where  $\mathcal{N}_a$  and  $\mathcal{N}_b$  correspond to the products set of locale  $a$  and  $b$ , respectively. In Figure 2b we use a heatmap to show the overlap ratio, where  $x$  and  $y$  axes stand for locale  $a$  and  $b$ , respectively. From the figure, we make several observations: (1) For the products in the three large locales, i.e., UK, DE, and JP, there are not many overlapping products, except the one between UK and DE locales;
(2) Considering the product overlap ratio between large locales and underrepresented locales, i.e., ES, FR, and IT, we can see a large product overlapping, indicating products in the underrepresented domain also appear in the large locales. Particularly, the overlap ratio between small locales and DE can reach around 0.4. Thus, it has the potential to facilitate knowledge transfer from large locales and areas to underrepresented regions.
Notably, despite the existence of overlapping products between different locales, there still remains a large proportion of distinguished products in each locale, indicating the difficulty of transferability with distribution shift. Moreover, the multilingual property, where the product textual description from different locales is in different languages, also induces to distribution shift issue. Such a multilingual issue is a long-established topic in the NLP domain. For instance, [29, 30, 31] point out morphology disparity, tokenization differences, and negative transfer in the multilingual scenario, leading to distribution shift.
Session length is an important factor in the session recommendation domain. Typically, a longer session length may lead to the interest shift challenge problem [15] with difficulties in capturing multiple user interests in one single session. Most existing algorithms [13, 16] show a better performance on the shorter sessions while failing down on the longer ones. As shown in Figure 2c. We can observe that the session length also exhibits a long-tail distribution: most sessions are short while only few sessions are with a length larger than 100.
Repeat pattern [32, 33, 34, 35] is also an important user pattern, which refers to the phenomenon that a user repeatedly engages the same products multiple times in a single session. The presence of repeat patterns in recommender systems can potentially result in the system offering more familiar products that match users' existing preferences, which may lead to a less diverse and potentially less satisfying user experience. On the other hand, the repeat pattern is also an important property utilized in the graph-based session recommendation algorithms [13, 17, 36, 37]. Typically, those graph-based algorithms construct a session graph where each node represents a product and each edge indicates two products interacted by the user consecutively. Complicated session graphs with different structure patterns can be built when sessions exhibit evident repeat patterns. In Figure 2d, we report the proportion of sessions with repeat patterns for the six locales and we can observe that there are around  $35\%$  sessions with repeat patterns across different locales. Furthermore, we examine the number of repeat products in those sessions with repeat patterns and report results on the distribution of repeated products in Figure 2e. We make two observations: (1) the number of repeated products varies on different sessions; and (2) the number of repeated products in a session also follows the long-tail distribution where most sessions only appear with a few repeated products.
Collaborative filtering pattern. Collaborative filtering is a widely used technique that generates recommended products based on the behavior of other similar users. It is generally utilized as an important data argumentation technique to alleviate the data sparsity issue, especially for short sessions [38, 39]. Since Amazon-M2 encompasses a much larger product set than existing datasets, we investigate whether collaborative filtering techniques can potentially operate in this challenging data environment. Specifically, we utilize the session collaborative filtering algorithm, Session-KNN (SKNN) [11], to identify sessions that are similar to the target user's current session. The similarity score of SKNN can be calculated in the following steps. First, for a particular session  $s$ , we first determines a set of its most similar sessions  $\mathcal{N}(s) \subseteq S$  with the cosine similarity  $\mathrm{sim}(s,s_j) = |s \cap s_j| / \sqrt{|s||s_j|}$ . Second, the score of a candidate product  $e$  in similar sessions  $\mathcal{N}(s)$  is then calculated by:  $\mathrm{score}(e,s) = \sum_{n \in \mathcal{N}(s)} \mathrm{sim}(s,n) I_n(e)$ , where the indicator function  $I_n(e)$  is 1 if  $n$  contains item  $e$ . Third, for each session  $s$ , we choose the 10th highest score  $(e,s)$  to indicate the retrieval quality of candidate products. In Figure 2f, we show the distribution of the 10th highest SKNN scores for all sessions. A notable observation is that the majority of sessions exhibit a high SKNN score, hovering around 1. This finding suggests that for most sessions, it is possible to retrieve at least 10 similar products to augment the session data.
4 Benchmark on the Proposed Three Tasks
4.1 Task 1. Next-product Recommendation
In this task, we evaluate the following popular baseline models (deep learning models) in session-based recommendation on our proposed Amazon-M2, with the help of Recobole package [3]:
Table 3: Experimental results on Task 1, next product prediction.
MRR@100	Recall@100
UK	DE	JP	Overall	UK	DE	JP	Overall
Popularity	0.2302	0.2259	0.2766	0.2426	0.4047	0.4065	0.4683	0.4243
GRU4Rec++	0.1656	0.1578	0.2080	0.1757	0.3665	0.3627	0.4185	0.3808
NARM	0.1801	0.1716	0.2255	0.1908	0.4021	0.4016	0.4559	0.4180
STAMP	0.2050	0.1955	0.2521	0.2159	0.3370	0.3320	0.3900	0.3512
SRGNN	0.1841	0.1767	0.2282	0.1948	0.3882	0.3862	0.4403	0.4032
CORE	0.1510	0.1510	0.1846	0.1609	0.5591	0.5525	0.5898	0.5591
MGS	0.1668	0.1739	0.2376	0.1907	0.5641	0.5479	0.4677	0.5194