SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Table 4: Experimental results on Task 2, next-product recommendation with domain shifts.
MRR@100	Recall@100
Methods	ES	FR	IT	Overall	ES	FR	IT	Overall
Heuristic	Popularity	0.2854	0.2940	0.2706	0.2829	0.5118	0.5166	0.4914	0.5058
Supervised training	GRU4Rec++	0.2538	0.2734	0.2420	0.2564	0.5817	0.5995	0.5753	0.5856
NARM	0.2598	0.2805	0.2493	0.2632	0.5894	0.6061	0.5803	0.5920
STAMP	0.2555	0.2752	0.2430	0.2578	0.5162	0.5365	0.5116	0.5217
SRGNN	0.2627	0.2797	0.2500	0.2640	0.5426	0.5543	0.5322	0.5429
CORE	0.1978	0.2204	0.1941	0.2045	0.6931	0.7215	0.6937	0.7034
MGS	0.2491	0.2775	0.2411	0.2560	0.5829	0.5811	0.5689	0.5766
Pretraining & finetuning	GRU4Rec++	0.2601	0.2856	0.2480	0.2646	0.5989	0.6236	0.5898	0.6042
NARM	0.2733	0.2917	0.2564	0.2734	0.6094	0.6315	0.5974	0.6127
STAMP	0.2701	0.2878	0.2584	0.2719	0.4792	0.4988	0.4637	0.4803
SRGNN	0.2747	0.2980	0.2612	0.2779	0.5835	0.6101	0.5714	0.5883
CORE	0.1685	0.1839	0.1632	0.1720	0.6946	0.7156	0.6851	0.6985
MGS	0.2612	0.2870	0.2693	0.2722	0.5747	0.6133	0.5812	0.5913
- GRU4REC++ [40] is an improved model based on GRU4Rec which adopts two techniques to improve the performance of GRU4Rec, including a data augmentation process and a method to account for shifts in the 1 input data distribution
- NARM [15] employs RNNs with attention mechanism to capture the user's main purpose and sequential behavior.
- STAMP [16] captures user's current interest and general interests based on the last-click product and whole session, respectively.
- SRGNN [13] is the first to employ GNN layer to capture user interest in the current session.
- CORE [41] ensures that sessions and items are in the same representation space via encoding the session embedding as a linear combination of item embeddings.
- MGS [42] incorporates product attribute information to construct a mirror graph, aiming to learn better preference understanding via combining session graph and mirror graph. Notably, MGS can only adapt categorical attributes. Therefore, we discretize the price attribute as the input feature.
In addition, we include a simple yet effective method, Popularity, by simply recommending all users the most popular products. We utilize Mean Reciprocal Rank@K (MRR@100) and Recall@100 to evaluate various recommendation algorithms. More results on NDCG@100 metric can be found in Appendix C. Corresponding codes can be found here.
Results & Observations. The experiment results across different locales can be found in Table 3. We can observe that the popularity heuristic generally outperforms all the deep models with respect to both MRR and Recall. The only exception is that CORE achieves better performance on Recall. On one hand, the success of the popularity heuristic indicates that the product popularity is a strong bias for this dataset. On the other hand, it indicates that the large product set in Amazon-M2 poses great challenges in developing effective recommendation algorithms. Thus, more advanced recommendation strategies are needed for handling the challenging Amazon-M2 dataset.
4.2 Task 2. Next-product Recommendation with Domain Shifts
The purpose of Task 2 is to provide next-product recommendations for underrepresented locales, specifically ES, IT, and FR. To evaluate the baseline methods, we consider two training paradigms. (1) Supervised training, which involves directly training models on the training data on ES, IT, and FR and then testing them on their test data. The goal of this paradigm is to evaluate the models' efficacy when no other data is available. (2) Pretraining & finetuning, which begins with pretraining the model using data from large locales, i.e., JP, UK, and DE, and then finetuning this pretrained model on the training data from ES, IT, and FR. This approach tests the models' ability to transfer knowledge from one source to another. In both paradigms, we incorporate the same baseline models used in Task 1, which include GRU4REC++ NARM, STAMP, SRGNN, and CORE. Additionally, we compare these methods with the popularity heuristic.
Results & Observations. The results across different locales can be found in Table 4. From the results, we have the following observations: (1) In the context of supervised training, the popularity heuristic outperforms all baselines in terms of MRR on underrepresented locales, which is consistent with observations from Task 1. Interestingly, despite this, most deep learning models surpass the popularity heuristic when considering Recall. This suggests a promising potential for deep learning models: they are more likely to provide the correct recommendations among the retrieved candidate set while the rankings are not high. (2) Finetuning most methods tends to enhance the performance in terms of both MRR and Recall compared to supervised training alone. This demonstrates that the knowledge of large locales can be transferred to underrepresented locales despite the existence of domain shifts. Importantly, it highlights the potential of adopting pretraining to enhance the performance on locales with limited data in the Amazon-M2 dataset.
Ablation Study. Notably, the aforementioned methods utilized random initialization as the product embeddings. To enhance the model's ability to transfer knowledge across different locales, we can initialize product embeddings with embeddings from textual attributes. Specifically, we introduce SRGNNF and GRU4RecF to enable SRGNN and GRU4Rec++ to leverage the textual attributes (i.e., product title), respectively. To obtain the text embedding, we opt for pre-trained Multilingual Sentence BERT [43] to generate additional item embeddings from the item titles. As shown in Table 5, this straightforward approach has unintended effects in our ablation study. This could be due to the fact that the pre-training data for Multilingual Sentence BERT does not align with the text of product titles. Therefore, how to effectively utilize textual features remains an open question. We look forward to more research concerning the integration of language models with session recommendation.
4.3 Task 3. Next-product Title Generation
In this task, the goal is to generate the title of the next product of interest for the user, which is a text generation task. Therefore, we adopt the bilingual evaluation understudy (BLEU) score, a classic metric in natural language generation, as the evaluation metric. For the same reason, in this task, we explore the effectiveness of directly applying language models as a baseline recommendation model. We fine-tune a well-known multilingual pre-trained language model, mT5 [44], using a generative objective defined on our dataset. Specifically, the language model receives the titles of the most recent  $K$  products and is trained to predict the title of the next product. We compare the performance by varying  $K$  from 1 to 3, while also investigating the impact of parameter size. Additionally, we offer a simple yet effective baseline approach, namely Last Product Title, in which the title for the next product is predicted as that of the last product. We randomly select  $10\%$  of training data for validation, and report the BLEU scores of different methods on validation and test set in Table 6.
Table 5: Ablation study on whether models can effectively leverage textual features.
MRR@100	Recall@100
ES	FR	IT	Overall	ES	FR	IT	Overall
SRGNN	0.2627	0.2797	0.2500	0.2640	0.5426	0.5543	0.5322	0.5429
SRGNNF	0.2107	0.2546	0.2239	0.2303	0.4541	0.5132	0.4788	0.4976
GRU4Rec++	0.2538	0.2734	0.2420	0.2564	0.5817	0.5995	0.5753	0.5856
GRU4RecF	0.2303	0.2651	0.2303	0.2427	0.4976	0.5631	0.5353	0.5454
Figure 3: Comparison between ground truth title and product titles generated by mT5-small,  $\mathrm{K} = 1$
Results & Observations. The results in Table 6 demonstrate that for the mT5 model, extending the session history length  $(K)$  does not contribute to a performance boost. Furthermore, the simple heuristic, Last Product Title, surpasses the performance of the finetuned language model. This observation highlights two issues: (1) The last product bears significant importance, as the user's next interest is often highly correlated with the most recent product. This correlation becomes
Table 6: BLEU scores in Task 3.
Validation	Test
mT5-small, K = 1	0.2499	0.2265
mT5-small, K = 2	0.2401	0.2176
mT5-small, K = 3	0.2366	0.2142
mT5-base, K = 1	0.2477	0.2251
Last Product Title	0.2500	0.2677
particularly evident when the product title serves as the prediction target. For instance, the next product title can have a large overlap with the previous product (e.g., same product type or brand). (2) The mT5 model did not function as effectively as expected, potentially due to a mismatch between the pre-training and downstream texts. Thus, even an escalation in the model's size (from mT5-small to mT5-base) did not result in a performance gain. This suggests the necessity for a domain-specific language model to leverage text information effectively. Moreover, we illustrate some examples of generated results compared with the ground truth title in Figure 3 to intuitively indicate the quality of the generated examples. We note that the generated titles look generally good, nonetheless, the generated ones still lack details, especially numbers, e.g.,  $(180\mathrm{x}200\mathrm{cm})$  in example 2. In addition, we anticipate that larger language models such as GPT-3/GPT-4 [45, 46] could achieve superior performance in this task, given their exposure to a more diverse corpus during pre-training. We reserve these explorations for future work.
In this section, we discuss the new opportunities that the proposed Amazon-M2 dataset can introduce to various research topics in academic study and industrial applications. Due to the space limit, we only list five topics in this section while leaving others in Appendix B.4.
Pre-Training & Transfer Learning. Our dataset comprises session data from six locales with different languages, three of which have more data than the remaining three. This property presents a unique opportunity to investigate pre-training and transfer learning techniques for recommendation algorithms [14, 47, 48], as demonstrated in Task 2. Due to the inherent sparsity of session data [49], building accurate models that can capture users' preferences and interests based on their limited interactions can be challenging, particularly when the number of sessions is small. One solution to address the data sparsity problem is to transfer knowledge from other domains. Our dataset is advantageous in this regard because it provides data from sources (locales/languages), which enables the transfer of user interactions from other locales, potentially alleviating the data sparsity issue.
Large Language Models in Recommendation. The rich textual features in Amazon-M2 give researchers a chance for leveraging large language models (LLMs) [50, 46, 45, 51] in providing recommendations. For example, there has been a growing emphasis on evaluating the capability of ChatGPT [45] in the context of recommender systems [52, 53, 54, 55, 56, 57, 58, 59, 60]. The proposed Amazon-M2 presents an excellent opportunity for researchers to explore and experiment with their ideas involving LLMs and recommendation.
Text-to-Text Generation. Our dataset presents ample opportunities for exploring natural language processing tasks, including text-to-text generation. In Task 3, we introduced a novel task that involves predicting the title of the next product, which requires the recommender system to generate textual results, rather than simply predicting a single product ID. This new task resembles to Question Answering and allows us to leverage advanced text-to-text generation techniques [61], such as the popular GPT models [62, 51], which have demonstrated significant success in this area.
Cross-Linguual Entity Alignment. Entity alignment is a task that aims to find equivalent entities across different sources [63, 64, 65]. Our dataset provides a good resource for evaluating various entity alignment algorithms, as it contains entities (products) from different locales and languages. Specifically, the dataset can be used to test the performance of entity alignment algorithms in cross-lingual settings, where the entities to be aligned are expressed in different languages.
Graph Neural Networks. As powerful learning tools for graph data, graph neural networks (GNNs) [66, 67, 68] have tremendously facilitated a wide range of graph-related applications including recommendation [13, 4], computation biology [69, 70], and drug discovery [71]. Our dataset provides rich user-item interaction data which can be naturally represented as graph-structured data. Thus, it is a good tool for evaluating various GNNs and rethinking their development in the scenarios of recommendation and transfer learning. In addition, the abundant textual and structural data in Amazon-M2 provides a valuable testing platform for the recent surge of evaluating LLMs for graph-related tasks [72, 73, 74, 75, 76].
Item Cold-Start Problem. The item cold-start problem [77, 38] is a well-known challenge in recommender systems, arising when a new item is introduced into the system, and there is insufficient data available to provide accurate recommendations. However, our dataset provides rich items attributes including detailed textual descriptions, which offers the potential to obtain excellent semantic embeddings for newly added items, even in the absence of user interactions. This allows for the development of a more effective recommender system that places greater emphasis on the semantic information of the items, rather than solely relying on the user's past interactions. Therefore, by leveraging this dataset, we can overcome the cold-start problem and deliver better diverse recommendations, enhancing the user experience.
Data Imputation. Research on deep learning requires large amounts of complete data, but obtaining such data is almost impossible in the real world due to various reasons such as damages to devices, data collection failures, and lost records. Data imputation [78] is a technique used to fill in missing values in the data, which is crucial for data analysis and model development. Our dataset provides ample opportunities for data imputation, as it contains entities with various attributes. By exploring different imputation methods and evaluating their performance on our dataset, we can identify the most effective approach for our specific needs.
This paper presents the Amazon-M2 dataset, which aims to facilitate the development of recommendation strategies that can capture language and location differences. Amazon-M2 provides rich semantic attributes including textual features, encompasses a large number of sessions and products, and covers multiple locales and languages. These qualities grant machine learning researchers and practitioners considerable flexibility to explore the data comprehensively and evaluate their models thoroughly. In this paper, we introduce the details of the dataset and provide a systematic analysis of its properties. Furthermore, through empirical studies, we highlight the limitations of existing session-based recommendation algorithms and emphasize the immense potential for developing new algorithms with Amazon-M2. We firmly believe that this novel dataset provides unique contributions to the pertinent fields of recommender systems, transfer learning, and large language models. The detailed discussion on broader impact and limitation can be found in Appendix D.
7 Acknowledgement
We want to thank AWS and amazon science for their support. We also appreciate the help from Yupeng Hou, in adding our dataset into RecBole.
[76] Jiawei Zhang. Graph-toolformer: To empower llms with graph reasoning ability via prompt augmented by chatgpt. arXiv preprint arXiv:2304.11116, 2023.
[77] Hoyeop Lee, Jinbae Im, Seongwon Jang, Hyunsouk Cho, and Sehee Chung. Melu: Meta-learned user preference estimator for cold-start recommendation. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 1073–1082, 2019.
[78] Stef Van Buuren. Flexible imputation of missing data. CRC press, 2018.
[79] Wayne Xin Zhao, Shanlei Mu, Yupeng Hou, Zihan Lin, Yushuo Chen, Xingyu Pan, Kaiyuan Li, Yujie Lu, Hui Wang, Changxin Tian, et al. Recbole: Towards a unified, comprehensive and efficient framework for recommendation algorithms. In proceedings of the 30th acm international conference on information & knowledge management, pages 4653-4664, 2021.
Data Splits. Following the setting in our KDDCUP competition<sup>3</sup>, the Amazon-M2 dataset contains three splits: training, phase-1 test and phase-2 test. For the purpose of model training and selection, we further split the original training set into  $90\%$  sessions for development (used for training), and  $10\%$  sessions for validation. Note that the numbers in Tables 3, 4, and 5 of the main content are for validation performance. Without specific mention, the test set mentioned in the main content indicates the phase-1 test. Due to the page limitation of main content, we defer the performances on the phase-2 test set to the appendix.
Hyperparameter Settings. The hyper-parameters of all the models are tuned based on the performance of the validation set. For Task 1 and Task 2, we follow the suggested hyper-parameter range to search for the optimal settings provided by Recbole toolkits [79]. By default, we only use the product ID to train the models since most of the popular session-based recommendation baselines are ID-based methods. We leave the exploration of other rich attributes such as price, brand, and description as future work. Specifically, the search ranges for different models are outlined below:
- GRU4REC++ : learning_rate [0.01,0.001,0.0001], dropout_prob: [0.0,0.1,0.2,0.3,0.4,0.5],
num_layers: [1,2,3], hidden_size: [128].
- NARM: learning_rate: [0.01,0.001,0.0001], hidden_size: [128], n_layers: [1,2], dropout_prob: ['[0.25,0.5]', '[0.2,0.2]', '[0.1,0.2]'].
- STAMP: learning_rate: [0.01, 0.001, 0.0001]
- SRGNN: learning_rate: [0.01, 0.001, 0.0001], step: [1, 2].
- CORE: learning_rate: [0.001, 0.0001], n_layers: [1, 2], hidden_dropout_prob: [0.2, 0.5], attn_dropout_prob: [0.2, 0.5].
- GRU4RECF: learning_rate: [0.01, 0.001, 0.0001], num_layers: [1, 2].
- SRGNNF: learning_rate: [0.01, 0.001, 0.0001], step: [1, 2].
- MGS: learning_rate: [0.01, 0.001, 0.0001]
For Task 3, we adopted the code example provided by HuggingFace which supports text-to-text generation. Moreover, we tune the following hyperparameters in mT5: weight Decay in the range of  $\{0,1\mathrm{e} - 8\}$ , learning_rate in the range of  $\{2\mathrm{e} - 5,2\mathrm{e} - 4\}$ , num_beams in the range of  $\{1,5\}$ . Additionally, we set the training batch size to 12, and the number of training epochs to 10.
Hardware and Software Configurations. We perform experiments on one server with 8 NVIDIA RTX A6000 (48 GB) and 128 AMD EPYC 7513 32-Core Processor @ 3.4 GHZ. The operating system is Ubuntu 20.04.1.
Metric details: Mean Reciprocal Rank(MRR)@K is a metric used in information retrieval and recommendation systems to measure the effectiveness of a model in providing relevant results. MRR is computed with the following two steps: (1) calculate the reciprocal rank. The reciprocal rank is the inverse of the position at which the first relevant item appears in the list of recommendations. If no relevant item is found in the list, the reciprocal rank is considered 0. (2) average of the reciprocal ranks of the first relevant item for each session.
$$
\operatorname {M R R} @ K = \frac {1}{N} \sum_ {t \in T} \frac {1}{\operatorname {R a n k} (t)}, \tag {1}
$$
where  $\mathrm{Rank}(t)$  is the rank of the ground truth on the top  $K$  result ranking list of test session  $t$ , and if there is no ground truth on the top  $K$  ranking list, then we would set  $\frac{1}{\mathrm{Rank}(t)} = 0$ . MRR values range from 0 to 1, with higher values indicating better performance. A perfect MRR score of 1 means that the model always places the first relevant item at the top of the recommendation list. An MRR score of 0 implies that no relevant items were found in the list of recommendations for any of the queries or users.
<sup>3</sup>https://www.iacrowd.com/challenges/amazon-kdd-cup-23-multilingual-recommendation-challenge<sup>4</sup>https://github.com/huggingface/transformers/blob/main/examples/pytorch/summarization/run_summarization_no Trainer.py
Hit@K is the proportion of the correct test product within a top k position in the recommended ranking list, defined as:
$$
\mathrm {H i t} @ \mathrm {K} = \frac {n _ {\mathrm {h i t}}}{N} \tag {2}
$$
where  $N$  denotes the number of test sessions.  $n_{\mathrm{hit}}$  is the number of test sessions with the next product in the top  $K$  of the ranked list.
Normalized Discounted Cumulative Gain(NDCG@K) quantifies the effectiveness of ranked lists by considering both the relevance of items and their positions within the list up to a specified depth  $K$ . This metric facilitates fair comparisons of ranking algorithms across different datasets and scenarios, as it normalizes the Discounted Cumulative Gain (DCG) score to a standardized range, typically between 0 and 1.
B More Dataset Details
B.1 Dataset Collection
The Amazon-M2 dataset is a collection of anonymous user session data and product data from the Amazon platform. Each session represents a list of products that a user interacted with during a 30-minute active window. Note that the product list in each session is arranged in chronological order, with each product represented by its ASIN number. Users can search for Amazon products using their ASIN numbers<sup>5</sup> and obtain the corresponding product attributes. We include the following product attributes:
- ASIN (id): ASIN stands for Amazon Standard Identification Number. It is a unique identifier assigned to each product listed on Amazon's marketplace, allowing for easy identification and tracking.
- Locale: Locale refers to the specific geographical or regional settings and preferences that determine how information is presented to users on Amazon's platform.
- Title: The Title attribute represents the name or title given to a product, book, or creative work. It provides a concise and identifiable name that customers can use to search for or refer to the item.
- Brand: The Brand attribute represents the manufacturer or company that produces the product. It provides information about the brand reputation and can influence a customer's purchasing decision based on brand loyalty or recognition.
- Size: Size indicates the dimensions or physical size of the product. It is useful for customers who need to ensure that the item will fit their specific requirements or space constraints.
- Model: The Model attribute refers to a specific model or version of a product. It helps differentiate between different variations or versions of the same product from the same brand.
- Material Type: Material Type indicates the composition or main material used in the construction of the product. It provides information about the product's primary material, such as metal, plastic, wood, etc.
Color Text: Color Text describes the color or color variation of the product. It provides information about the product's appearance and helps customers choose items that match their color preferences.
- Author: Author refers to the individual or individuals who have written a book or authored written content. It helps customers identify the creator of the work and plays a significant role in book purchasing decisions.
- Bullet Description (desc): Bullet Description is a concise and brief description of the product's key features, benefits, or selling points. It highlights the most important information about the item in a clear and easily scannable format.
The dataset spans a period of 3 weeks, with the first 2 weeks designated as the training set and the remaining week as the test set. To enhance evaluation, the test set is further randomly divided into two equal subsets, i.e., phase-1 test and phase-2 test.
B.2 Additional data analysis
The additional analysis of the session length and the repeat pattern on each individual locale can be found in Figure 4 and 5, respectively. We can still observe evident long-tail distributions for both