59 60 al 62 a 64 65 66 67 65 69 70 71 ... 12 13 14
ABSTRACT
15 10 Masato Hashimoto NTT DOCOMO, INC. Tokyo, Japan masato.hashimoto.px@nttdocomo.com Sho Maeoki NTT DOCOMO, INC. Tokyo, Japan syou.maeoki.rz@nttdocomo.com and Data Mining, August 6–10, 2022, Long Beach, CA, USA. ACM, New York, NY, USA, 4 pages. https://doi.org/10.1145/nnnnnn.nnnnnn 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 19 92 93 94 95 96 77 95 99 100 101 102 103 104 105 106 107 103 109 110 111 112112 113 114 115 116 54 55 56 57 58 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38
CCS CONCEPTS
• Applied computing → Electronic commerce .
KEYWORDS
42 43 44 KDD Cup, Multilingual Session Recommendation, Text Generation ACM Reference Format: Yusuke Fukushima, Keiichi Ochiai, Yoshitaka Inoue, Masato Hashimoto, and Sho Maeoki. 2023. Word Filtering Approach for Next Product Title Gen- eration. In KDDCUP *23: ACM SIGKDD Conference on Knowledge Discovery 49 50 53 sa s1 Yoshitaka Inoue DOCOMO Innovations, Inc. Sunnyvale, CA , USA yoshitaka.inoue@docomoinnovations.com Keiichi Ochiai" NTT DOCOMO, INC. Tokyo, Japan ochiaike@nttdocomo.com Yusuke Fukushima" NTT DOCOMO, INC. Tokyo, Japan yuusuke.fukushima.fw@nttdocomo.com 45 46 az 48 *Equal contribution al 2 Session-based Recommendations. Session-based recommenda- tion is a research field receiving attention recently [11]. According to survey paper [11], a unique characteristic of session-based rec- ommendations is to capture dynamic user preferences in the short- term compared to existing recommendations such as collaborative filtering and content-based recommendation, which usually model
RELATED WORK
1 With the proliferation of e-commerce, e-commerce platform providers are trying to improve the shopping experience of their users. For example, improvements to the user experience are expected if a platform can recommend products that match the user interests or present product titles based on a series of past product brows- ing histories (referred to as session data). Although session-based recommendations have been studied for some domains such as e- commerce products and videos [4], limitations still remain such as the language of the data is limited and providing a relevant product title, which is important information as a factor leading to a click. To this end, the organizer of KDD Cup 2023 provided the following three tasks and a dataset referred to as the Amazon Multilingual Multi-locale Session Dataset (Amazon-M2). ●  Task 1: Next Product Recommendation ● Task 2: Next Product Recommendation for Underrepresented Languages/Locales ● Task 3: Next Product Title Generation The provided data are a collection of anonymized customer sessions containing products from six different locales [5]. In this paper, we introduce a solution that placed 6th on the public leaderboard for Task 3 in the competition. The key concept for the solution is that if a word appears only a few times in all product names, it is probably an unnecessary word that can be considered noise. In addition, we introduce a second solution that is equivalent to 10th on the public leaderboard. The key concept for the second solution is to add a prediction for the Bilingual Evaluation Understudy (BLEU) [7] score if the last title is submitted to the first (best) solution, and use this prediction to determine whether to take the best solution as it is or to modify further the title. 40 The rise of e-commerce has prompted e-commerce platform providers to make efforts toward improving the shopping experience of their users. One approach involves the optimization of product recom- mendations tailored to user preferences and the presentation of product titles based on historical session data, which encompass past instances of browsing various products. In considering these, the organizers of KDD Cup 2023 launched a competition focused on session-based recommendations, accompanied by a comprehensive dataset known as the Amazon Multilingual Multi-locale Session Dataset (Amazon-M2). In this paper, we, NTT-DOCOMO-LABS- RED, present our solution that achieved a 6th place ranking on the public leaderboard for Task 3 (Next Product Title Generation) in the competition. The proposed solution centers around the key notion that words appearing infrequently in product titles are likely extraneous and introduce noise to the prediction. By identifying and removing such words, we aim to enhance the overall quality of the solution. Furthermore, we introduce a second solution that at- tained a ranking equivalent to 10th place on the public leaderboard. The core idea behind the second solution involves incorporating a prediction for the BLEU score by submitting the last title to the first solution. This prediction is then utilized to determine whether to retain the best solution as it is or to apply further modifications to the title. 39 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 long-term yet static user preferences. In session-based recommen- dations, Recurrent Neural Networks (RNNs) are often used as the architecture. In addition to RNN-based architectures, Transformer- based models are have been proposed such as in [9] and [2] based on the success of Transformers [10] in natural language processing (NLP) tasks, e.g., next word prediction, language translation, and summarization. However, no existing study has considered the task of next product title prediction. Natural Language Processing. Because the task of next prod- uct title prediction is related to text generation tasks, we briefly review the related studies. A straightforward strategy in recent years is to fine-tune a generative pre-trained language model such as BERT [3] and Generative Pre-trained Transformer (GPT) [8] using task-specific training data [6]. This approach is referred to as "pre-train then fine-tune" [6]. Although pre-trained language models have been used for a variety of tasks, including question answering and task-specific prediction, none have been used to predict product titles. 135
APPROACH
3
136 137 138 139 140 Task Description. Task 3 which we introduce our solution is specifically defined as: Given a session data and the attributes of each product, the goal of this task is to predict the title of the next product that a customer will engage with 1 . 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 1.80 181 182 183 184 185 186 187 189 183 190 191 192 193 194 195 196 197 193 199 200 201 202 203 204 205 206 207 203 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 ● Length of product titles in a session ● Difference in product titles between consecutive products in a session ● Average and standard deviation of product title length in a session We design the second solution as described in Algorithm 1. First, the BLEU score is predicted from session information (line 1). The prediction probability threshold for binary classification is set higher than 0.5 to emphasize precision because we would like to capture more cases where the model predicts that the BLEU score is 0 and the actual BLEU score is also 0. Then, if the predicted BLEU score is 0 and the token length of the last product title in a session as a prediction is less than or equal to TH1en, then the second last product title is set as the (temporary) predicted title (line 3); otherwise, the last product title is set as the (temporary) predicted title (line 5). Finally, we apply the word filtering approach • When deleting a word that appears only in the above n products, delete it with a probability of p. Note that words that appear only in n – 1 or fewer products are deleted with 100 percent probability regardless of the value of p . 2 This is because deleting words that appear in only n – 1 products is clearly less likely to affect the number of n-gram matches between the inference result and the correct answer than deleting words that appear in only n products.
3.2
Second Solution: Combining Word Filtering
Using BLEU Score Prediction Model
First, because we would like to discern a tendency in the case that the last product title in a session is a prediction, we calculate BLEU scores for training data with the last title in a session as a prediction. Figure 1 shows the distribution of BLEU scores if the last product title in a session is used as a prediction. From this figure, we hypothesize that if we can classify whether or not the BLEU score is 0, the prediction performance can be improved for the method that uses the last product title in a session as a prediction. Thus, we build a supervised machine learning (ML) model to predict whether or not the BLEU score is 0 from session information. The label is 1 if the BLEU score in the case that the last product title in a session as a prediction exceeds a certain threshold, and 0 otherwise. We use XGBoost [1] as a supervised ML model and the following session information is used as features: ● Session length
First Solution: Word Filtering Approach
3.1
In Task 3, we can achieve a good score by using the last product title in each session as the inference result. We improved the score by removing infrequently appearing words based on this method. If there is no change in the number of n-grams matched by the correct answer and the inference result, the BLEU score becomes better with fewer tokens in the inference result. Despite the large number of product titles included in the current product data, there are a number of words that appear in only one product title. If we make the assumption that past sessions do not contain the correct product, then the last product in the session, i.e. , the product we use as the inference result, is different from the actual correct product. If this assumption is correct, then the BLEU score will always improve by removing these words used in only one product title from the inference results, since a word that appears in only one product will never be included in both the inference results and the correct product. The next point that needs to be considered is that if the inference result is shorter than the correct answer, the BLEU score will be penalized. When deleting words from the inference result, it is necessary to investigate which language products can be prioritized to delete words from the inference result to improve the score further without penalty. To investigate this, we set the following two parameters for each language. In total, we used 12 parameters. ● Remove words that appear only in n or fewer product titles. 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 253 254 2SS 256 258 259 261 264 265 266 267 268 269
4
270 271
4.1
272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 Input:  Term Frequency Dict Dict , BLEU score prediction model Model, Session information s Output:  Next Product Title pred_title 1: pred_label ← Model(s) 2:  if pred_label = 0 and token_len(s.last_title) <= TH1en then pred_title ← s_second_last_title 3: 4: else pred_title ← s_last_title 5: 6:  end if {// The following process is equal to the first solution} 7:  for word in pred_title do if word appeared in Dict then 8: r ← random() 9: if r < THrandom  then 10: remove the word from pred_title 11: end if 12: end if 13:
described in Section 3.1. Term THrandom in line 10 is the same as THlang described in Section 4.2.
EVALUATION
Evaluation setting
Dataset. The KDD Cup organizer provided a dataset, referred to as Amazon-M2, for this competition [5]. The dataset includes user sessions containing products from six different locales (English, German, Japanese, French, Italian, and Spanish). User sessions com- prise a sequential collection of products with which a user has interacted, organized in chronological order. On the other hand, product attributes include various details such as product title, price in local currency, brand, color, and description. Each product can be identified by a unique Amazon Standard Identification Number (ASIN). The statistics of the provided dataset are given in Table 1. Metric. The evaluation metrics for this task is BLEU score [7]. BLEU is a metric of the similarity of sentences that is often used for performance evaluation in machine translation.
Evaluation of First Solution
Two parameters for each language are described in Section 3.1. For simplification of the notation, parameter ρ is expressed by adding the value of its probability to n – 1. For example, if n = 2 and p = 0.5, 291 292 294 293 295
296 298 299 300 301 302 303 304 305 306 307 303 309 310 311 312 313 314 315 316 317 318 319 321 322 323 325 326 327 328 329 330 331 332 333 334 335 336 337 333 339 340 341 342 343 344 345 346 347 343 Algorithm 1 Procedure of the second solution for a session
it is expressed as n=1.5. As another example, for n = 1 and p = 1, it is represented as n = 1. From the definition of the two parameters, this manner of expression is a natural representation given that the first solution results are equal when n = 1 and ρ = 1, and when n = 2 and ρ = 0. In this section, we define TH1ang = (n − 1) * ρ to represent the above calculation. The parameters for each language are expressed by adding the abbreviation of the language to TH1ang such as TH Jp for a Japanese product. Examples of the submission results in Phase 2 are given in Ta- ble 2. Because of the switch from Phase 1 to Phase 2 during the investigation of the impact of each language, we were not able to conduct a consistent investigation. However, for Spanish, French, and English there was a significant improvement in score using this method. On the other hand, the scores for German and Japanese did not improve even when large numbers of words were removed, and these languages were relatively easy to penalize. We believe that this is a language-specific problem of having many proper nouns and coined words. Italian exhibited intermediate characteristics. Finally, the best conditions were found to be a combination of the following parameters and the best score is 0.26906. ● The values of THES, THFR, THUK = 1.0 ● The value of TH1T = 0.1 ● The values of THjp, THDE = 0.0 4.3 Here, we describe three evaluations for the second solution. More specifically, evaluations of the (1) BLEU score prediction model, (2) offline evaluation, and (3) online evaluation for next product title generation. (1) Evaluation for BLEU score prediction model. First, we ran- domly sampled 300,000 sessions from the published dataset. Then, we split the sampled dataset into training and validation groups. The training data were randomly sampled from 70% of the sampled dataset and the remaining 30% was used as the validation data. We set the label threshold to 0.1. The label ratios were imbalanced: we had 28.4% for label 1 (BLEU score >= 0.1) and 71.6% for label 0 (BLEU score < 0.1) in total. We empirically tuned the threshold for binary classification, and finally set the threshold to 0.8. Under these settings, precision was 0.889 and recall was 0.257 for BLEU score prediction. (2) Offline evaluation. For validation data, we compared BLEU scores among three methods: (1) the last product title in a session as a prediction (Baseline), (2) the first proposed solution, and (3) the second proposed solution. The evaluation results are given in Table 3. The results suggest that the second solution using BLEU score prediction may be effective.
4.2
14:  end for 15: return pred_title 252 262 263 324 320 260 257 297
[1] Tianqi Chen and Carlos Guestrin. 2016. Xgboost: A scalable tree boosting system. In Proceedings of the 22nd acm sigkdd international conference on knowledge discovery and data mining. 785–794. [2] Gabriel de Souza Pereira Moreira, Sara Rabhi, Jeong Min Lee, Ronay Ak, and Even Oldridge. 2021. Transformers4rec: Bridging the gap between nlp and sequential/session-based recommendation. In Proceedings of the 15th ACM Con- ference on Recommender Systems. 143–153. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: [3] Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 (2018). [4] Balázs Hidasi, Alexandros Karatzoglou, Linas Baltrunas, and Domonkos Tikk. 2015. Session-based recommendations with recurrent neural networks. arXiv preprint arXiv:1511.06939 (2015). Wei Jin, Haitao Mao, Zheng Li, Haoming Jiang, Chen Luo, Hongzhi Wen, Haoyu [5] Han, Hanqing Lu, Zhengyang Wang, Ruirui Li, Zhen Li, Monica Xiao Cheng, Rahul Goutam, Haiyang Zhang, Karthik Subbian, Suhang Wang, Yizhou Sun, Jiliang Tang, Bing Yin, and Xianfeng Tang. 2023. Amazon-M2: A Multilingual Multi-locale Shopping Session Dataset for Recommendation and Text Generation. (2023). Bonan Min, Hayley Ross, Elior Sulem, Amir Pouran Ben Veyseh, Thien Huu [6] Nguyen, Oscar Sainz, Eneko Agirre, Ilana Heintz, and Dan Roth. 2021. Recent advances in natural language processing via large pre-trained language models: A survey.  Comput. Surveys (2021). [7] Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. BLEU: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics. 311–318. [8] Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. 2018. Improving language understanding by generative pre-training. (2018). [9] Fei Sun, Jun Liu, Jian Wu, Changhua Pei, Xiao Lin, Wenwu Ou, and Peng Jiang. 2019. BERT4Rec: Sequential recommendation with bidirectional encoder rep- resentations from transformer. In Proceedings of the 28th ACM international conference on information and knowledge management, 1441-1450. [10] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need. Advances in neural information processing systems 30 (2017). [11] Shoujin Wang, Longbing Cao, Yan Wang, Quan Z Sheng, Mehmet A Orgun, and Defu Lian. 2021. A survey on session-based recommender systems. ACM Computing Surveys (CSUR) 54, 7 (2021), 1–38. 357 358 359 360 361 362 363 364 365 (3) Online evaluation We compared BLEU scores among the three methods described above in the online setting by submitting them to the competition platform. The evaluation results are also given in Table 3. The scores are BLEU scores on the public leaderboard. Unfortunately, although the second solution cannot outperform the first solution, the score (0.26813) is equivalent to 10th on the public leaderboard. 423 424 425 426 427 367 368 369 370 371 372 373 374 In this paper, we introduced solutions for Task 3 in the competition. The first solution adopted a word filtering approach based on the key concept that if a word appears only a few times in all product names, it is probably an unnecessary word that can be considered noise. The first solution placed 6th on the public leaderboard. The second solution added a BLEU score prediction model to try to improve the prediction performance. The performance of the second solution was equivalent to 10th on the public leaderboard. 428 42 430 431 432 433 434 435 375 349 407 350 403 351 409 352 410 353 411 354 412 355 413 356 414 415 416 417 418 419 420 421 422 366 376 377 378 436 437 379 380 438 439 381 440 382 383 441 384 442 443 385 444 386 445 387 388 446 447 389 390 443 449 391 450 392 393 451 394 452 453 395 396 454 397 455 456 398 457 399 400 458 459 401 460 402 403 461 462 404 463 405 406 464