SlowGuess/NAACL_2022_OCR huggingface dataset:
Table 3 shows the results on Entity Linking. Similar to Query Search, this task also requires the models to tie text semantics (queries/product attributes) to a customer session, which requires a deeper understanding of customer preferences. It is easier than Query Search as product attributes are of lower variance. However, the product attributes that the customer prefer rely more on session information, as they may have been reflected in the past search queries and viewed products. In this task, CERES outperforms domain-specific models and general-domain models by averagely  $9\%$  in MAP@1 and  $6\%$  in MAP@32 and MAP@64.
In this section we present further studies to understand: 1) the effect of training data sizes in the downstream task; 2) the effects of different components in CERES for both the pretraining and finetuning stages. following observations:
CERES is highly effective when training data are scarce. We compare CERES with two strongest baselines (BERT, and Product-BERT) when the training sample size varies. Figure 5 shows the MAP@64 scores of these methods on Product Search and Query Search when training size varies. Clearly, the advantage of CERES is greater when training data is extremely small. With a training size of 300, CERES can achieve a decent performance of about  $37.55\%$  in Product Search and  $36.37\%$  in Query Search, while the baseline models cannot be trained sufficiently with such small-sized data. This shows that the efficient utilization of session-level information in pretraining and fine-tuning stages make the model more data efficient than other pretrained models.
(a) Product Search
(b) Query Search
Figure 5: Effect of sample size on Product Search and Query Search. x-axis represents the training data size and y-axis represents MAP@64.
Method	map@1	recall@1	map@32	recall@32	map@64	recall@64
SR-GNN	36.313	37.284	50.683	99.592	60.413	99.689
NISER+	37.193	38.144	52.855	98.293	62.371	99.111
MERLIN	89.744	90.166	93.067	98.98	93.075	99.33
BERT	85.096	84.688	89.172	99.082	89.18	99.301
RoBERTa	79.647	78.963	83.207	95.396	83.25	97.494
Electra	85.897	86.32	89.841	99.344	89.845	99.519
Product-Bert	91.026	91.71	93.856	99.563	93.856	99.563
SQSP-Bert	85.577	85.795	90.049	99.038	90.057	99.301
CERES	92.628	93.094	94.848	99.551	94.853	99.65
Table 1: The performance of different methods for Product Search, after fine-tuning with 30,000 training sessions.
Method	map@1	recall@1	map@32	recall@32	map@64	recall@64
BERT	47.276	47.627	60.143	92.553	60.214	95.417
RoBERTa	26.603	26.323	37.722	74.468	37.839	80.196
Electra	32.853	32.788	47.512	90.426	47.632	95.663
Product-BERT	52.724	52.973	66.035	95.99	66.065	97.463
SQSP-BERT	45.833	46.29	60.195	92.881	60.26	95.499
CERES	59.936	60.284	72.329	97.463	72.331	97.627
Table 2: The performance of different methods for Query Search, after fine-tuning with 30,000 training sessions.
Graph-Conditioned Transformer is Vital to Pretraining. Without the Graph-Conditioned Transformer in pretraining, our model is essentially the same as domain-specific baselines, such as Product-BERT, which are trained on session data but only with intra-item text signals. While SQSP-BERT has access to session-level information when maximizing the masked language modeling objective, the lack of a dedicated module for GMLM results in worse performance, as shown in the main experiment results.
We could train the Graph-Conditioned Transformer from scratch in the finetuning stage. We present a model called CERES w/o Pretrain, which attaches the Graph-Conditioned Session Transformer to Product-BERT as the Item Transformer Encoder. As shown in Figure 6, this ablation method achieves MAP@64 scores of  $89.341\%$  in Product Search,  $64.890\%$  in Query Search, and  $74.031\%$  in Entity Linking, which are below Product-BERT. This shows that the pretraining stage of the Graph-Conditioned Transformer is necessary to facilitate its ability to aggregate and propagate session-level information for downstream tasks.
Graph-Conditioned Transformer Improves Item-level Embeddings. We also present CERES w/o Cond, which has the same pretrained model as CERES, but only uses the Item Transformer Encoder in the finetuning stage. The Item Transformer Encoder is used to compute session item embeddings that contain only item-level information, and then takes the average of these embeddings as session embedding. As shown in Figure 6, CERES w/o Cond achieves  $94.741\%$ ,  $72.175\%$ , and  $81.03\%$  respectively in Product
Figure 6: Results on three tasks on ablation models. y-axis represents MAP@64. CERES w/o Cond is CERES without the Graph-Conditioned Transformer in the finetuning stage. CERES w/o Pretrain is CERES without pretraining the Graph-Conditioned Transformer, but instead training it from scratch in the finetuning stage. CERES w/o GNN is CERES pretrained without the GNN module.
Search, Query Search, and Entity Linking, observing a drop of  $0.1\%$  to  $0.2\%$  in performance compared with CERES. The performance drop is minor and CERES w/o Cond still outperforms baseline pretrained language models. Hence, the Graph-Conditioned Transformer in the pretraining stage helps the Item Transformer Encoder to learn better item-level embeddings that can be used for more effective leveraging of session information in the downstream tasks.
Graph Neural Networks Improve Representation of Sessions. In CERES w/o GNN, we pretrain a CERES model without a Graph Neural Network. Specifically, CERES w/o GNN skips the neighborhood information aggregation for items, and uses item-level embeddings obtained by the Item Transformer Encoder directly as latent conditioning tokens. We train and finetune this model with the same setup as CERES. Without GNN, the model's performance is consistently lower than CERES,
Method	map@1	recall@1	map@32	recall@32	map@64	recall@64
BERT	55.609	55.353	66.386	90.511	66.481	95.073
RoBERTa	66.506	65.754	74.516	93.248	74.561	95.438
Electra	62.321	62.365	62.985	68.296	63.122	74.318
Product-Bert	66.827	66.393	74.611	94.404	74.641	96.046
SQSP-Bert	63.942	64.872	72.232	91.241	72.307	94.891
CERES	75.481	75.456	81.121	95.255	81.16	96.898
Table 3: The performance of different methods for Entity Linking, after fine-tuning with 30,000 training sessions.
achieving  $93.453\%$ ,  $71.231\%$ ,  $80.26\%$  MAP@64 in three downstream tasks, observing a  $1.13\%$  performance drop. This shows that GNN's aggregation of information can help item-level embeddings encode more session-level information, improving performance in downstream tasks.
Model Efficiency. CERES has additional few GNN and Transformer layers attached to the end of the model. The additional layers bring  $\sim 20\%$  additional inference time compared to standard BERT with 12 layers and 768 hidden size.
Pretrained language models such as BERT (Devlin et al., 2018), BART (Lewis et al., 2019), ELECTRA (Clark et al., 2020), RoBERTa (Liu et al., 2019) have pushed the frontiers of many NLP tasks by large margins. Their effectiveness and efficiency in parallelism have made them popular and general-purpose language encoders for many text-rich applications. However, they are not designed to model relational and graph data, and hence are not the best fit for e-commerce session data.
Researchers have also sought to enhance text representations in pretrained models with knowledge graphs (Shen et al., 2020; Liu et al., 2020; Yao et al., 2019; Sun et al., 2020, 2021). While these models consider a knowledge graph structure on top of text data, they generally use entities or relations in knowledge graphs to enhance text representations, but cannot encode arbitrary graph structures. This is not sufficient in session-related applications as session structures are ignored.
Many works have been proposed to learn pretrained graph neural networks. Initially, methods were proposed for domain-specific graph pretraining (Hu et al., 2019a,b; Shang et al., 2019). However, they rely on pre-extracted domain-specific node-level features, and cannot be extended to either session data or text data as nodes. Recently, many works have been proposed to pretrain on general graph structure (Hu et al., 2020; You et al., 2020; Qiu et al., 2020a). However, they cannot encode the semantics of text data as nodes.
Contextual information in sessions have been
shown beneficial to various related recommendation tasks, such as product recommendation (Wu et al., 2019b; Dehghani et al., 2017; Jannach and Ludewig, 2017; Gupta et al., 2019) and query rewriting (Li et al., 2017; Cucerzan and White, 2007). Many existing session-based recommendation methods seek to exploit the transitions between items (Yap et al., 2012; Rendle et al., 2010; Wang et al., 2018; Li et al., 2017) and considering sessions as graphs (Xu et al., 2019; Ruihong et al., 2021; Wang et al., 2020).
6 Limitations and Risks
This paper limits the application of CERES to session data with text descriptions. CERES has the potential of being a universal pretraining framework for arbitrary heterogeneous data. For example, sessions can include product images and customer reviews for more informative multimodal graphs. We leave this extension for future work.
Session data are personalized experience for customers and could cause privacy issues if data are not properly anonymized. In application, the model should be used to avoid exploitation or leakage of customers personal profiles and preferences.
We proposed a pretraining framework, CERES, for learning representations for semi-structured e-commerce sessions. We are the first to jointly model intra-item text and inter-item relations in session graphs with an end-to-end pretraining framework. By modeling Graph-Conditioned Masked Language Modeling, our model is encouraged to learn high-quality representations for both intra-item and inter-item information during its pretraining on massive unlabeled session graphs. Furthermore, as a generic session encoder, our model enabled effective leverage of session information in downstream tasks. We conducted extensive experiments and ablation studies on CERES in comparison to state-of-the-art pretrained models and recommendation systems. Experiments show that CERES can produce higher quality text representations as well as better leverage of session
graph structure, which are important to many e-commerce related tasks, including product search, query search, and query understanding.
This work was supported in part by NSF IIS-2008334, IIS-2106961, CAREER IIS-2144338, and ONR MURI N00014-17-1-2656.
Kevin Clark, Minh-Thang Luong, Quoc V Le, and Christopher D Manning. 2020. Electra: Pre-training text encoders as discriminators rather than generators. arXiv preprint arXiv:2003.10555.
Silviu Cucerzan and Ryen W White. 2007. Query suggestion based on user landing pages. In Proceedings of the 30th annual international ACM SIGIR conference on Research and development in information retrieval, pages 875-876.
Mostafa Dehghani, Sascha Rothe, Enrique Alfonseca, and Pascal Fleury. 2017. Learning to attend, copy, and generate for session-based query suggestion. In Proceedings of the 2017 ACM on Conference on Information and Knowledge Management, pages 1747-1756.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805.
Priyanka Gupta, Diksha Garg, Pankaj Malhotra, Lovekesh Vig, and Gautam M Shroff. 2019. Niser: Normalized item and session representations with graph neural networks. arXiv preprint arXiv:1909.04276.
Weihua Hu, Bowen Liu, Joseph Gomes, Marinka Zitnik, Percy Liang, Vijay Pande, and Jure Leskovec. 2019a. Pre-training graph neural networks. arXiv preprint arXiv:1905.12265.
Weihua Hu, Bowen Liu, Joseph Gomes, Marinka Zitnik, Percy Liang, Vijay Pande, and Jure Leskovec. 2019b. Strategies for pre-training graph neural networks. arXiv preprint arXiv:1905.12265.
Ziniu Hu, Yuxiao Dong, Kuansan Wang, Kai-Wei Chang, and Yizhou Sun. 2020. Gpt-gnn: Generative pre-training of graph neural networks. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 1857-1867.
Dietmar Jannach and Malte Ludewig. 2017. When recurrent neural networks meet the neighborhood for session-based recommendation. In Proceedings of the Eleventh ACM Conference on Recommender Systems, pages 306-310.
Surya Kallumadi, Tracy Holloway King, Shervin Malmasi, and Maarten de Rijke. 2021. Ecom'21: The sigir 2021 workshop on ecommerce. In Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 2685-2688.
Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2019. Albert: A lite bert for self-supervised learning of language representations. arXiv preprint arXiv:1909.11942.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. 2019. Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. arXiv preprint arXiv:1910.13461.
Jing Li, Pengjie Ren, Zhumin Chen, Zhaochun Ren, Tao Lian, and Jun Ma. 2017. Neural attentive session-based recommendation. In Proceedings of the 2017 ACM on Conference on Information and Knowledge Management, pages 1419-1428.
Weijie Liu, Peng Zhou, Zhe Zhao, Zhiruo Wang, Qi Ju, Haotang Deng, and Ping Wang. 2020. K-bert: Enabling language representation with knowledge graph. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 2901-2908.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692.
Bamshad Mobasher, Honghua Dai, Tao Luo, and Miki Nakagawa. 2001. Effective personalization based on association rule discovery from web usage data. In Proceedings of the 3rd international workshop on Web information and data management, pages 9-15.
Jiezhong Qiu, Qibin Chen, Yuxiao Dong, Jing Zhang, Hongxia Yang, Ming Ding, Kuansan Wang, and Jie Tang. 2020a. GCC: Graph contrastive coding for graph neural network pre-training. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 1150-1160.
Ruihong Qiu, Zi Huang, Jingjing Li, and Hongzhi Yin. 2020b. Exploiting cross-session information for session-based recommendation with graph neural networks. ACM Transactions on Information Systems (TOIS), 38(3):1-23.
Steffen Rendle, Christoph Freudenthaler, and Lars Schmidt-Thieme. 2010. Factorizing personalized markov chains for next-basket recommendation. In Proceedings of the 19th international conference on World wide web, pages 811-820.
Qiu Ruihong, Huang Zi, Chen Tong, and Yin Hongzhi. 2021. Exploiting positional information for session-based recommendation. arXiv preprint arXiv:2107.00846.
Junyuan Shang, Tengfei Ma, Cao Xiao, and Jimeng Sun. 2019. Pre-training of graph augmented transformers for medication recommendation. arXiv preprint arXiv:1906.00346.
Tao Shen, Yi Mao, Pengcheng He, Guodong Long, Adam Trischler, and Weizhu Chen. 2020. Exploiting structured knowledge in text via graph-guided representation learning. arXiv preprint arXiv:2004.14224.
Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. 2019. Megatron-lm: Training multi-billion parameter language models using model parallelism. arXiv preprint arXiv:1909.08053.
Yu Sun, Shuohuan Wang, Shikun Feng, Siyu Ding, Chao Pang, Junyuan Shang, Jiaxiang Liu, Xuyi Chen, Yanbin Zhao, Yuxiang Lu, et al. 2021. Ernie 3.0: Large-scale knowledge enhanced pre-training for language understanding and generation. arXiv preprint arXiv:2107.02137.
Yu Sun, Shuohuan Wang, Yukun Li, Shikun Feng, Hao Tian, Hua Wu, and Haifeng Wang. 2020. Ernie 2.0: A continual pre-training framework for language understanding. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 8968-8975.
Shoujin Wang, Liang Hu, Longbing Cao, Xiaoshui Huang, Defu Lian, and Wei Liu. 2018. Attention-based transactional context embedding for next-item recommendation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32.
Ziyang Wang, Wei Wei, Gao Cong, Xiao-Li Li, XianLing Mao, and Minghui Qiu. 2020. Global context enhanced graph neural networks for session-based recommendation. In Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 169-178.
Shu Wu, Yuyuan Tang, Yanqiao Zhu, Liang Wang, Xing Xie, and Tieniu Tan. 2019a. Session-based recommendation with graph neural networks. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 346-353.
Shu Wu, Yuyuan Tang, Yanqiao Zhu, Liang Wang, Xing Xie, and Tieniu Tan. 2019b. Session-based recommendation with graph neural networks. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 346-353.
Chengfeng Xu, Pengpeng Zhao, Yanchi Liu, Victor S Sheng, Jiajie Xu, Fuzhen Zhuang, Junhua Fang, and Xiaofang Zhou. 2019. Graph contextualized self-attention network for session-based recommendation. In *IJCAI*, volume 19, pages 3940–3946.
Liang Yao, Chengsheng Mao, and Yuan Luo. 2019. Kgbert: Bert for knowledge graph completion. arXiv preprint arXiv:1909.03193.
Ghim-Eng Yap, Xiao-Li Li, and S Yu Philip. 2012. Effective next-items recommendation via personalized sequential pattern mining. In International conference on database systems for advanced applications, pages 48-64. Springer.
Jiaxuan You, Yichen Wang, Aditya Pal, Pong Eksom-batchai, Chuck Rosenberg, and Jure Leskovec. 2019. Hierarchical temporal convolutional networks for dynamic recommender systems. In The world wide web conference, pages 2236-2246.
Yuning You, Tianlong Chen, Yongduo Sui, Ting Chen, Zhangyang Wang, and Yang Shen. 2020. Graph contrastive learning with augmentations. Advances in Neural Information Processing Systems, 33:5812-5823.
Hanchu Zhang, Leonhard Hennig, Christoph Alt, Changjian Hu, Yao Meng, and Chao Wang. 2020. Bootstrapping named entity recognition in e-commerce with positive unlabeled learning. arXiv preprint arXiv:2005.11075.
Attribute	Value
Title	Pour-over Coffee Maker
Bullet Description	Just coffee maker.
Color	N/A
Brand	Chemex
Manufacturer	Chemex
Product Type	Coffee Maker
Table 4: Example Product Table. Each product is guaranteed to have a title. Most products have bullet descriptions, which can be split into multiple entries. Products could have other attributes, such as color, brand, product type, etc. as well.
A Details on Session Data
A.1 Product Attributes.
A product is represented with a table of attributes. Each product is guaranteed to have a product title and bullet description. In this paper, we regard the product title as the representative sequence of the product, called "product sequence". A product may have additional attributes, such as product type, color, brand, and manufacturer, depending on specific products.
A.2 Alternative Pretraining Corpora
In this section we introduce alternative pretraining corpora that encode information in a session, including products and queries, but not treating sessions as a whole.
A.2.1 Product Corpus
In this corpus, we gathered all product information that appeared in the sessions from August 2020 to September 2020. Each product will have descriptions such as product title and bullet description, and other attributes like entity type, product type, manufacturer, etc. Particularly, bullet description is composed of several lines of descriptive facts about the product. All products without titles are removed. Each of the remaining product forms a paragraph, where the product title comes as the first sentence, followed by the entries of bullet descriptions each as a sentence, and product attributes.
An example document in this corpora is as follows:
A.2.2 Single-Query Single-Product (SQSP) Corpus
In this corpus, we treat each session as a document and each query-product pair as a sentence. A query-product pair in the document are the pairs of queries and products that are either viewed or clicked with the given queries. A query-product pair looks like the follows:
where the first [SEARCH] special token indicates a field of query keywords, and [TITLE] indicates fields of product information starting with product tittles. In this corpus, we model the one-to-one relation between queries and products.
A.2.3 Session Corpus
In this corpus, we treat each session as a document and sequentially put text representations of items in a session to the document with special tokens indicating the fields of items. An example document looks like the follows:
In this example, the customer first attempted to search with keywords 1 and then modified the keywords to keywords 2. The customer then clicked on product 1. At last, the customer modified his search to keywords 3 and purchased product 2. In this corpus, session information is present in a document, but the specific relations between elements are not specified. The comparison of different datasets are in Table 5.
A.3 Alternative Pretraining Methods
We introduce the alternative pretraining models.
- Product-Bert. It is pretrained on the Product Corpus. Specifically, we treat each product in the Product Corpus as an article. Product titles is always the first sentence, followed by paragraphs of bullet descriptions, which can contain multiple sentences. Then, each additional product attribute is a sentence added after the bullet descriptions.
Corpus	Product Info	Query Info	Relational	Session Context
Product	✓	X	X	X
SQSP	✓	✓	✓	X
Session-Corpus	✓	✓	X	✓
Session-Graph	✓	✓	✓	✓
Table 5: Comparison of different pretraining dataset. Product Corpus has access only to product information. SQSP models on the queries and query-product relations, without access to session context. Session Corpus has access to contextual information in a session, but does not model on relations between objects. Session-Graph has access to all information and models on the relational nature of nodes in the session graph.
Product Bert is trained for 300,000 steps, with a 12-layer transformer with a batch size of 6144 and peak learning rate of 1e-3,  $1\%$  linear warm-up steps, and  $1e - 2$  linear weight decay to a minimum learning rate of 1e-5.
- SQSP-Bert. It is pretrained on SQSP Corpus. The SQSP Bert uses the same Transformer backbone as Product Bert. Given each query-product pair, SQSP feeds the text pair sequence to the Transformer for token embeddings for masked language modeling loss. In addition to language modeling, for each query-product pair, we sample a random product for the query as a negative query-product pair. The text pair sequence of the negative sample is also fed to the Transformer. Then, a discriminator is trained in the pretraining stage to distinguish the ground-truth query-product pairs and randomly sampled pairs. The discriminator's classification loss should serve as a contrastive loss.
SQSP Bert is trained with the same configuration of Product Bert.
B Details on Evaluation Metrics
Mean Average Precision. Suppose that for a session,  $m$  items are relevant and  $N$  items are retrieved by the model, the Average Precision (AP) of a session is defined as
$$
\mathrm {A P} @ \mathrm {N} = \frac {1}{\operatorname* {m i n} (m , N)} \sum_ {k = 1} ^ {N} P (k) \mathrm {r e l} (k), \tag {4}
$$
where  $P(k)$  is the precision of the top  $k$  retrieved items, and  $rel(k)$  is an indicator function of whether the  $k$ th item is relevant. As we have at most one relevant item for each session, the above metric reduces to  $\frac{1}{r}$ , where  $r$  is the rank of the relevant item in the retrieved list, and  $k = \infty$  when the relevant item is not retrieved. MAP@N averages
AP@N over all sessions,
$$
\mathrm {M A P} @ \mathrm {N} = \frac {1}{| \mathcal {S} |} \sum_ {s \in \mathcal {S}} \frac {1}{r _ {s}} \tag {5}
$$
where  $r_s$  is the rank of the relevant item for a specific session  $s$ . MAP in this case is equivalent to MRR.