SlowGuess/NAACL_2022_OCR huggingface dataset:
Rui Feng†, Chen Luo‡, Qingyu Yin‡, Bing Yin‡, Tuo Zhao† and Chao Zhang†
$^{\dagger}$ Georgia Institute of Technology
$\ddagger$  Amazon Inc.
{rfeng, tourzhao, chaozhang}@gatech.edu
{cheluo,qingyy, alexbyin}@amazon.com
User sessions empower many search and recommendation tasks on a daily basis. Such session data are semi-structured, which encode heterogeneous relations between queries and products, and each item is described by the unstructured text. Despite recent advances in self-supervised learning for text or graphs, there lack of self-supervised learning models that can effectively capture both intra-item semantics and inter-item interactions for semi-structured sessions. To fill this gap, we propose CERES, a graph-based transformer model for semi-structured session data. CERES learns representations that capture both inter- and intra-item semantics with (1) a graph-conditioned masked language pretraining task that jointly learns from item text and item-item relations; and (2) a graph-conditioned transformer architecture that propagates inter-item contexts to item-level representations. We pretrained CERES using  $\sim 468$  million Amazon sessions and find that CERES outperforms strong pretraining baselines by up to  $9\%$  in three session search and entity linking tasks.
User sessions are ubiquitous in online e-commerce stores. An e-commerce session contains customer interactions with the platform in a continuous period. Within one session, the customer can issue multiple queries and take various actions on the retrieved products for these queries, such as clicking, adding to cart, and purchasing. Sessions are important in many e-commerce applications, e.g., product recommendation (Wu et al., 2019a), query recommendation (Cucerzan and White, 2007), and query understanding (Zhang et al., 2020).
This paper considers sessions as semi-structured data, as illustrated in Figure 1. At the higher level, sessions are heterogeneous graphs that contain interactions between items. At the lower level, each
Figure 1: Illustration of a customer session. A session consists of two types of items: queries and products. The customer searched for 3 keywords sequentially and interacted with the products returned by the search engine.
graph node has unstructured text descriptions: we can describe queries by search keywords and products by titles, attributes, customer reviews, and other descriptors. Our goal is to simultaneously encode both the graph and text aspects of the session data to understand customer preferences and intents in a session context.
Pretraining on semi-structured session data remains an open problem. First, existing works on learning from session data usually treat a session as a sequence or a graph (Xu et al., 2019; You et al., 2019; Qiu et al., 2020b). While they can model inter-item relations, they do not capture the rich intra-item semantics when text descriptions are available. Furthermore, these models are usually large neural networks that require massive labeled data to train from scratch. Another line of research utilizes large-scale pretrained language models (Lan et al., 2019; Liu et al., 2019; Clark et al., 2020) as text encoders for session items. However, they fail to model the relational graph structure. Several works attempt to improve language models with a graph-structured knowledge base, such as in (Liu et al., 2020; Yao et al., 2019; Shen et al., 2020). While adjusting the semantics of entities according to the knowledge graph, they fail to encode general graph structures in sessions.
We propose CERES (Graph Conditioned Encoder Representations for Session Data), a pretraining model for semi-structured e-commerce session data, which can serve as a generic session encoder that simultaneously captures both intra
Human Language Technologies, pages 219 - 230
item semantics and inter-item relations. Beyond training a potent language model for intra-item semantics, our model also conditions the language modeling task on graph-level session information, thus encouraging the pretrained model to learn how to utilize inter-item signals. Our model architecture tightly integrates two key components: (1) an item Transformer encoder, which captures text semantics of session items; and (2) a graph conditioned Transformer, which aggregates and propagates inter-item relations for cross-item prediction. As a result, CERES models the higher-level interactions between items.
We have pretrained CERES using 468,199,822 sessions and performed experiments on three session-based tasks: product search, query search, and entity linking. By comparing with publicly available state-of-the-art language models and domain-specific language models trained on alternative representations of session data, we show that CERES outperforms strong baselines on various session-based tasks by large margins. Experiments show that CERES can effectively utilize session-level information for downstream tasks, better capture text semantics for session items, and perform well even with very scarce training examples.
We summarize our contributions as follows: 1) We propose CERES, a pretrained model for semistructured e-commerce session data. CERES can effectively encode both e-commerce items and sessions and generically support various session-based downstream tasks. 2) We propose a new graph-conditioned transformer model for pretraining on general relational structures on text data. 3) We conducted extensive experiments on a large-scale e-commerce benchmark for three session-related tasks. The results show the superiority of CERES over strong baselines, including mainstream pretrained language models and state-of-the-art deep session recommendation models.
2 Customer Sessions
A customer session is the search log before a final purchase action. It consists of customer-query-product interactions: a customer submits search queries obtains a list of products. The customer may take specific actions, including view and purchase on the retrieved products. Hence, a session contains two types of items: queries and products, and various relations between them established by customer actions.
We define each session as a relational graph  $G = (\mathcal{V}, \mathcal{E})$  that contains all queries and products in a session and their relations. The vertex set  $\mathcal{V} = (\mathcal{Q}, \mathcal{P})$  is partitioned into ordered query set  $\mathcal{Q}$  and unordered product set  $\mathcal{P}$ . The queries  $\mathcal{Q} = (q_1, \ldots, q_n)$  are indexed by order of the customer's searches. The edge set  $\mathcal{E}$  contains two types of edges:  $\{(q_i, q_j), i < j\}$  are one-directional edges that connect each query to its previous queries; and  $\{q_i, p_j, a_{ij}\}$  are bidirectional edges that connects the  $i$ th query and  $j$ th product, if the customer took action  $a_{ij}$  on product  $p_j$  retrieved by query  $q_j$ .
The queries and products are represented by textual descriptions. Specifically, each query is represented by customer-generated search keywords. Each product is represented with a table of textual attributes. Each product is guaranteed to have a product title and description. In this paper, we call "product sequence" as the concatenation of title and description. A product may have additional attributes, such as product type, color, brand, and manufacturer, depending on their specific categories.
In this section we present the details of CERES. We first describe our designed session pretraining task in Section 3.1, and then describe the model architecture of CERES in Section 3.2.
3.1 Graph-Conditioned Masked Language Modeling Task
Suppose  $\mathcal{G} = (\mathcal{V},\mathcal{E})$  is a graph on  $T$  text items as vertices,  $v_{1},\ldots ,v_{T}$  , each of which is a sequence of text tokens:  $v_{i} = [v_{i1},\dots,v_{iT_{i}}]$ $i = 1,\dots,T$  We propose graph-conditioned masked language modeling (GMLM), where masked tokens are predicted with both intra-item context and inter-item context:
$$
p _ {\mathrm {G M L M}} \left(v _ {\text {m a s k e d}}\right) = \prod_ {j \text {t h m a s k e d}} \mathbb {P} \left(v _ {i j} \mid \mathcal {G}, \left\{v _ {i k} \right\} _ {k \text {t h u n m a s k e d}}\right), \tag {1}
$$
which encourages the model to leverage information graph-level inter-item semantics efficiently in order to predict masked tokens. To optimize (1), we need to learn token-level embeddings that are infused with session-level information, which we introduce in Section 3.2.2. Suppose certain tokens in the input sequence of items as masked (detailed below), we optimize the predictions of the masked tokens with cross entropy loss. The pretraining framework is illustrated in Figure 3.
Figure 2: Model illustration. CERES first produces intra-item embeddings in the Item Transformer Encoder. Then, the Graph-Conditioned Transformer aggregates and propagates session-level information to obtain inter-item embeddings.
Figure 3: Pretraining framework illustration. CERES learns both inter-item and intra-item embeddings for item tokens for Masked LM and Graph-Conditioned Masked LM. In practice, we find it beneficial to optimize both.
Token Masking Strategy. To mask tokens in long sequences, including product titles and descriptions, we follow (Devlin et al., 2018) and choose  $15\%$  of the tokens for masking. For short sequences, including queries and product attributes, there is a  $50\%$  probability that a short sequence will be masked, and for those sequences  $50\%$  of their tokens are randomly selected for masking.
To model the probability in (1), we design two key components in the CERES model: 1) a Transformer-based item encoder, which produces token-level intra-item embeddings that contain context information within a single item; and 2) a graph-conditioned Transformer for session encoding, which produces session-level embeddings that encodes inter-item relations, and propagates the session information back to the token-level. We illustrate our model architecture in Figure 2.
3.2.1 Item Transformer Encoder
The session item encoder aims to encode intra-item textual information for each item in a session. We design the item encoder based on Transformers, which allows CERES to leverage the expressive power of the self-attention mechanism for modeling domain-specific language in e-commerce sessions. Given an item  $i$ , the transformer-based item
encoder compute its token embeddings as follows:
$$
\begin{array}{l} \left[ \mathbf {v} _ {i 1}, \dots , \mathbf {v} _ {i T _ {i}} \right] = \operatorname {T r a n s f o r m e r} _ {\text {i t e m}} ([ v _ {i 1}, \dots , v _ {i T _ {i}} ]) \tag {2} \\ \mathbf {v} _ {i} = \operatorname {P o o l} ([ v _ {i 1}, \dots , v _ {i T _ {i}} ]), \\ \end{array}
$$
where  $\mathbf{v}_{ij}$  is the embedding of the  $j$ th token in the  $i$ th item, and  $\mathbf{v}_i$  is the pooled embedding of the  $i$ th item. At this stage,  $\{\mathbf{v}_{ij}\}, \{\mathbf{v}_i\}$  are embeddings that only encode the intra-item information.
Details of Item Encoding. We detail the encoding method for the two types of items, queries and products, in the following paragraphs.
Each query  $q_{i} = [q_{i1},\dots ,q_{iT_{i}}]$  is a sequence of tokens generated by customers as search keywords. We add a special token at the beginning of the queries, [SEARCH], to indicate that the sequence represents a customer's search keywords. Then, to obtain the token-level embedding of the queries and the pooled query embedding by taking the embedding of the special token [SEARCH].
Each product  $p_i$  is a table of  $K$  attributes:  $p^1, \ldots, p^K$ , where  $p^1$  is always the product sequence, which is the concatenation of product title and bullet description. Each attribute  $p_i^k = [p_{i1}^k, p_{i2}^k, \ldots]$  starts with a special token [ATTRTYPE], where ATTRTYPE is replaced with the language descriptor of the attribute. Then, the Transformer is used to compute token and sentence embeddings for all attributes. The product embedding is obtained by average pooling of all attribute's sentence embeddings.
3.2.2 Graph-Conditioned Session Transformer
The Graph-Conditioned Session Transformer aims to infuse intra-item and inter-item information to produce item and token embeddings. For this purpose, we first design a position-aware graph neural network (PGNN) to capture the inter-item dependencies in a session graph to produce
Figure 4: Illustration of cross-attention over latent conditioning tokens. The item token embeddings perform self-attention as well as cross-attention over latent conditioning tokens, thus incorporating session-level information. Latent conditioning tokens perform self-attention to update their embeddings, but do not attend to item tokens to preserve session-level information.
item embeddings. The effect of PGNN is analyzed in Section 4.4. Then conditioned on the PGNN-learned item embedding, we propose a cross-attention Transformer, which produces infused item and token embeddings for the Graph-Conditioned Masked Language Modeling task.
Position-Aware Graph Neural Network. We use a GNN to capture inter-item relations. This will allow CERES to obtain item embeddings that encode the information from other locally correlated items in the session. Let  $[\mathbf{v}_1, \dots, \mathbf{v}_N]$  denote the item embeddings produced by the intra-item transformer encoder. We treat them as hidden states of nodes in the session graph  $\mathcal{G}$  and feed them to the GNN model, obtaining session-level item embeddings  $[\mathbf{v}_1^h, \dots, \mathbf{v}_N^h]$ .
The items in a session graph are sequential according to the order the customers generated them. To let the GNN model learn of the positional information of items, we train an item positional embedding in the same way BERT (Devlin et al., 2018) trains positional embeddings of tokens. Before feeding the item embeddings to GNN, the pooled item embeddings are added item positional embeddings according to their positions in the session's item sequence. In this way, the item embeddings  $\{\mathbf{v}^i\}_{i\in \mathcal{V}}$  are encoded their positional information as well.
Cross-Attention Transformer. Conditioned on PGNN, we design a cross-attention transformer which propagates session-level information in PGNN-produced item embeddings to all tokens to produce token embeddings that are infused with both intra-item and inter-item information.
In order to propagate item embeddings to tokens, we treat item embeddings as latent tokens that can be treated as a "part" of item texts. for each item  $i$  we first expand  $\mathbf{v}_i^h$  to  $K$  latent conditioning tokens by using a multilayer perceptron module to map  $\mathbf{v}_i^h$  to  $K$  embedding vectors  $[\mathbf{v}_{i1}^h,\dots ,\mathbf{v}_{iK}^h ]$  of the
same size. For each item  $i$ , we compute its latent conditioning tokens by averaging all latent tokens in its neighborhood. Suppose  $N(i)$  is the set of all neighboring items in the session graph, itself included. In each position, we take the average of the latent token embeddings in  $N(i)$  as the  $k$ th latent conditioning token,  $\overline{\mathbf{v}}_{ik}^{h}$ , for the  $i$ th item. Then, we concatenate the latent conditioning token embeddings and the item token embeddings obtained by the session item encoder:
$$
\left[ \overline {{\mathbf {v}}} _ {i 1} ^ {h}, \dots , \overline {{\mathbf {v}}} _ {i K} ^ {h}, \mathbf {v} _ {i 1}, \dots , \mathbf {v} _ {i N _ {i}} \right]. \tag {3}
$$
Finally, we compute the token-level embeddings with session information by feeding the concatenated sequence to a shallow cross-attention Transformer. The cross-attention Transformer is of the same structure as normal Transformers. The difference is that we prohibit the latent conditioning tokens from attending over original item tokens to prevent the influx of intra-item information potentially diluting session-level information stored in latent conditioning tokens. Illustration of cross-attention Transformer is provided in Figrule 4.
We use the embeddings produced by this cross-attention Transformer as the final embeddings for modeling the token probabilities in Equation (1) and learning the masked language modeling tasks. During training, the model is encouraged to learn good token embeddings with the Item Transformer Encoder, as better embeddings  $\{\mathbf{v}_{ij}\}_{j=1}^{N_i}$  is necessary to improve the quality of  $\{\mathbf{v}_{ij}^c\}_{j=1}^{N_i}$ . The Graph-Conditioned Transformer will be encouraged to produce high-quality session-level embeddings for the GMLM task. Hence, CERES is encouraged to produce high-quality embeddings that unify both intra-item and inter-item information.
3.3 Finetuning
When finetuning CERES for downstream tasks, we first obtain session-level item embeddings. The session embedding is computed as the average of all item embeddings. To obtain embedding for a single item without session context, such as for retrieved items in recommendation tasks, only the Item Transformer Encoder is used.
To measure the relevance of an item to a given session, we first transform the obtained embeddings by separate linear maps. Denote the transformed session embeddings as s and item embeddings as y. The similarity between them is computed by cosine similarity  $\mathrm{d}_{\mathrm{cos}}(\mathbf{s},\mathbf{y})$  . To finetune
the model, we optimize a hinge loss on the cosine similarity between sessions and items.
Dataset. We collected customer sessions from Amazon for pretraining and finetuning on downstream tasks. 468,199,822 customer sessions are collected from August 1 2020 to August 31 2020 for pretraining. 30,000 sessions are collected from September 2020 to September 7 2020 for downstream tasks. The pretraining and downstream datasets are from disjoint time spans to prevent data leakage. All data are cleaned and anonymized so that no personal information about customers was used. Each session is collected as follows: when a customer performs a purchase action, we backtrace all actions by the customer in 600 seconds before the purchase until a previous purchase is encountered. The actions of customers include: 1) search, 2) view, 3), add-to-cart, and 4) purchase. Search action is associated with customer generated query keywords. View, add-to-cart, and purchase are associated with the target products. All the products in the these sessions are gathered with their product title, bullet description, and various other attributes, including color, manufacturer, product type, size, etc. In total, we have 37,580,637 products. The sessions have an average of 3.24 queries and 4.36 products. Queries have on average 5.63 tokens, while product titles and bullet descriptions have averagely 17.42 and 96.01 tokens.
Evaluation Tasks and Metrics. We evaluate all the compared models on the following tasks: 1) Product Search. In this task, given observed customer behaviors in a session, the model is asked to predict which product will be purchased from a pool of candidate products. The purchased products are removed from sessions to avoid trivial inference. The candidate product pool is the union of all purchased products in the test set and the first 10 products returned by the search engine of all sessions in the test set.
2) Query Search. Query Search is a recommendation task where the model retrieves next queries for customers which will lead to a purchase. Given a session, we hide the last query along with products associated with it, i.e. viewed or purchased with the removed query. Then, we ask the model to predict the last query from a pool of candidate
queries. The candidate query pool consists of all last queries in the test set.
3) Entity Linking. In this task we try to understand the deeper semantics of customer sessions. Specifically, if customer purchases a product in a session, the task is to predict the attributes of the purchased product from the rest contexts in the session. In total, we have 60K possible product attributes.
Baselines. The compared baselines can be categorized into three groups:
1) General-domain pretrained language models which include BERT (Devlin et al., 2018), RoBERTa (Liu et al., 2019), and ELECTRA (Clark et al., 2020). These models are state-of-the-art pretrained language models, which can serve as general-purpose language encoders for items and enable downstream session-related tasks. Specifically, the language encoders produce item embeddings first, and compose session embeddings by pooling the items in sessions. To retrieve items for sessions, one can compare the cosine similarity between sessions and retrieved items.
2) Pretrained session models which are pretrained models on e-commerce session data. Specifically, we pretrain the following language models using our session data: a) Product-BERT, which is a domain-specific BERT model pretrained with product information; b) SQSP-BERT, where SQSP is short for Single-query Single-Product. SQSP-BERT is pretrained on query-product interaction pairs with language modeling and contrastive learning objectives. They are used in the same manner in downstream tasks as general-domain pretrained language models. The detailed configurations are provided in the Appendix.
3) Session-based recommendation methods including SR-GNN (Wu et al., 2019b) and NISER+ (Gupta et al., 2019), which are state-of-the-art models for session-based product recommendation on traditional benchmarks, including YOOCHOOSE and DIGINETICA; and Nvidia's MERLIN (Mobasher et al., 2001), which is the best-performing model in the recent SIGIR Next Items Prediction challenge (Kallumadi et al., 2021)
To evaluate the performance on these tasks, we employ standard metrics for recommendation systems, including MAP@K, and Recall@K.
The implementation details for pretraining and finetuning stages are described as follows.
Pretraining details. We developed our model based on Megatron-LM (Shoeybi et al., 2019). We used 768 as the hidden size, a 12-layer transformer blocks as the backbone language model, a two-layer Graph Attention Network and three-layer Transformer as the conditioned language model layers. In total, our model has 141M parameters. The model is trained for 300,000 steps with a batch size of 512 sessions. The parameters are updated with Adam, with peak learning rate as  $3e - 5$ ,  $1\%$  steps for linear warm-up, and linear learning rate decay after warm-up until the learning rate reaches the minimum  $1e - 5$ . We trained our model on 16 A400 GPUs on Amazon AWS for one week.
Finetuning details. For each downstream task, we collected 30,000 sessions for training, 3000 for validation and 5000 for testing. For each of the pretrained model, we finetune them for 10 epochs with a maximal learning rate chosen from [1e-4, 1e-5, 5e-5, 5e-6] to maximize MAP@1 on the validation set. The rest of the configuration of optimizers is the same as in pretraining.
4.3.1 Product Search
Table 1 shows the performance of different methods for the product search task. We observe that CERES outperforms domain-specific methods by more than  $1\%$  and general-domain methods by over  $6\%$  in MAP@1. The second best performing model is Product-BERT, which is pretrained on product information alone.
We also compared with session-based recommendation systems. SR-GNN and NISER+ model only session graph structure but not text semantics; hence they have limited performance because of the suboptimal representation of session items. While MERLIN can capture better text semantics, its text encoder is not trained on domain-specific e-commerce data. While it can outperform general-domain methods, its performance is lower than Product-BERT and CERES. The benefits of joint modeling of text and graph data and the Graph-Conditioned MLM allow CERES to outperform existing session recommendation models.
4.3.2 Query Search
Table 2 shows the performance of different methods on Query Search. Query Search is a more difficult task than Product Search because customer-generated next queries are of higher variance. In this challenging task, CERES outperforms the best
domain-specific model by over  $7\%$  and general-domain model by  $12\%$  in all metrics.
4.3.3 Entity Linking