Year: 2020

Abstract:
Existing pre-trained language models (PLMs) have demonstrated the effectiveness of self-supervised learning for a broad range of natural language processing (NLP) tasks. However, most of them are not explicitly aware of domain-specific knowledge, which is essential for downstream tasks in many domains, such as tasks in e-commerce scenarios. In this paper, we propose K-PLUG, a knowledgeinjected pre-trained language model based on the encoder-decoder transformer that can be transferred to both natural language understanding and generation tasks. We verify our method in a diverse range of e-commerce scenarios that require domain-specific knowledge. Specifically, we propose five knowledgeaware self-supervised pre-training objectives to formulate the learning of domainspecific knowledge, including e-commerce domain-specific knowledge-bases, aspects of product entities, categories of product entities, and unique selling propositions of product entities. K-PLUG achieves new state-of-the-art results on a suite of domain-specific NLP tasks, including product knowledge base completion, abstractive product summarization, and multi-turn dialogue, significantly outperforms baselines across the board, which demonstrates that the proposed method effectively learns a diverse set of domain-specific knowledge for both language understanding and generation tasks. The code, data, and models will be publicly available1.

heading: 1 INTRODUCTION
text: Pre-trained language models (PLMs), such as ELMo (Peters et al., 2018), GPT (Radford et al., 2018), BERT (Devlin et al., 2019), RoBERTa (Liu et al., 2019), and XLNet (Yang et al., 2019), have made remarkable breakthroughs in many natural language understanding (NLU) tasks, including text classification, reading comprehension, and natural language inference. These models are trained on large-scale text corpora with self-supervision based on either bi-directional or auto-regressive pre-training. Equally promising performances have been achieved in natural language generation (NLG) tasks, such as machine translation and text summarization, by MASS (Song et al., 2019), UniLM (Dong et al., 2019), BART (Lewis et al., 2020), T5 (Raffel et al., 2019), PEGASUS (Zhang et al., 2020), and ProphetNet (Yan et al., 2020). In contrast, these approaches adopt Transformerbased sequence-to-sequence models to jointly pre-train for both the encoder and the decoder.
While these PLMs can learn rich semantic patterns from raw text data and thereby enhance downstream NLP applications, many of them do not explicitly model domain-specific knowledge. As a result, they may not be as sufficient for capturing human-curated or domain-specific knowledge that is necessary for tasks in a certain domain, such as tasks in e-commerce scenarios. In order to overcome this limitation, several recent studies have proposed to enrich PLMs with explicit knowledge, including knowledge base (KB) (Zhang et al., 2019; Peters et al., 2019; Xiong et al., 2020; Wang et al., 2019; 2020), lexical relation (Lauscher et al., 2019; Wang et al., 2020), word sense (Levine et al., 2020), part-of-speech tag (Ke et al., 2019), and sentiment polarity (Ke et al., 2019; Tian et al., 2020). However, these methods only integrate domain-specific knowledge into the encoder, and the decoding process in many NLG tasks benefits little from these knowledge.
1Our code is available at https://github.com/ICLR21Anonymous/knowledge_pretrain.
To mitigate this problem, we propose a Knowledge-injected Pre-trained Language model that is suitable for both Natural Language Understanding and Generation (K-PLUG). Different from existing knowledge-injected PLMs, K-PLUG integrates knowledge into pre-training for both the encoder and the decoder, and thus K-PLUG can be adopted to both downstream knowledge-driven NLU and NLG tasks. We verify the performance of the proposed method in various e-commerce scenarios. In the proposed K-PLUG, we formulate the learning of four types of domain-specific knowledge: e-commerce domain-specific knowledge-bases, aspects of product entities, categories of product entities, and unique selling propositions (USPs) (Garrett, 1961) of product entities. Specifically, e-commerce KB stores standardized product attribute information, product aspects are features that play a crucial role in understanding product information, product categories are the backbones for constructing taxonomies for organization, and USPs are the essence of what differentiates a product from its competitors. K-PLUG learns these types of knowledge into a unified PLM, enhancing performances for various language understanding and generation tasks.
To effectively learn these four types of valuable domain-specific knowledge in K-PLUG, we proposed five new pre-training objectives: knowledge-aware masked language model (KMLM), knowledge-aware masked sequence-to-sequence (KMS2S), product entity aspect boundary detection (PEABD), product entity category classification (PECC), and product entity aspect summary generation (PEASG). Among these objectives, KMLM and KMS2S learn to predict the masked single and multiple tokens, respectively, that are associated with domain-specific knowledge rather than general information; PEABD determines the boundaries between descriptions of different product aspects given full product information; PECC identifies the product category that each product belongs to; and PEASG generates a summary for each individual product aspect based on the entire product description.
After pre-training K-PLUG, we fine-tune it on three domain-specific NLP tasks, namely, ecommerce knowledge base completion, abstractive product summarization, and multi-turn dialogue. The results show that K-PLUG significantly outperforms comparative models on all these tasks.
Our main contributions can be summarized as follows:
• We present K-PLUG that learns domain-specific knowledge for both the encoder and the decoder in a pre-training language model framework, which benefits both NLG and NLU tasks.
• We formulate the learning of four types of domain-specific knowledge in e-commerce scenarios: e-commerce domain-specific knowledge-bases, aspects of product entities, categories of product entities, and unique selling propositions of product entities, which provide critical information for many applications in the domain of e-commerce. Specifically, five self-supervised objectives are proposed to learn these four types of knowledge into a unified PLM.
• Our proposed model exhibits clear effectiveness in many downstream tasks in the ecommerce scenario, including e-commerce KB completion, abstractive product summarization, and multi-turn dialogue.

heading: 2 RELATED WORK

heading: 2.1 PLMS IN GENERAL
text: Unsupervised pre-training language model has been successfully applied to many NLP tasks. ELMo (Peters et al., 2018) learns the contextual representations based on a bidirectional LM. GPT (Radford et al., 2018) predicts tokens based on the context on the left-hand side. BERT (Devlin et al., 2019) adopts a bi-directional LM to predict the masked tokens. XLNet (Yang et al., 2019) predicts masked tokens in a permuted order through an autoregressive method. MASS (Song et al., 2019) pre-trains the sequence-to-sequence LM to recover a span of masked tokens. UniLM (Dong et al., 2019) combines bidirectional, unidirectional, and sequence-to-sequence LMs. T5 (Raffel et al., 2019) and BART (Lewis et al., 2020) present denoising sequence-to-sequence pre-training. PEGASUS (Zhang et al., 2020) pre-trains with gap-sentence generation objective. While humancurated or domain-specific knowledge is essential for downstream knowledge-driven tasks, these methods do not explicitly consider external knowledge like our proposed K-PLUG.
ties, categories of product entities, and
::::::::::::::::::::::::::::::::::::: unique selling propositions of product entities. Pre-training
objectives include knowledge-aware masked language model (KMLM), knowledge-aware masked sequence-to-sequence (KMS2S), product entity aspect boundary detection (PEABD), product entity category classification (PECC), and product entity aspect summary generation (PEASG).

heading: 2.2 INJECTING KNOWLEDGE INTO PLMS
text: Recent work investigates how to incorporate knowledge into PLMs for NLU. ERNIE (Sun et al., 2019) enhances language representation with the entity/phrase-level masking. ERNIE (Zhang et al., 2019) identifies and links entity mentions in texts to their corresponding entities in KB. Similar to ERNIE (Zhang et al., 2019), KnowBERT (Peters et al., 2019) injects KBs into PLM. Xiong et al. (2020) leverages an entity replacement pre-training objective to learn better representations for entities. KEPLER (Wang et al., 2019) adopts the knowledge embedding objective in the pretraining. Besides, SKEP (Tian et al., 2020), SenseBERT (Levine et al., 2020), SentiLR (Ke et al., 2019), and K-ADAPTER (Wang et al., 2020) propose to integrate sentiment knowledge, word sense, sentiment polarity, and lexical relation into PLM, respectively. However, most of these studies are focused on integrating knowledge for language understanding task, work of utilizing domainspecific knowledge for pre-training for language generation tasks are limited. Inspired by these work, we construct K-PLUG that learns domain-specific knowledge into a PLM for both NLU and NLG tasks.

heading: 3 KNOWLEDGE-INJECTED PRE-TRAINING
text: In this section, we explain the data used to pre-train K-PLUG, its model architecture, and our pretraining objectives.

heading: 3.1 DATA PREPARATION
text: We collect the pre-training data from a mainstream Chinese e-commerce platform2, which contains approximately 25 million textual product descriptions and covers 40 product categories. With an average length of 405 tokens, these product descriptions constitute a corpus with a size of 10B Chinese characters. Each product description consists of information on 10.7 product aspects on average, and each product aspect is accompanied with a summary highlighting its prominent features, as shown in Figure 1(a). Additionally, the e-commerce KB and USPs (further explained below) used in our pre-training data are as specified by the e-commerce platform and its online stores.
2https://www.jd.com/

heading: 3.2 MODEL ARCHITECTURE
text: K-PLUG adopts the standard sequence-to-sequence Transformer architecture (Vaswani et al., 2017), consisting of a 6-layer encoder and a 6-layer decoder as Song et al. (2019). We set the size of hidden vectors as 768, and the number of self-attention heads as 12. More details about the experimental settings are in the appendix.

heading: 3.3 KNOWLEDGE FORMULATION AND PRE-TRAINING OBJECTIVES
text: We formulate the learning of four types of knowledge in a unified PLM: e-commerce KB, aspects of product entities, categories of product entities, and USPs of product entities. Specifically, ecommerce KB stores standardized product attribute information, e.g., (Material: Cotton) and (Collar Type: Pointed Collar). It provides details about the products (Logan IV et al., 2017). Aspects of product entities are features of a product, such as the sound quality of a stereo speaker, etc. (Li et al., 2020). Categories of product entities such as Clothing and Food are widely used by e-commerce platforms to organize their products so to present structured offerings to their customers (Luo et al., 2020; Dong et al., 2020) USPs of product entities are the essence of what differentiates a product from its competitors (Garrett, 1961). For example, a stereo speaker’s USP exhibiting its supreme sound quality could be “crystal clear stereo sound”. An effective USP immediately motivates the purchasing behavior of potential buyers.
We propose and evaluate five novel self-supervised pre-training objectives to learn the abovementioned four types of knowledge in the K-PLUG model (see Figure 1).

heading: Knowledge-aware Masked Language Model (KMLM)
text: Inspired by BERT (Devlin et al., 2019), we adopt the masked language model (MLM) to train the Transformer encoder as one of our pre-training objectives, which learns to predict the masked tokens in the source sequence (e.g., “The company is [MASK] at the foot of a hill.”). Similar to BERT, we mask 15% of all tokens in a text sequence; 80% of the masked tokens are replaced with the [MASK] token, 10% with a random token, and 10% left unchanged. Particularly, given an original text sequence x = (x1, ..., xm, ..., xM ) with M tokens, a masked sequence is produced by masking xm through one of the three ways explained above, e.g., replacing xm with [MASK] to create x̃ = (x1, ..., [MASK], ..., xM). MLM aims to model the conditional likelihood P (xm|x̃), and the loss function is:
LMLM = logP (xm|x̃) (1)
The major difference from BERT is that our KMLM prioritizes knowledge tokens, which contain knowledge regarding product attributes and USPs, when selecting positions to mask and, in the case that the knowledge tokens make up less than 15% of all tokens, randomly picks non-knowledge tokens to complete the masking.

heading: Knowledge-aware Masked Sequence-to-Sequence (KMS2S)
text: K-PLUG inherits the strong ability of language generation from the masked sequence-to-sequence (MS2S) objective. The encoder takes a sentence with a masked fragment (several consecutive tokens) as the input, and the decoder predicts this masked fragment conditioned on the encoder representations (e.g., “The company [MASK] [MASK] [MASK] the foot of a hill.”).
Given a text sequence x = (x1, ..., xu, ..., xv, ..., xM ), a masked sequence x̃ = (x1, ..., [MASK], ..., [MASK], ..., xM) is produced by replacing the span xu:v , ranging from xu to xv , with the [MASK] token. MS2S aims to model P (xu:v|x̃), which can be further factorized into a product P (xu:v|x̃) = ∏v t=u P (xt|x̃) according to the chain rule. The loss function is:
LMS2S = v∑ t=u logP (xt|x̃) (2)
We set the length of the masked span as 30% of the length of the original text sequence. Similar to KMLM, KMS2S prioritizes the masking of text spans that cover knowledge tokens.

heading: Product Entity Aspect Boundary Detection (PEABD)
text: A product description usually contains multiple product entity aspects. Existing work (Li et al., 2020) proves that product aspects influence the quality of product summaries from the views of importance, non-redundancy, and readability, which are not directly taken into account in language modeling. In order to train a model that understands product aspects, we leverage the PEABD objective to detect boundaries between the product entity aspects. It is essentially a sequence labeling task based on the representations of K-PLUG’s top encoder layer.
Given a text sequence x = (x1, ..., xM ), the encoder of K-PLUG outputs a sequence h = (h1, ..., hM ), which is fed into a softmax layer, and generates a probability sequence y. The loss function is:
LPEABD = − ∑ t ŷt log yt (3)
where y ∈ {[0, 1]} are the ground-truth labels for the aspect boundary detection task.

heading: Product Entity Category Classification (PECC)
text: Product entity categories are the backbones for constructing taxonomies (Luo et al., 2020; Dong et al., 2020). Each product description document corresponds to one of the 40 categories included in our corpus, such as Clothing, Bags, Home Appliances, Shoes, Foods, etc. Identifying the product entity categories accurately is the prerequisite for creating an output that is consistent with the input.
Given a text sequence x = (x1, ..., xM ), a softmax layer outputs the classification score, y, based on the representation of the encoder classification token, [CLS]. The loss function maximizes the model’s probability of outputting the true product entity category as follows:
LPECC = −ŷ log y (4) where ŷ is the ground-truth product category.

heading: Product Entity Aspect Summary Generation (PEASG)
text: Inspired by PEGASUS (Zhang et al., 2020), which proves that using a pre-training objective that more closely resembles the downstream task leads to better and faster fine-tuning performance, we propose a PEASG objective to generate a summary from the description of a product entity aspect. Unlike extracted gap-sentences generation in PEGASUS, our method constructs a more realistic summary generation task because the aspect summary naturally exists in our pre-training data.
Given an aspect description sequence x = (x1, ..., xM ), and an aspect summary sequence y = (y1, ..., yT ), PEASG aims to model the conditional likelihood P (y|x). The loss function is:
LPEASG = ∑ t logP (yt|x,y<t) (5)

heading: 4 EXPERIMENTS AND RESULTS

heading: 4.1 PRE-TRAINED MODEL VARIANTS
text: To evaluate the effectiveness of pre-training with domain-specific data and with domain-specific knowledge separately, we implement pre-training experiments with two model variants: C-PLUG and E-PLUG, whose configurations are the same as that of K-PLUG.
• C-PLUG is a pre-trained language model with the original objectives of MLM and MS2S, trained with a general pre-training corpus, CLUE (Xu et al., 2020), which contains 30GB of raw text with around 8B Chinese words.
• E-PLUG is a pre-trained language model with the original objectives of MLM and MS2S, trained with our collected e-commerce domain-specific corpus.

heading: 4.2 DOWNSTREAM TASKS
text: We fine-tune K-PLUG on three downstream tasks: e-commerce KB completion, abstractive product summarization, and multi-turn dialogue. The e-commerce KB completion task involves the
prediction of product attributes and values given product information. The abstractive product summarization task requires the model to generate a product summary from textual product description. The multi-turn dialogue task aims to output the response by utilizing a multi-turn dialogue context. The domain-specific knowledge we defined in this paper is essential for these tasks.

heading: 4.2.1 E-COMMERCE KB COMPLETION
text: Task Definition. E-commerce KB provides abundant product information that is in the form of (product entity, product attribute, attribute value), such as (pid#133443, Material, Copper Aluminum). For the E-commerce KB completion task, the input is a textual product description for a given product, and the output is the product attribute values.
Dataset. We conduct experiments on the dataset of MEPAVE (Zhu et al., 2020). This dataset is collected from a major Chinese e-commerce platform, which consists of 87,194 instances annotated with the position of attribute values mentioned in the product descriptions. There are totally 26 types of product attributes such as Material, Collar Type, Color, etc. The training, validation, and testing sets contain 71,194/8,000/8,000 instances, respectively.
Model. We consider the e-commerce KB completion task as a sequence labeling task that tags the input word sequence x = (x1, ..., xN ) with the label sequence y = (y1, ..., yN ) in the BIO format. For example, for the input sentence “A bright yellow collar”, the corresponding labels for “bright” and “yellow” are Color-B and Color-I, respectively, and O for the other tokens. For an input sequence, K-PLUG outputs an encoding representation sequence, and a linear classification layer with the softmax predicts the label for each input token based on the encoding representation.
Model P R F1
LSTM 79.68 86.43 82.92 ScalingUp 65.48 93.78 77.12 BERT 78.27 88.62 83.12 JAVE 80.27 89.82 84.78 M-JAVE 83.49 90.94 87.17
C-PLUG 89.79 96.47 93.02 E-PLUG 89.91 96.75 93.20 K-PLUG 93.58 97.92 95.97

heading: Baselines.
text: Result. Table 1 shows the experimental results. We observe that our K-PLUG performs better than baselines. C-PLUG achieves significantly better performance than BERT, which indicates that MS2S can also benefit the NLU task. E-PLUG outperforms C-PLUG, showing that training with domainspecific corpus is helpful. K-PLUG further exhibits a 2.51% improvement compared with E-PLUG. In short, we can conclude that the improvement is due to both the domain-specific pre-training data and knowledge-injected pre-training objectives.

heading: 4.2.2 ABSTRACTIVE PRODUCT SUMMARIZATION
text: Task Definition. Abstractive product summarization task aims to capture the most attractive information of a product that resonates with potential purchasers. The input for this task is a product description, and the output is a condensed product summary.
Dataset. We perform experiments on the dataset of Li et al. (2020), which contains 1.4 million instances collected from a major Chinese e-commerce platform, covering three categories of product: Home Appliances, Clothing, and Cases & Bags. Each instance in the dataset is a (product information, product summary) pair, and the product information contains an image, a title, and other product descriptions. In our work, we do not consider the visual information of products. Notice that the task of abstractive product summarization and product entity aspect summary generation (PEASG) are partly different. The abstractive product summarization task aims to generate a complete and cohesive product summary given a detailed product description. Given a product aspect description, PEASG aims to produce an aspect summary that basically consists of condensed USPs. In addition, for abstractive product summarization task, the average length of the product summaries is 79, while the lengths of the product aspect summaries are less than 10 in general.
Model. Abstractive product summarization task is an NLG task that takes the product description as the input and product summary as the output.

heading: Baselines.
text: • LexRank (Erkan & Radev, 2004) is a graph-based extractive summarization method. • Seq2seq (Bahdanau et al., 2015) is a standard seq2seq model with an attention mechanism. • Pointer-Generator (PG) (See et al., 2017) is a seq2seq model with a copying mechanism. • Aspect MMPG (Li et al., 2020) is the-state-of-the-art method for abstractive product sum-
marization, taking both textual and visual product information as the input.
Result. Table 2 shows the experimental results, including ROUGE-1 (RG-1), ROUGE-2 (RG-2), and ROUGE-L (RG-L) F1 scores (Lin & Hovy, 2003). K-PLUG clearly performs better than other text-based methods. E-commerce knowledge plays a significant role in the abstractive product summarization task, and domain-specific pre-training data and knowledge-injected pre-training objectives both enhance the model. K-PLUG achieves comparable results with the multimodal model, Aspect MMPG. The work of Li et al. (2020) suggests that product images are essential for this task, and we will advance K-PLUG with multimodal information in the future.

heading: Model Home Applications Clothing Cases&BagsRG-1 RG-2 RG-L RG-1 RG-2 RG-L RG-1 RG-2 RG-L

heading: 4.2.3 MULTI-TURN DIALOGUE
text: Task Definition. The multi-turn dialogue task aims to output a response based on the multi-turn dialogue context (Shum et al., 2018). The input for this task is the dialogue context consisting of previous question answering, and the output is the response to the last question.
Dataset. We conduct experiments on two datasets of JDDC (Chen et al., 2020) and ECD (Zhang et al., 2018). JDDC is collected from the conversations between users and customer service staffs from a popular e-commerce website in China and contains 289 different intents, which are the goals of a dialogue, such as updating addresses, inquiring prices, etc, from after-sales assistance. There are 1,024,196 multi-turn sessions and 20,451,337 utterances in total. The average number of turns for each session is 20, and the average tokens per utterance is about 7.4. After pre-processing, the training, validation, and testing sets include 1,522,859/5,000/5,000 (dialogue context, response) pairs, respectively. ECD is collected from another popular e-commerce website in China and covers over 5 types of conversations based on 20 commodities. Additionally, for each ground-truth response, negative responses are provided for discriminative learning. The training, validation, and testing sets include 1,000,000/10,000/10,000 (dialogue context, response) pairs, respectively.
Model. We test with two types of K-PLUG: retrieval-based K-PLUG on the ECD dataset and generative-based K-PLUG on the JDDC dataset. For the retrieval-based approach, we concatenate the dialogue context and use [SEP] token to separate context and response. The [CLS] representation is fed into the output layer for classification. The generative-based approach is a sequence-to-
sequence model, which is the same as the model adopted in the abstractive product summarization task.
Baselines. The baselines also include both the retrieval-based (BM25, CNN, BiLSTM, and BERT) and generative-based approaches. Other baselines are as follows.
• SMN (Wu et al., 2017) matches a response with each utterance in the context.
• DUA (Zhang et al., 2018) is a deep utterance aggregation model based on the fine-grained context representations.
• DAM (Zhou et al., 2018) matches a response with the context based using dependency information based on self-attention and cross-attention.
• IoI (Tao et al., 2019) is a deep matching model by stacking multiple interactions blocks between utterance and response.
• MSN (Yuan et al., 2019) selects relevant context and generates better context representations with the selected context.
Result. Table 3 and 4 show the experimental results on the JDDC and ECD datasets, respectively. We report ROUGE-L (RG-L) F1, BLEU, and recall at position k in n candidates (Rn@k). We can observe that, both on the retrieval-based and generative-based tasks, K-PLUG achieves new state-ofthe-art results, and e-commerce knowledge presents consistent improvements. K-PLUG is evidently superior to BERT, possibly due to BERT’s lack of domain-specific knowledge for pre-training with the general MLM objective.
We further perform a human evaluation on the JDDC dataset. We randomly choose 100 samples from the test set, and three experienced annotators are involved to determine whether K-PLUG outperforms E-PLUG with respect to (1) relevance between the response and the contexts and (2) readability of the response. The results are shown in Table 5. We can see that the percentage of “Win”, which denotes that the results of K-PLUG is better than E-PLUG, is significantly larger than “Lose” (p-value < 0.01 for t-test). Kappa values (Fleiss, 1971) confirm the consistency for different annotators.

heading: 4.3 ABLATION STUDIES
text: To better understand our model, we perform ablation experiments to study the effects of different pre-training objectives.
Result. The ablation results are shown in Table 6. We can conclude that the lack of any pretraining objective hurts performance across all the tasks. KMS2S is the most effective objective for the abstractive product summarization and generative conversation tasks since this objective is highly close to the essence of NLG. Product-aspect-related objectives, i.e., PEABD and PEASG, contribute much to the abstractive product summarization task, which proves that this task requires comprehensively understanding the product description from the view of product aspects, going beyond individual tokens.

heading: 5 CONCLUSION
text: We present a knowledge-injected pre-trained model (K-PLUG) that is a powerful domain-specific language model trained on a large-scale e-commerce corpus designed to capture e-commerce knowledge, including e-commerce KB, product aspects, product categories, and USPs. The pre-training framework combines masked language model and masked seq2seq with novel objectives formulated as product aspect boundary detection, product aspect summary generation, and product category classification tasks. Our proposed model demonstrates strong performances on both natural language understanding and generation downstream tasks, including e-commerce KB completion, abstractive product summarization, and multi-turn dialogue.

heading: A APPENDIX

heading: A.1 EXPERIMENTS SETTINGS FOR PRE-TRAINING
text: We adopt GELU activation (Hendrycks & Gimpel, 2016) as in GPT (Radford et al., 2018). We use Adam optimizer (Kingma & Ba, 2015) with a learning rate of 5e-4, β1 = 0.9, β2= 0.98, L2 weight decay of 0.01, learning rate warm-up over the first 10,000 steps and linear decay of the learning rate. The dropout probability is 0.1. The maximum sequence length is set to 512 tokens. Pre-training was performed with 4 Telsa V100 GPUs. The pre-training is done within 10 epochs, which takes around 10 days, and the fine-tuning takes up to 1 day. We use the beam search with a beam size of 5 for inference for the NLG tasks. Other hyper-parameters can be found in our code.

heading: A.2 CASE STUDIES
text: We present some examples from the test set of each task, with comparisons of the ground-truth result and the outputs produced by the models of E-PLUG and K-PLUG.
unique selling propositions of product entities.

Review summaries:
This paper proposes pretraining language model for e-commerce domain. Specifically, the authors design five pretraining objectives to incorporate various domain knowledge into the the models with an encoder-decoder architecture. When further finetuned on language understanding and generation tasks in the e-commerce domain, the proposed models named K-PLUG outperforms the existing baseline models including those pretrained on general domains. The paper is generally easy to follow. Designs of the pretraining objectives are reasonable and empirically effective. Experiments are solid and convincing.