K-PLUG: KNOWLEDGE-INJECTED PRE-TRAINED LANGUAGE MODEL FOR NATURAL LANGUAGE UNDERSTANDING AND GENERATION
Authors: Song Xu
Haoran Li
Peng Yuan
Yujia Wang
Youzheng Wu
Xiaodong He
Ying Liu
Bowen Zhou
Venue:  (2021)

Abstract: Existing pre-trained language models (PLMs) have demonstrated the effectiveness of self-supervised learning for a broad range of natural language processing (NLP) tasks. However, most of them are not explicitly aware of domain-specific knowledge, which is essential for downstream tasks in many domains, such as tasks in e-commerce scenarios. In this paper, we propose K-PLUG, a knowledge-injected pre-trained language model based on the encoder-decoder transformer that can be transferred to both natural language understanding and generation tasks. We verify our method in a diverse range of e-commerce scenarios that require domain-specific knowledge. Specifically, we propose five knowledge-aware self-supervised pre-training objectives to formulate the learning of domain-specific knowledge, including e-commerce domain-specific knowledge-bases, aspects of product entities,  categories of product entities, and unique selling propositions of product entities. K-PLUG achieves new state-of-the-art results on a suite of domain-specific NLP tasks, including product knowledge base completion, abstractive product summarization, and multi-turn dialogue, significantly outperforms baselines across the board, which demonstrates that the proposed method effectively learns a diverse set of domain-specific knowledge for both language understanding and generation tasks. The code,  data, and models will be publicly available.


Under review as a conference paper at ICLR 2021
K-PLUG: KNOWLEDGB-INEECED PRE-TRAINED LANGUAGE MODEL FOR NATURAL LANGUAGE UN- DERSTANDING AND GENERATION
Anonymous authors Paper under double-bling review
ABSTRACT
Existing pre-trained language models (PLMs) have demonstrated the effective- ness of self-supervised learning for a broad range of natural language processing (NLP) tasks. However, most of them are not explicitly aware of domain-specific knowledge, which is essential for downstream tasks in many domains, such as tasks e-commerce scenarios. this paper, we propose K-PLUG, a knowledge- injected pre-trained language model based on the encoder-decoder transformer that can transferred to both natural language understanding and generation tasks. We verify our method in diverse range of e-commerce scenarios that require domain-specific knowledge. Specifically, we propose five knowledge- aware self- supervied pre- pre-training objectives to formulate the learning of domain- specific knowledge, including e-commerce domain-specifte knowledge-basee pects of product entities, categories of product entities, and unique selling propo- sitions of product entities. K-PLUG achieves new state-of-the-ee results on suite of domain-specific NLP tasks, including product knowledge base completion, ab- stractive product summarization, and multi-turn dialogue, significantly outper- forms baselines across the board, which demonstrates that the proposed method effectively learns a diverse set of domain-specinf knowledge for both language understanding and generation tasks. The code, data, and models will be publicly available
as-
INTRODUCTION
Pre-trained language models (PLMs), such as ELMo Petees et al. 2018), GPT (Radford et al. 2018), BERT (Devlin et al. 2019). RoBERTa (Liu al. 2019), and XL.Net Yang et 2019), have made remarkable breakthroughs in many natural language understanding (NLU) tasks, including text classification, reading comprehension, and natural language inference. These models are trained on large-scale text corpora with self-supervision based on either bi-directional or auto-regressive pre-training. Equally promising performances have been achieved in natural language generation (NLG) tasks, such as machine translation and text summarization, by MASS (Song et al. 2019). UniLM (Dong et al. 2019), BART (Lewis et al. 2020). T5 (Raffel et al. 2019). PEGASUS (Zhang al. 2020), and ProphetNet Yan al. 2020). In contrast, these approaches adopt Transformer- based lequeecce-o-sequeee
models to jointly pre-train for both the encoder and the decoder.
While these PLMs can learn rich semantic patterns from raw text data and thereby enhance down- stream NLP applications, many of them do not explicitly model domain -specffic knowledge, As a result, they may not be sufficient for capturing human-curated domain-specific knowledge that is necessary for tasks in a certain domain, such as tasks in e-commerce scenarios. order to over- come this limitation, several recent studies have proposed to enrich PLMs with explicit knowledge, including knowledge base (KB) (Zhagg et al.] [019 Peters et al. 2019 Xiong et al.) 2020; Wang al. 2019: 2020), lexical relation (Lauscher al. 2019: Wang al.] 2020). word sense (Levine et al. 2020), offt-ofs tag (Ke et al. 2019), and sentiment polarity (Ke et al. [2019] Tian et al.,
2020. However, these methods only integrate domain-specific knowledge into the encoder, and the decoding process in many NLG tasks benefits little from these knowledge.
Ouu code is available
CI.221 Awwnynonnnnnoonoooooooo wwt.kkkwwweeeeeeeeeeeeeee edge_pretrainn

---

Under review as a conference paper at ICLR 2021
To mitigate this problem, we propose Knowledge-injeeted Pre-trained Language model that is suitable for both Natural Language Understanding and Generation (K-PLUG). Different from exist- ing knowledge-injected PLMs, K-PLUG integrates knowledge into pre-training for both the encoder and the decoder, and thus K-PLUG can be adopted to both downstream knowledge-driven NLU and NLG tasks. We verify the performance of the proposed method in various e-commerce scenarios. In the proposed K-PLUG, we formulate the learning of four types of domain-specific knowledge: e-commerce domain-specific knowledge-bases, aspects of product entities, categories of product entities, and unique selling propositions (USPs) (Garrett 1961) of product entities. Specifically, e-commerce KB stores standardized product attribute information, product aspects are features that play crucial role in understanding product information, product categories are the backbones for constructing taxonomies for organization, and USPs are essence of what differentiates a prod- uet from its competitors. K-PIUG learns these types of knowledge into unified PLM, enhancing performances for various language understanding and generation tasks.
To effectively learn these four types of valuable domain-specific knowledge in K-PLUG, proposed five new pre-training objectives: knowledge-aware masked language model (KMLM), knowledge- masked
toreencee---eeueeee (KMS2S), product entity aspect boundary detec- tion (PEABD), product entity category classification (PECC), and product entity aspect summary generation (PEASG). Among these objectives, KMLM and KMS2S learn predict the masked sin- gle and multiple tokens, respectively, that are associated with domain-specific knowledge rather than general information; PEABD determines the boundaries between descriptions of different product aspects given full product information; PECC identifies the product category that each product be- longs to; and PEASG generates summary for each individual product aspect based on the entire product description.
After pre-training K-PLUG, we fine-tune on three domain- speein--seific NLP tasks, namely, e- commerce knowledge base completion, abstractive product summarization, and multi-tury turn dialogue. The results show that K-PLUG significantly outperforms comparative models on all these tasks.
Our main contributions can be summarized as follows:
We present K-PLUG that learns domain-specific knowledge for both the encoder and the decoder in pre-training language model framework, which benefits both NLG and NLU tasks.
We formulate the learning of four types of domain-specing knowledge in e-commerce sce- narios: e-commerce domain-speciff knowledge-bait aspects of product entities, cate- gories of product entities, and unique selling propositions of product entities, which pro- vide critical information for many applications in the domain of e-commerce. Specifically, five sulf-supervised objectives are proposed to learn these four types of knowledge into unified PLM.
Our proposed model exhibits clear effectiveness in many downstream tasks in the e- commerce scenario, including e-commerce KB completion, abstractive product summa- rization, and multi-turn dialogue.
2 RELATED WORK
2.1 PLMs IN GENERAL
Unsupervised pre-training language model has been successfully applied to many NLP tasks.
ELMo (Peters et al.] 2018) learns the contextual representations based on a bidirectional LM.
GPT Radffrd et a 2018) predicts tokens based on the context on the left-hand side. BERT (Devlin
al. 2019) adopts bi-directional LM to predict the masked tokens. XLNNe (Yang et al.) 2019)
predicts masked tokens in permuted order through an autoregressive method. MASS (Song et al.) LM to recover span of masked tokens. UniLM (Dong et al. 2019) combines bidirectional, unidirectional, and sequence-to-sequee
2019) pre-trains the sequence-to-sequeee
LMs. T5 (Raffel 2019) and BART (Lewis et al., (020) present denoising sequence- toaeceetto-eeee to-sequence pre-training.
et al.
PEGASUS (Zhang et al. 2020) pre-train traitr with gap-sentence generation objective. While human- curated or domain -smaii-ppecific knowledge is essential for downstream knowledge-drive tasks, these methods do not explicitly consider external knowledge like proposed K-PLUG.
2

---

Under review as a conference paper at ICLR 2021
[Relggoosor
Category: Refrigerator
snowledge-basess
Type motortimerter
Encoder
Encoder
Encoder
[Maerrial loonner
Aspect 1:
Summary: crisper drawer with
vest window
Description: the intelligent oouli--
(b) Pre-training objective: KMLM
(d) Pre-training objective:
(e) Pre-training objective:
channel eir outle? saaie Roneer
witer
crispe
windore
Waler
Aspect
Summary: high quality condenser
Description: just cooling effect
Encoder
Decoder
Encoder
Decoder
and low www.mnm0oormmmmmm wmmpeetooottt copper
abomieum condenser...
(CAS)
conting
reduce
(a) Pre-mmining data
Pre-training objective: KMS2S
(1) Pre-training objective: PEASG
Pre-training data consists of 25 million textual product descriptions depicting multiple product aspects. We define knowledge as e-commerce knowledge-basee aspects of product enti- fies, categories of product entities, and unigue selling propositions of product entities. Pre-training objectives include knowledge-awer scquence-to-sequence (KMS2S), product entity aspect boundary detection (PEABD), product entity category classification (PECC), and product entity aspect summary generation (PEASG).
Figure 1:
masked language model (KMLM), wowledge-aware masked
2.2
INJECTING KNOWLEDGE INTO PLMS
Recent work investigates how to incorporate knowledge into PLMs for NLU. ERNIE (Sun et al. 2019) enhances language representation with the entity/phrase-level masking. ERNIE (Zhang al.
2019 identifies and links entity mentions in texts to their corresponding entities in KB. Similar ERNIE (Zhang al. 2019). KnowBERT (Peters al., 2019) injects KBs into PLM. Xiong itt al. (2020) leverages an entity replacement pre-training objective to learn better representations entities. KEPLER (Wang et al. 2019) adopts the knowledge embedding objective in the pre- training. Besides, SKEP (Tian et al. 2020). SenseBERT (Levine et al. 2020), SentiLR (Ke et al.) 2019). and K-ADAPTER (Wang et al. (020)) propose to integrate sentiment knowledge, word sense, sentiment polarity, and lexical relation into PLM, respectively. However, most of these studies are focused on integrating knowledge for language understanding task, work utilizing domain- specific knowledge for pre-training for language generation tasks limited. Inspired by these work, we construct K- K-PLUG that learns domain-specific knowledge into a PLM for both NLU and NLG tasks.
KNOWLEDGE-INECTED PRE-TRAINING
In this section, we explain the data used to pre-train K-PLUG, its model architecture, and our pre- training objectives.
3.1 DATA PREPARATION
We collect the pre-training data from mainstream Chinese e-commerce platform which contains approximately 25 million textual product descriptions and covers 40 product categories. With an av- erage length of 405 tokens, these product deseriptions constitute corpus with size of 10B Chinese characters. Each product description consists of information 10.7 product aspects on average, and each product aspect is accompanied with a summary highlighting its prominent features, as shown in Figure a).. Additionally, the e-commerce KB and USPs (further explained below) used in our pre-training data are as specified by the e-commerce platform and its online stores.
hhtps://wwwwjd.com/
3

---

Under review as a conference paper at ICLR 2021
3.2 MODEL ARCHITECTURE
K-PLUG adopts the standard sequencce-o-seequenee Transformer architecture (Vaswani et al. 2017). consisting of 6-layer encoder and 6-layer decoder as Song al. (2019). We the size of hidden vectors as 768, and the number of self-attention heads as 12. More details about the experimental settings are in the appendix.
3.3 KNOWLEDGE FORMULATION AND PRE-TRAINING OBJECTIVES
We formulate the learning of four types of knowledge in a unified PLM: e-commerce KB, aspects of product entities, categories of product entities, and USPs of product entities. Specifically, e- commerce KB stores standardized product attribute information, e.g., (Material: Cotton) and (Col- lar Type: Pointed Collar). It provides details about the products (Logan IV et al.) 2017). Aspects of product entities are features of product, such the sound quality of a stereo speaker, etc. eLi al., 2020). Categories of product entities such as Clothing and Food are widely used by e-commerce platforms to organize their products so to present structured offerings their customers (Luo al. 2020 Dong et al. 2020) USPs of product entities are the essence of what differentiates a product from its competitors (Garrett [961). For example, a stereo speaker's USP exhibiting its supreme sound quality could be "crystal clear stereo sound". An effective USP immediately motivates the purchasing behavior of potential buyers.
We propose and evaluate five novel self-supervised pre-training objectives to learn the above- mentioned four types of knowledge in the K-PLUG model (see Figure
Knowledge-aware Masked Language Model (KMLM)
Inspired by BERT (Devlin et al., 2019), we adopt the masked language model (MLM) to train the Transformer encoder as one of our pre-training objectives, which learns to predict the masked tokens in the source sequence (e.g., "The company is [MASK] BERT, mask 15% of all tokens in a text sequence; 80% of the masked tokens are replaced with the [MASK] token, 10% with a random token, and 10% left unchanged. Particularly, given an original text sequence masking Zm through one of the three ways explained above, egg., replacing Zm with [MASK] to create (Ii, [M.SK]..., MLM aims to model the conditional likelihood P(#m ()), and the loss function is:
the foot hill."). Similar to
(I1, ..... Tm....,.... with M tokens, masked sequence is produced by
LMLM = log Par x)
The major difference from BERT is that KMLM prioritizes knowledge tokens, which contain knowledge regarding product attributes and USPs, when selecting positions to mask and, in the case that the knowledge tokens make up less than 15% of all tokens, randomly picks non-knowledge tokens to complete the masking.
Knowledge-aware Masked Sequence-to-Sequene (KMS2S)
K-PLUG inherits the strong ability of language generation from the masked (MS2S) objective. The encoder takes a sentence with masked fragment (several consecutive to- kens) as the imput, and the decoder predicts this masked fragment conditioned on the encoder repre- sentations (e.g.." "The company [MASK] [MASK] [MASK] the foot of a hill.").
Given text sequence x (II...., [MASK], ..., [MASK]. ...,, XM) is produced by replacing the span Tuans ranging from 2., to with the [MASK] token. MS2S aims to model P( (T). which can be further factorized into a product P P( ()) II=_, ((x, (e) according to the chain rule. The loss function is:
(#1..., ..... Te. .... I... a masked sequence 2
L LM525 Σ log P( (2)
We set the length of the masked span 30% of the length of the original text sequence. Similar to KMLM, KMS2S prioritizes the masking of text spans that cover knowledge tokens.
4

---

Under review as a conference paper at ICLR 2021
Product Entity Aspect Boundary Detection (PEABD)
A product description usually contains multiple product entity aspects. Existing work (Li et al, 2020) proves that product aspects influence the quality of product summaries from the views of importance, non-redundancy, and readability, which are not directly taken into account in language modeling. In order train a model that understands product aspects, leverage the PEABD ob- jective to detect boundaries between the product entity aspects. It is essentially a sequence labeling task based on the representations of K-PLUG's top encoder layer.
Given a text sequence a = (Ii, TM.. the encoder of K-PLUG outputs a sequence = (h1 h,)), which is fed into softmax layer, and generates probability sequence y. The loss function is:
LPEABD = =-
Σ
Vi log yt
(3)
where y E {[0, 1]} the ground-truth labels for the aspect boundary detection task.
Product Entity Category Classification (PECC)
Product entity categories are the backbones for constructing taxonomies (Luo et al. 2020 Dong et al. 2020). Each product description document corresponds to one of the 40 categories included in our corpus, such as Clothing, Bags, Home Appliances, Shoes, Foods, etc. Identifying the product entity categories accurately is the prerequisite for creating an output that consistent with the input.
Given text sequence on the representation of the encoder classification token, [CLS]. The loss function maximizes the model's probability of outputting the true product entity category follows:
(I1,-..;; T.M. softmax layer outputs the classification score, y, based
LPECC = - log y
where ý is the ground-truth product category.
Product Entity Aspect Summary Generation (PEASG)
Inspired by PEGASUS (Zhang et al. 2020), which proves that using a pre-training objective that more closely resembles the downstream task leads to better and faster fine-tuning performance, we propose PEASG objective generate summary from the description of a product entity aspect. Unlike extracted gap-sentences generation in PEGASUS, our method constructs more realistic summary generation task because the aspect summary naturally exists our pre-training data.
Given an aspect description sequence æ = (#1. ,, and an aspect summary sequence = (y), ..., 1r), PEASG aims to model the conditional likelihood P(y (az).. The loss function is:
LPEASG Σ
log ( (x, yee)
(5)
4 EXPERIMENTS AND RESULTS
4.1 PRE-TRAINED MODEL VARIANTS
To evaluate the effectiveness of pre-training with domain-specific data and with domain-specific knowledge separately, we implement pre-training experiments with two model variants: C-PLUG and E-PLUG, whose configurations are the same as that of K-PLUG.
C-PLUG is a pre-trained language model with the original objectives of MLM and MS2S, trained with general pre-training corpus, CLUE (Xu et al. 2020 which contains 30GB of raw text with around 8B Chinese words.
E-PLUG is pre-trained language model with the original objectives MLM and MS2S, trained with our collected e-commerce domain-specific corpus.
4.2 DOWNSTREAM TASKS
We fine-tune K-PLUG on three downstream tasks: e-commerce KB completion, abstractive prod- uet summarization, and multi-tum dialogue. The e-commerce KB completion task involves the
5

---

Under review as a conference paper at ICLR 2021
prediction of product attributes and values given product information. The abstractive product sum- marization task requires the model to generate a product summary from textual product description. The multi-turn dialogue task aims to output the response by utilizing multi-turn dialogue context. The domain-specific knowledge we defined this paper is essential for these tasks.
4.2.1 E-COMMERCE KB COMPLETION
Task Definition. E-commerce KB provides abundant product information that is in the form of (product enfity, product aftribute, attribute value), such as (pidf Material, Copper Alu- minum). For the E-commerce KB completion task, the input is a textual product description for a given product, and the output is the product attribute values.
Dataset. We conduct experiments on the dataset MEPAVE (Zhu et al., 2020). This dataset collected from major Chinese e-commerce platform, which consists of 19.194 instances annotated with the position of attribute values mentioned in the product descriptions. There are totally 26 types of product attributes such Material, Collar Type, Color, etc. The training, validation, and testing sets contain 71, 11119994/00088..00 instances, respectively.
Model. We consider the e-commerce KB completion task as sequence labeling task that tags the input word sequence x = (Ii,..., FN) with the label sequence = "u.....''' in the BIO format. For example, for the input sentence "A bright yellow collar", the corresponding labels for "bright" and "yellow" are Color-B and Color-l, respectively, and O for the other tokens. For an input sequence, K-PLUG outputs an encoding representation sequence, and linear classification layer with the softmax predicts the label for each input token based on the encoding representation.
Baselines.
Table 1: Experimental results with the F1 score for the e-commerce KB completion task. The results in the first block are taken from Zhu
ScalingUp Xu et al. 2019) adopts BiLSTM, CRF, and attention mechanism extract attributes.
JAVE (Zhu et al. 22200) is joint attribute and value extraction model based on a pre-trained BERT.
et al. 2020
M-JAVE (Zhu et al. 2020) is multimodal JAVE model, which additionally utilizes product image in- formation.
Model
P
R
F1
LSTM
79.68
86.43
82.92
ScalingUp
65.48
93.78
77.12
BERT
78.27
88.62
83.12
Result. Table [ shows the experimental results. We observe that our K-PLUG performs better than baselines. C-PLUG achieves significantly better performance than BERT, which indicates that MS2S can also benefit the NLU task. E-PLUG outperforms C-PLUG, showing that training with domain- specific corpus is helpful. K-PLUG further exhibits 2.51%
JAVE
80.27
89.82
84.78
M-JAVE
83.49
90.94
87.17
C-PLUG
89.79
96.47
93.02
E-PLUG
89.91
96.75
93.20
K-PLUG
93.58
97.92
95.97
improvement compared with E-PLUG. In short, we can conclude that the improvement is due to both the Nommann-specife pre-training data and wewwledge-injecteee pre-training objectives.
4.2.2 ABSTRACTIVE PRODUCT SUMMARIZATION
Task Definition. Abstractive product summarization task aims to capture the most attractive infor- mation of a product that resonates with potential purchasers. The input for this task is a product description, and the output condensed product summary.
Dataset. We perform experiments on the dataset Li et al. (202),, which contains 1.4 million in- stances collected from a major Chinese e-commerce platform, covering three categories of product: Home Appliances, Clothing, and Cases & Bags. Each instance in the dataset is a (product infor- mation, product summary) pair, and the product information contains an image, title, and other product descriptions. In our work, we do not consider the visual information of products. Notice that the task of abstractive product summarization and product entity aspect summary generation (PEASG) are partly different. The abstractive product summarization task aims generate a com- plete and cohesive product summary given a detailed product description. Given a product aspect description, PEASG aims to produce an aspect summary that basically consists of condensed USPs. In addition, for abstractive product summarization task, the average length of the product summaries is 79, while the lengths of the product aspect summaries less than 10 in general.

---

Under review as a conference paper at ICLR 2021
Model. Abstractive product summarization task is an NLG task that takes the product description as the input and product summary as the output.
Baselines.
LexRank (Erkan & Radev
2004) is graph-based extractive summarization method.
Seq2seq (Bahdanau et al.
2015 is a standard seq2seq model with an attention mechanism.
Pointer-Generator (PG) See et al. 2017) is seq2seq model with copying mechanism.
Aspect MMPG (Li et al.
2020) is the-state-of oofute--f-thhaatt the-art method for abstractive product sum- marization, taking both textual and visual product information as the input.
Result. Table shows the experimental results, including ROUGE-1 (RG-1), ROUGE-2 (RG-2), and ROUGE-L (RG-L) F1 scores (Lin & Hovy] (003). K-PLUG clearly performs better than other text-based methods. E-commerce knowledge plays significant role in the abstractive product sum- marization task, and domain-specific pre-training data and Knowledge-injected pre-training objec- tives both enhance the model. K-PLUG achieves comparable results with the multimodal model, Aspect MMPG. The work Li et al. (2020) suggests that product images are essential for this task, and we will advance K-PLUG with multimodal information the future.
Table 2: Experimental results with the ROUGE score for the abstractive product summarization task, The results in bold are the best performances among the models taking only texts as the input, and denotes the model taking both product images and texts as the input. The results in the first and second blocks are taken from Li et al. (2020).
Home Applications RG-2
Clothing RG-2
Cases& &Bas
Model
RG-1
RG-L
RG-1
RG-L
RG-1
RG-2
RG-L
LexRank
24.06
10.01
18.19
26.87
9.01
17.76
27.09
9.87
18.03
Seq2seq
21.57
7.18
17.61
23.05
6.84
16.82
23.18
6.94
17.29
MASS
28.19
8.02
18.73
26.73
8.03
17.72
27.19
9.03
18.17
PG
31.11
10.93
21.11
29.11
9.24
19.92
31.31
10.27
21.79
Aspect MMPG*
34.36
12.52
22.35
31.93
11.09
21.54
33.78
12.51
22.43
C-PLUG
32.75
11.62
21.76
31.73
10.86
20.37
32.04
10.75
21.85
E-PLUG
33.11
12.07
22.01
32.61
11.03
20.98
32.37
11.14
21.98
K-PLUG
33.56
12.50
22.15
33.00
11.24
21.43
33.87
11.83
22,35
4.2.3 MULTI-TURN DIALOGUE
Task Definition. The multi-tur turn dialogue task aims to output response based the multi-turn dialogue context (Shum et al. 2018). The input for this task is the dialogue context consisting of previous question answering, and the output is the response to the last question.
Dataset. We conduct experiments on two datasets JDDC (Chen et af. 2020) and ECD (Zhang al., 2018). JDDC is collected from the conversations between users and customer service staffs from a popular e-commerce website in China and contains 289 different intents, which are the goals of dialogue, such updating addresses, inquiring prices, etc, from after-sales assistance. There are 1,024,196 multi-turn sessions and 20,351,337 utterances in total. The average number of turns for each session is 20, and the average tokens per utterance is about 7.4. After pre-processing, the train- ing, validation, and testing sets include 1,522,859/5,0000 0000 (dialogue context, response) pairs, respectively. FCD collected from another popular e-commerce website China and covers over 5 types of conversations based on 20 commodities. Additionally, for each ground-truth response, negative responses are provided for discriminative learning. The training, validation, and testing sets include 0w..000001000001000
(dialogue context, response) pairs, respectively.
Model. We test with two types of K-PLUG: retrieval-based K-PLUG on the ECD dataset and generative-based K-PLUG on the JDDC dataset. For the retrieval-baid based approach, concatenate the dialogue context and use [SEP] token to separate context and response. The [CLS] representa- tion is fed into the output layer for classification. The generative-basee approach sequence-to-

---

Under review as a conference paper at ICLR 2021
sequence model, which is the same the model adopted in the abstractive product summarization task.
Baselines.
The basclines also include both the retrieval berseval-based (BM25, CNN, BiLSTM, and BERT) and generative-basee approaches. Other baselines are as follows.
SMN Wu et al. 2017)
matches response with each utterance in the context.
2018) is a deep utterance aggregation model based on the fine-grained DUA Zhang et al.
context representations.
DAM (Zhou et al. information based on self-attention and cross-attention.
2018) matches response with the context based using dependency
2019) is a deep matching model by stacking multiple interactions blocks Iol TTo et al.
between utterance and response.
MSN (Yuan et 2019 selects relevant context and generates better context representa-
tions with the selected context.
Result. Table 3 and 4 show the experimental results on the JDDC and ECD datasets, respectively. We report ROUGE-L (RG-L) FI, BLEU, and recall at position in n candidates (R, @k). We can observe that, both the retrieval based and generative- the-art results, and e-commerce knowledge presents consistent improvements. K-PLUG is evidently superior to BERT, possibly due to BERT's lack of domain-spectin knowledge for pre-training with the general MLM objective.
tasks, K- K-PLUG achieves new state-of-
We further perform human evaluation the JDDC dataset. We randomly choose 100 samples from the test set, and three experienced annotators are involved to determine whether K-PLUG outperforms E-PLUG with respect (1) relevance between the response and the contexts and (2) readability of the response. The results are shown in Table 5 We can see that the percentage of "Win", which denotes that the results of K-PLUG is better than E-PLUG, is significantly larger than "Lose" (p-value < 0.01 for t-test). Kappa values (Feiss 1971) confirm the consistency for different annotators.
Table 4: Experimental results for the multi- turn conversation task on the ECD dataset. The results in the first block are taken from Zhang et al. (0118)
Table 3: Experimental results for the multi- turn conversation task the JDDC dataset. The results in the first block are taken from
Chen al. 2202))
Model
Rio @1
R10@?
810
CNN
32.8
51.5
79.2
Model
RG-L
BLEU
BiL.TMM
35.5
52.5
82.5
BM25
19.47
9.94
SMN
45.3
65.4
88.6
BERT
19.90
10.27
DUA
50.1
70.0
92.1
Seq2Seq
22.17
14.15
DAM
52.6
72.7
93.3
PG
23.62
14.27
Tol-local
56.3
76.8
95.0
MSN
60.6
77.0
93.7
C-PLUG
25,47
16.75
BERT
54.3
73.4
94.3
E-PLUG
25,93
17.12
K-PLUG
26.60
17.80
C-PLUG
62.7
76.8
95.0
E-PLUG
65.8
80.1
95.6
K-PLUG
73.5
82.9
96.4
Table 5: Human evaluation results (%). "Win" denotes that the generated response of K-PLUG is better than E-PLUG.
Relevance
Readability
Win
Lose
Tie
Kappa
Win
Lose
Tie
Kappa
29.00
21.00
50.00
0.428
7.00
2.00
91.00
0.479
8

---

Under review as a conference paper at ICLR 2021
4.3
ABLATION STUDIES
To better understand our model, we perform ablation experiments to study the effects of different pre-training objectives.
Result. The ablation results are shown in Table We can conclude that the lack of any pre- training objective hurts performance across all the tasks. KMS2S is the most effective objective for the abstractive product summarization and generative conversation tasks since this objective is highly close the essence of NLG. Product-asppect-rllate objectives, i.e., PEABD and PEASG, contribute much to the abstractive product summarization task, which proves that this task requires comprehensively understanding the product description from the view of product aspects, going beyond individual tokens.
Table 6: Experimental results for ablation studies.
KB Completion F1
Abstractive Product Summarization
Multi-Turn
Model
Home Applications
Clothing
Cases& Bags
Conversation
RG-1
RG-2
RG-1
RG-2
RG-1
RG-2
RG-L
BLEU
K-PLUG
95.97
33.56
12.50
33.00
11.24
33.87
11.83
26.60
17.80
KMLM
95.88
33.52
12.43
32.87
11.20
33.75
11.70
26.43
17.62
KMS2S
95.76
33.13
12.14
32.12
10.97
33.74
11,43
25.82
16.97
-PABBD
95.89
33.26
12.30
32.96
11.14
33.69
11.17
26.07
17.58
-PECC
95,59
33.24
12.17
32.25
11.12
33.59
11.18
26.02
17.16
-PEASG 95.48
33.39
12.36
32.57
11.16
33.78
11.45
26.12
17.38
CONCLUSION
We present knowledge-injected pre-trained model (K-PLUG) that is powerful domain-specific language model trained on a large-scale e-commerce corpus designed to capture e-commerce knowl- edge, including e-commerce KB, product aspects, product categories, and USPs. The pre-training framework combines masked language model and masked seq2seq with novel objectives formu- lated as product aspect boundary detection, product aspect summary generation, and product cat- egory classification tasks. Our proposed model demonstrates strong performances on both natural language understanding and generation downstream tasks, including e-commerce KB completion, abstractive product summarization, and multi-turn dialogue.
REFERENCES
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. 3rd International Conference on Learning Representations, (ICLR), 2015.
Meng Chen, Ruixue Liu, Lei Shen, Shaozu Yuan, Jingyan Zhou, Youzheng Wu, Xiaodong He, and Bowen Zhou. The jddc corpus: A large-scale multi-turn chinese dialogue dataset for e-commerce customer service. Proceedings of The 12th Language Resources and Evaluation Conference (LREC), pp. 459-466, 2020.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter the Association for Computational Linguisties: Human Language 2019.
Technologies (NAACL). pp.
Li Dong, Nan Yang, Wenhui Wang. Furu Wei, Xiaodong Liu, Yu Wang, Jianfeng Gao, Ming Zhou, and Hsiao- Wuen Hon. Unified language model pre-training for natural language understanding and generation. In Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems, (NeurIPS), pp. 13042-13054, 2019.
Xin Luna Dong, Xiang He, Andrey Kan, Xian Li. Yan Liang, Jun Ma, Yifan Ethan Xu, Chenwei Zhang, Tong Zhao, Gabriel Blanco Saldana, al. noto Self-driving knowledge collection

---

Under review as a conference paper at ICLR 2021
for products of thousands of types. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining (KDD), pp. 1124-2734, 2020.
Günes Erkan and Dragomir Radev. LexRank: Graph-based lexical centrality as salience in text summarization. Journal of Artificial Intelligence Research (JAIR), 22:457-477 044474479, 2004.
Joseph I. Fleiss. Measuring nominal scale agreement among many raters. Psychological bulletin, 76(5):378, 1971.
Thomas M Garrett. Reality in advertising. America, 1961.
Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (gelus).
arXiv preprint
arXiv: 08415, 2016.
Pei Ke, Haozhe Ji, Siyang Liu, Xiaoyan Zhu, and Minlie Huang. SentiLR: Linguistic knowledge enhanced language representation for sentiment analysis. arXiv preprint arXiv: arXiv:191.002:. 191244499, 2019.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In 3rd Interna- tional Conference on Learning Representations, (ICLR), 2015.
Anne Lauscher, Ivan Vulić, Edoardo Maria Ponti, Anna Korhonen, and Goran Glavaš. Informing unsupervised pretraining with external linguistic knowledge. arXiv preprint arXiv: (rXXv:1909002399 2019.
Yoav Levine, Barak Lenz, Or Dagan, Ori Ram, Dan Padnos, Or Sharir, Shai Shalev-Shwartz, Amnon Shashua, and Yoav Shoham. SenseBERT: Driving some sense into BERT. In Proceedings the 58th Annual Meeting of the Association for Computational Linguistics (ACL), pp. 4656-4667, 2020.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. BART: Denoising sequence-t--scqueeec pre- training for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (ACL), pp. 7871-7880, 2020.
Haoran Li, Peng Yuan, Song Xu, Youzheng Wu, Xiaodong He, and Bowen Zhou Aspect-aware multimodal summarization for chinese e-commerce products. In Proceedings of the Thirty-Forth AAAI Conference on Artificial Intelligence (AAAI), pp. 8188-8195, 2020.
Chin-Yew Lin and Eduard Hovy. Automatic evaluation of summaries using n-gram co-occurrence statistics. In Proceedings of the 2003 Human Language Technology Conference of the North American Chapter of the Association for Computational Linguistics (NAACL), pp. 150-157, 2003.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. RoBERTa: A robustly optimized BERT pre- training approach. arXiv preprint a90iv 1692, 2019.
Robert I. Logan IV, Samuel Humeau, and Sameer Singh. Multimodal attribute extraction. arXiv preprint arXiv:1711.1111 2017.
Xusheng Luo, Luxin Liu, Yonghua Yang, Le Bo, Yuanpeng Cao, Jinghang Wu, Qiang Li, Keping Yang, and Kenny Q Zhu. AliCoCo: Alibaba e-commerce cognitive concept net. In Proceedings of the 2020 ACM SIGMOD International Conference on Management of Data (SIGMOD). pp- 313-327, 2020.
Matthew Peters, Mark Neumann, Mohit lyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. Deep contextualized word representations. In Proceedings of the 2018 Con- ference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL), pp. 2227-2237, 2018
Matthew E. Peters, Mark Neumann, Robert Logan, Roy Schwartz, Vidur Joshi, Sameer Singh, and Noah A. Smith. Knowledge enhanced contextual word representations. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th Interna- tional Joint Conference Natural Language Processing (EMNIPP-IJCNLP)) pp. 43-54, 2019.
10

---

Under review as a conference paper at ICLR 2021
Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language under- standing by generative pre-training 2018.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint arXiv: 10683, 2019.
Abigail See, Peter J. Liu, and Christopher D. Manning. Get to the point: Summarization with pointer-generator networks. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (ACL), pp. 1073-1083, 2017.
Heung-Yeune Yeung Shum, Xiao-dong He, and Di Li. From Eliza to Xiaoice: challenges and opportunities with social chatbots. Frontiers of Information Technology & Electronic Engineering, 19:10-26, 2018.
Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, and Tie- Yan Liu. MASS: masked sequence to sequence pre-training for language generation. In Proceedings of the 36th International Conference on Machine Learning (ICML), pp. 5926-5936, 2019.
Yu Sun, Shuohuan Wang. Yukun Li, Shikun Feng, Xuyi Chen, Han Zhang, Xin Tian, Danxiang Zhu, Hao Tian, and Hua Wu, ERNIE: Enhanced representation through knowledge integration. arXiv preprint arXiv: 1904:990400 1ww..19444....3 2019.
Chongyang Tao, Wei Wu, Can Xu, Wenpeng Hu, Dongyan Zhao, and Rui Yan. One time of in- teraction may not be enough: Go deep with an concction-ove-inttraattte network for response selection in dialogues. In Proceedings of the 57th Anmual Meeting of the Association for Compu- tational Linguistics, pp. 1-11, 2019.
Hao Tian, Can Gao, Xinyan Xiao, Hao Liu, Bolei He, Hua Wu, Haifeng Wang. and Feng Wu. SKEP: Sentiment knowledge enhanced pre-training for sentiment analysis. In Proceedings of the 58th Anmual Meeting of the Association for Computational Linguistics (ACL), pp. 4067 2020.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances Neural Infor- mation Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017.
(NeurIPS), pp.
Ruize Wang. Duyu Tang, Nan Duan, Zhongyu Wei, Xuanjing Huang, Cuihong Cao, Daxin Jiang, Ming Zhou, et al. K-Adapter: Infusing knowledge into pre-trained models with adapters. arXiv preprint ar arXiv:2002.00 01 000, 2020.
Xiaozhi Wang, Tianyu Gao, Zhaocheng Zhu, Zhiyuan Liu, Juanzi Li, and Jian Tang. KEPLER: A unified model for knowledge embedding and pre-trained language representation. arXiv preprint arXiv:1911.06136 2019.
Yu Wu, Wei Wu, Chen Xing, Ming Zhou, and Zhoujun Li. Sequential matching network: A new architecture for multi-turn response selection in retrieval-based chatbots. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (ACL), pp. 496-505, 2017.
Wenhan Xiong, Jingfei Du, William Yang Wang, and Veselin Stoyanov. Pretrained encyclopedia: language model. In 8th International Conference on
Weakly supervised Learning Representations (ICLR), 2020.
Huimin Xu, Wenting Wang, Xin Mao, Xinyu Jiang. and Man Lan. Scaling up open tagging from tens to thousands: Comprehension empowered attribute value extraction from product title. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics (ACL), pp. 5214-5223, 2019.
Liang Xu, Xuanwei Zhang, Lu Li. Hai Hu, Chenjie Cao, Weitang Liu, Junyi Li. Yudong Li, Kai Sun, Yechen Xu, et al. CLUE: A chinese language understanding evaluation benchmark. arXiv preprint arXiv:2004.05900 2020.
11

---

Under review as a conference paper at ICLR 2021
Yu Yan, Weizhen Qi. Yeyun Gong, Dayiheng Liu, Nan Duan, Jiusheng Chen, Ruofei Zhang, and Ming Zhou, ProphetNet: Predicting future n-gram for sequence-to-squenne pre-training. arXiv preprint orXiv:2001.04063 2020.
Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le. XLNET: Generalized autoregressive pretraining for language understanding. In Advances in Neu- ral Information Processing Systems 32: Annual Conference on Neural Information Processing Systems (NeurIPS), pp. 5753-5763, 2019.
Chunyuan Yuan, Wei Zhou, Mingming Li, Shangwen Lv, Fuqing Zhu, Jizhong Han, and Songlin Hu. Multi-hop selector network for multi-turn response selection retrieval-based chatbots. In Proceedings of the 2019 Conference Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp- 111-120, 2019.
Jingqing Zhang, Yao Zhao, Mohammad Saleh, and Peter Liu. PEGASUS: Pre-training with tracted gap-sentences for abstractive summarization. In Proceedings of the 37th International Conference on Machine Learning (ICML), 2020.
Zhengyan Zhang, Xu Han, Zhiyuan Liu, Xin Jiang, Maosong Sun, and Qun Liu, ERNIE: Enhanced language representation with informative entities. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics (ACL), pp. 1441-1451, 2019.
Zhuosheng Zhang, Jiangtong Li, Pengfei Zhu, Hai Zhao, and Gongshen Liu. Modeling multi-turn conversation with deep utterance aggregation. In Proceedings of the 27th International Confer- ence on Computational Linguistics (COLING), pp. 3740-3752, 2018.
Xiangyang Zhou, Lu Li, Daxiang Dong, Yi Liu, Ying Chen, Wayne Xin Zhao, Dianhai Yu, and Hua Wu. Multi Multi-turn turn response selection for chatbots with deep attention matching network. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1118-1127, 2018
Tiangang Zhu, Yue Wang. Haoran Li, Youzheng Wu, Xiaodong He, and Bowen Zhou. Multimodal joint attribute prediction and value extraction for e-commerce product. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2020.
A APPENDIX
A.1 EXPERIMENTS SETTINGS FOR PRE-TRAINING
We adopt GELU activation (Hendrycks & Gimpel 2016) in GPT (Radford et al. 2018). We use Adam optimizer (Kingma & Ba| 2015) with a learning rate of 5e-4, 31 = 0.9, 33== 0.98, T.2 weight decay of 0.01, learning rate warm-up over the first 10,000 steps and linear decay of the learning rate. The dropout probability is 0.1. The maximum sequence length is set to 512 tokens. Pre-training was performed with 4 Telsa V100 GPUs The pre-training is done within 10 epochs, which takes around 10 days, and the fine-tuning takes up to day. We use the beam search with a beam size of 5 for inference for the NLG tasks. Other hyper- syper-parameters can be found in code.
12

---

Under review as a conference paper at ICLR 2021
A.2 CASE STUDIES
We present some examples from the test set of each task, with comparisons of the ground-truth result and the outputs produced by the models of E-PLUG and K-PLUG.
Table 7: Case study for the e-commerce KB complete task. The product attribute and the corre- sponding attribute value is presented as [attribute value] Ialuelproderr rately complete the e-commerce KB, while the E-PLUG model sometimes fails. The translation texts are given below the original examples.
product auribute- The K-PLUG model accu-
Ground-truth
ECCO Bj i [i] [ij [AS Mld ya
(ECCO's non-sll son-slip simple [ankle boots height)
E-PLUG
ECCO Bj h [i] 25 jej IE Woym
(ECCO's s non-si slip simple [ankle thets height)
K-PLUG
ECCO DJ 1m [ii]
(ECCO's non-slip simple [ankle boots Ishat beight
Ground-truth
a21 [[ J su [S] K
(A21's [sports] style windbreaker jacket with [contrasting color]aesipe style)
E-PLUG
a21 [ Jutt [ f [
K<
(A21's sports [windbreaker]ang ltyle jacket with [contrasting color stylelayle
K-PLUG
a21 [[ | x K
(A21's sports (windbreakerraayee jacket with [contrasting color]aesign style)
Ground-truth
[ [f (5 1f |
(made from Now-strech]elassieee [cotton fabric]maerial for a silky smooth touch)
E-PLUG
( Jass [10 11 i5 1t [HT #4 J Re 1i
(made from [low-strech] pants
[cotton fabric Immerial for a silky smooth touch)
K-PLUG
A (+ Jaae 16 fi ME
(made from [low-strech] www-strechJelastiiiy lelustictty
[cotton mmbtic]
material for a silky smooth touch)
13

---

Under review as a conference paper at ICLR 2021
Table 8: Case study for the abstractive product summarization task (Home Applications category). The K-PLUG model generates summaries describing more information about e-commerce knowledge bases and unique selling propositions
pew.iin.eeopppsstionssssssssssso of product entities.
Input
B MA EEAD, 310pa I 1732 ( 1it 53dbBP 3 BEAM B 7
|e ty
Description
[ SR...
(Sacon's 's European style kitchen range hood. Extra powerful ventilation. Intelligent touch control. Stainless steel. Superior quality with extra reliability, 310 air pressure. 17 m suction. Protection against head bumping. Limited noise of 53 dB. Highly powered. Captures large volume of fumes. Leaves no cooking smells ...)
Ground-truth
310p0 AD L T ! (This kitchen range hood with extra suction and ventilation is ideal for a clean and fresh kitchen. Perfectly combining 3/0pa of air pressure and 17m" of suction power, it filters out grease before you realize it. Never have to worry again about keeping your loved ones from cooking fumes.
1 AL BEA [n] MR a "
E-PLUG
2 HELL BL, B
I #T (I
FR F
(This kitchen range hood from Sacon is made of stainless steel with cutting edge technology. Its design adds contemporary style to your European style kitchen. It fcatures extra suction and yentilation for immediate filtering of fumes and grease. Its intelligent touch control panel makes extremely easy use.
K-PLUG
1 B4 310pa 53 DI A 11:0 fhe St (This stainless steel kitchen range hoad features extra suction and ventilation of 17 m² for immediate filtering of grease and fumes, leaving your kitchen fresh. Its 310 pa air pressure ensures clear filtering without backflow. Its low working noise of 53 dB keeps your kitchen nice and quiet.)
A P 11
Input Description
# S J pu
* at.. #
M 7 #4 LED E
T
JL, $ # A 1 4
(Canbo's affordable SL thermal control natural gas residential kitchen and bathroom water heater. Provides warm water for doing dishes and cleaning up in winter. High-quality accessories. High concentration orgeen-free copper tank. LED digital screen. Closed boiling system for safety. Easy temperature control. Multiple security checks. Oxygen-free free copper tank. Lets you enjoy your showers...
Ground-truth
XHE
S +
7 PA WE TH
(High concentration oxygen-free tank with added heating area boils water quickly with stability. Closed boiling technigue increases efficiency and saves energy. Thermal control and LED. digital
screen for adjustable water temperature anytime.) )
E-PLUG
X
# * 1 A
T EX H
T FURIEER E
(This gas water heater has large volume and provides warm water year round. It adopts the closed boiling technique for l'aster warming up and shorter wait time in shower. It works in low pressure condition and is suitable for older neighborhoods. )
K-PLUG
BALLDD M IL J
MERF HORLSHA
(Equipped with LED digital screen for clear thermal control and temperature adjustment. Closed boiling technique speeds up warming and increases efficiency. High concentration oxygen-free copper tank for faster warming up.)
14

---

Under review as a conference paper at ICLR 2021
Table 9: Case study for the abstractive product summarization task (Clothing category). The K- PLUG model generates summaries describing more information about e-commerce knowledge bases and unique selling propositions of product entities.
Input Description
7 MT $ B # #trin. E An 1 M K f सU
Hi Mi TE テス ii M
Bith B. प TE 557.. H XMj
(K-Boxing's men's t-Shirt. This summer's new arrival. Contrasting colors. Soft and comfortable.
Flower print. Short-sleeve. Black. Business casual. Crewneck. Summer. Urban style. Short-sleeve t-shirt. Naturally draping with high durability. Stylish crewneck. Basic and causal. Multiple
color chioces. Modal knit fabric. Soft and clastic. Loose fit. Flower print. Glitter embroidery. Stylish and casual ....)
Reference
(Made from modal knit fabric. Incredibly soft and clastic. Features a slightly looser fir. Natural draping with high durability. Decorated with flower print and glitter embroidery for a stylish and casual look. Finished with a classy crewneck, oppealing to the modern sense of fashion)
E-PLUG
M A of ni 45 dot RY di BE 21 -
it
(The crewneck design helps in clongating the neck. The light wash color gives an attractive yet effortless look. It dresses you up in a country asethetic style, comfy and relaxed just like the summer breeze.)
K-PLUG
F. 1 THUE..
(Made from modal knit fabric. Incredibly seft and stretchy. Slightly looser füt Flower print and glitter embmiddry. Stylish and casual. The basic crewneck design easily helps in elongating the neck.
Input
#.. Flll y). KK 9059. 43 T
3 # BAAk. RUMAET MU A 0
buse Iu 76 1 it.
Description
(Jeep's men's 's down jacket. Mid-thigh length. Filled with 90% down. Winter jacket. This winter's new arrival. Hoodedd. The down fill provides extra warmth. Warm down jacket. Black. White duck down. Urban style. Built for perfect look. Designed with artistic details. Creating a legend for 76 years. A must-have to keep up with current fashion trends. Keeps you warmer than ever...
Reference
*HA S
KEE TE EO R
(Made from polyester. Feel Feelseft, lightweight, and breathable. Keeps you dry and comfortable.
Filled primarily with white duck down, fluffy and light. Features thick yet slim-fit design. Keeps you warm in cold climates. )
E-PLUG
MEET X
(This down jacket features a mid-thigh length. keeping stylish silhouette and giving you an elegant
and mature look. The hood and letter print on the jacket make you look younger.
K-PLUG
* Te Hi HREES. Ki M TA SHLE.
A E. t 1 (Made from polyester. Soft and breathable. Filled primarily with white duck down, flfff and light.
Lightweight but warm. Features a hooded design. Locks in the heat and keeps out the cold Comfortable
to wear.
15

---

Under review as a conference paper at ICLR 2021
Table 10: Case study for the abstractive product summarization task (Bags& Cases category). The K- PLUG model generates summaries describing more information about e-commerce knowledge bases and unique selling propositions of product entities.
Input Description
# म A1 tr, BI Hu EL K $J #
Hy f.. S te yit E t X logo 78
- #
dd. A AE K 1 /
(Fion. This fall's new arrival. Corss body bag for women. Flower print magnetic snap closure. Square-shaped. Stylish tote bag. Well made long bag straps. Finished with flat lock stiching. Zippers produced by YKK. Flat and smooth surface. Anti-fading coloring. Made from imported
PVC coated fabric. Logo print fits nicely. Soft and comfortable. Water-proof and durable.
Lightweight leather shoulder strap. Adjustable length. 2 ways to carry. Finished with flat and even flatlock stiching.)
Reference
A ii IT. R K
(Made from imported PVC coated fabric. Soft and comfortable. Water:proof and durable. Specially made smoothly meshing zippers. Long shoulder strap with adjustable length, made with leather for
light weight and reduced pressure. ways to wear, cross body orwith the top handles.) )
E-PLUG
H T JE #IEE
KO 2 # # IE
(This tote bag is made friom imported high-qiuality PVC coated fabric. It has delicate texture and flower print decorations, giving it an elegant look. can hold many items, making it ideal for every day use.)
K-PLUG
H TEERTE KITT BA
F. KE Ln A 3
(Made from imported PVC coated fahric. Logo print fits nicely. Soft and comfortable, Water-proof and durable, Leather shoulder strap for light weight and reduced pressure with adjustable length. 2 ways to wear. cross body or with the top handles. Flat and even flatlocking finish shows its high quality. )
Input Description
NT M..
2 ( HL. F ZR TA T 13 3 677TH E
WEST. #TB
M
HA HU
(American' Tourister's 's aluminum frame suitcase. For men and women. 20 Inch. Carry-on luggage. Spinner wheels. Essential for the creative journey. Strong and durable. Breaks the limits of time and style. Stable and effortless steering thanks to the doubled wheel design. Includes dividers for easy organization of packed items. Extra stability with corner protection. Triple locks for added security. Stylish design without compromising on durability. Classic yet fashionable. Closes and seals nicely.
Reference
M EL V BE M.
11
(Conrer protection gives and basic and simple look withadded stability and durgbility. Aluminum alloy frame makes it strong and close and seal nicely. Triple locks for added stability and security. Includes dividers for casy and organized packing.)
E-PLUG
it + RI BE
A H1 HREMM
AA SARHH
(This suitcase in black suitable for various occasions. It is made from high-quality materials with delicate texture and [increased durability]. This large suitcase is built to provide room for extra items.)
K-PLUG
AL#
3 #
(Made with aluminum alloy frame for added stability. Closes and seals nicely. Triple-lock design to provide extra security. Stable and effortless steering thanks to the 8-wheel design. TAS-approved locks
protect your items and cause no damage during security check.)
16

---

Under review as a conference paper at ICLR 2021
Case study for the multi-turn dialogue task the ECD dataset. The K-PLUG model produces more accurate responses for the questions related to e-commerce knowledge bases.
Table 11:
Q1
(Is it paper-peel walnut?)
(Yes, is.)
Q2
(Is it of premium quality?)
A2
f E
(What do you mean?)
A
(Aren't these walnuts sized differently? I'm getting kilos. Are you still here?)
E-PLUG
88 Fi FR R AA
(This is paper-peel walnut, which you can peel casily with bare hands.)
K-PLUG
(They mostly have diameter between 3-5 cm.)
Q1
1 R
(What's the difference between these two?)
A1
2: - li t H8 AR 1520
(Hi, cotton one above has size 1520.)
Q2
A= [
(These three only differ size?)
E-PLUG
(Sure. That's exactly the difference between these three wiping products. They are made for the purpose.)
K-PLUG
r[jj (f) 1820 FA of in
(The middle one made nonwoven fabric size 1820, and bottom one made natural fiber, with size of 2020, extra large and thick.)
Q1
(Do you offer free shipping for this tea knife?)
A1
(Yes we do.)
Q2
(How much is it?)
E-PLUG
= (It's because we have this set price for all orders below 30 boxes.)
1/ T
K-PLUG
#
(It's 48 yuan for single tea knife.)
17


Reviews:
[{"review_id": "p4kqwBQuUgJ", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\n#### Summary\n\nThis paper introduces K-PLUG, an in-domain transformer-style pretrained language model. The domain on focus is e-commerce.\n\nTo enforce the PLM to be domain-aware, the authors identify several key types of knowledge for e-commerce: knowledge-bases, product aspects, product categories and product unique selling propositions (USP). They design specific pretraining objectives based on each type of knowledge.\n\nExperiments on several downstream tasks exhibited the usefulness of the proposed model.\n\n#### Strength\n\n- The idea of domain-specific PLM is likely to be useful in practice but not widely explored. Specifically, in the e-commerce field people often have heavy needs for automated text processing and also a large amount of data, making this line of work more useful in practice.\n- The pretraining objectives are properly designed, as they capture domain knowledge and have supervisions from the large-scale pretraining dataset. The experiment results also justified the design.\n\n#### Weakness\n\n- The proposed method (objectives) is hard to transfer to other domains. More importantly, it requires highly-structured and large-scale in-domain data for pretraining, which might not be available in other domains.\n- Some downstream tasks are too similar to pretraining tasks. For instance, Abstractive Product Summarization looks very similar to Product Entity Aspect Summary Generation (PEASG) in pretraining, given that the image modality is ignored. I understand that if pretraining tasks are similar to downstream tasks, the performance on downstream tasks will be better. However, such similarity limits the ability of the experiments on downstream tasks to show the usefulness of the pretrained model. In other words, when the tasks are too similar, it becomes less clear whether the performance gain comes from the pretraining method or merely a larger training set. Also, have you tried directly using the pretraining dataset in downstream tasks (it looks possible for E-commerce KB Completion and Abstractive Product Summarization)?\n\n#### Questions & Suggestions\n\n- In Figure 1a, in the pretraining data aspect descriptions, the knowledge-base values and USPs are highlighted. How are these labeled? Are they directly available from the platform?\n- For the pretraining dataset and downstream task datasets, each of them is said to be collected from \"a Chinese e-commerce platform\". Are they from the same or different platforms? It might be more clear to explicitly provide the platform names.\n- In the formulation of KMLM and KMS2S (page 4) you mentioned \"knowledge tokens\", what are \"knowledge tokens\"? Are they knowledge-base tokens and/or USP tokens?\n- For multi-turn dialog, since it is highly user-oriented, it would be better to have human evaluations.\n\n#### Typos\n\n- Page 5, PEASG, Line 6: \"a aspect\"\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "nqECRKnCCE5", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\nOverview:\n\nThe authors introduce K-PLUG, a knowledge-injected language model with domain-specific knowledge for both NLU and NLG tasks. They evaluate their method in e-commerce scenarios. They use five pre-training objectives, including domain-specific knowledge-bases, entities aspects, entities categories, and entities selling propositions. \n\n\nReasons to accept:\n* The authors present K-PLUG that pre-trained on Chinese e-commerce data, which contains approximately 25 million textual product descriptions.\n\n* The authors evaluate three different downstream tasks, e-commerce KB completion, abstractive product summarization, and multi-turn dialogue, and show good results compared to its baselines. \n\nReasons to reject:\n* The paper is not easy-to-follow with too many abbreviations.\n\n* The main concern of this work is: the comparison to existing works is too weak. As mentioned in Section 2.2, there are many related works also working on injecting knowledge or retrieving knowledge to improve language model pretraining. For example, KEPLER, SKEP, SenseBERT, and K-ADAPTER. However, the authors do not compare their training objectives/strategies to theirs, it is very hard for us to make a conclusion that which is better. It is not an apple-to-apple comparison. The only baselines to compare within this paper is C-PLUG and E-PLUG.\n\n* The dataset used to pre-train the model contains expensive human annotation, which is not scalable. For example, the category labels (used for Product Entity Category Classification) and the summary (used for Product Entity Aspect Summary Generation). In other words, this kind of makes the pre-training process like a \"multi-task\" learning approach with human annotations, which in my opinion is hard to collect more data. \n\n* The ablation study results that are shown in Table 5 basically suggest that many of the used objective functions are not important and could be ignored as the difference with and without is really marginal (I am sure some of them are not significantly better and could be the same if the authors tune a bit the hyper-parameters).\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "bVdAfqq9tT", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: This paper proposes pretraining language model for e-commerce domain. Specifically, the authors design five pretraining objectives to incorporate various domain knowledge into the the models with an encoder-decoder architecture. When further finetuned on language understanding and generation tasks in the e-commerce domain, the proposed models named K-PLUG outperforms the existing baseline models including those pretrained on general domains. The paper is generally easy to follow. Designs of the pretraining objectives are reasonable and empirically effective. Experiments are solid and convincing.\n\nPros:\nThe paper demonstrates a good practice on how to pretrain a language model for a specific domain by injecting a variety of structured and unstructured domain knowledge. Experiments verify that the knowledge-aware pretraining is indeed effective when you want to boost the performances of the tasks in specific domain. The experiments are well designed and quite comprehensive.\n\nConcerns:\n1. The paper is generally lack of novelty and inspiration, however. Most of the pretraining objectives have been tackled more or less in previous work. It is not surprising that when pretraining on the large amount of domain-specific data, the performance on the downstream tasks will improve. It will be more inspiring if the authors can give some high-level principles. Is it possible to design a more general framework that can work well when shifting to a new domain? What are the principles of designing pretrianing objectives with regard to domain knowledge and is there any way to evaluate them without need of expensive pretraining and finetuning?\n2. The presentation of paper could be improved, especially for section 3. For example, from the notation it is unclear which training objectives are defined on encoder and which are on both encoder and decoder. Also in figure 1, there seems no clear distinction between the encoder and the decoder. \n\nAdditional questions: \n1. How the five objectives are weighted in the pretraining? \n2. What are the “knowledge-tokens” and “non-knowledge” tokens in KMLM and KMS2S? Are the results sensitive to them?\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "YoBOodOR02H", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes K-PLUG, a procedure for pre-training an encoder-decoder transformer with domain knowledge.\nA main claimed contribution that domain-specific knowledge is incorporated both in the encoder and the decoder (as opposed to focusing on the encoder side).\nThe case study is e-commerce, where the authors incorporate information from a knowledge base about products.\n\nMore concretely, the authors propose to apply multiple pre-training objectives adapted to the knowledge source:\n1. Token masking - similar to BERT, where the masked tokens correspond to `\"knowledge tokens\" rather than randomly selected tokens.\n2. Encoder-decoder masking - Again, it is proposed to mask segments of text that cover `\"knowledge tokens\".\n3. Product Entity Aspect Boundary Detection: tag mentions of `product aspects' in the text. \n4. Product Entity Category Classification: Associating a given text with the product category\n5. Product Entity Aspect Summary Generation: adapting previous work, this objective generates a summary from the description of a `\"product entity aspect\".\n\nThe approach is evaluated on several language understanding and generation tasks that pertain to the product knowledge source, including product knowledge base completion, abstractive product summarization, and multi-turn dialogue. \n\nPROS:\n- Integrating knowledge in transformer-based model is a topic of interest\n- The experimental evaluation is solid. The authors compare with previous works, and present sensible ablation results.\n\nCONS:\n- Some details of the approach are under-specified/formulized. The results may therefore not be reproducible. (see comments)\n- The paper is not self-contained (see comments)\n- It is not clear if and how the approach extends to other types of knowledge, outside of the e-commerce and products domain.\n\nDetailed comments:\n\n- The major difference from BERT is that our KMLM prioritizes knowledge tokens - what are the `\"knowledge tokens\"? is it based on lexical match? does it only include `\"content\" words (with no prepositions, for example)? is ambiguity accounted for? is there a preliminary step of entity linking?\n\n- What is a product `\"aspect\" exactly? how are `\"attributes\" and `\"features\" different?\n\n- The baseline method JAVE is missing a citation\n\n- Table 1: why not report also precision and recall, in addition to F1? there seems to be enough space.\n\n- Table 2: note that the bold-faced results are not the best for all columns.\n\n- The evaluation tasks are very briefly described. Perhaps examples can be given? with some explanation of how the proposed approach helps?\n\n- Some terms are under-specified: e.g., `\"intents\" in Sec. 4.2.3. \n\n- 'For the retrieval-based approach, we concatenate the dialogue context and use [SEP] token to separate context and response. \nThe [CLS] representation is fed into the output layer for classification.' - what is the context? \n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a new method for pre-training of language models in the e-commerce domain. It introduces five objectives for pre-training by incorporating domain knowledge into the model.

Pros • The paper is generally easy to follow. • Design of the pre-training objectives is reasonable. • Experimental results are solid and convincing. • A useful method is proposed, and its effectiveness has been verified in the e-commence domain.

Cons • Novelty of the work might not be enough. • Presentations can be improved.
• It is not clear whether the proposed approach can be applied to other domains which may not have enough structured data.

The authors have made several things clearer in the rebuttal. They have also added new experimental results. However, the overall quality of the paper does not reach the level of ICLR from the viewpoint of novelty, significance, and clarity.


Decision: Reject