K-PLUG: KNOWLEDGE-INJECTED PRE-TRAINED LANGUAGE MODEL FOR NATURAL LANGUAGE UNDERSTANDING AND GENERATION

1 INTRODUCTION . Pre-trained language models ( PLMs ) , such as ELMo ( Peters et al. , 2018 ) , GPT ( Radford et al. , 2018 ) , BERT ( Devlin et al. , 2019 ) , RoBERTa ( Liu et al. , 2019 ) , and XLNet ( Yang et al. , 2019 ) , have made remarkable breakthroughs in many natural language understanding ( NLU ) tasks , including text classification , reading comprehension , and natural language inference . These models are trained on large-scale text corpora with self-supervision based on either bi-directional or auto-regressive pre-training . Equally promising performances have been achieved in natural language generation ( NLG ) tasks , such as machine translation and text summarization , by MASS ( Song et al. , 2019 ) , UniLM ( Dong et al. , 2019 ) , BART ( Lewis et al. , 2020 ) , T5 ( Raffel et al. , 2019 ) , PEGASUS ( Zhang et al. , 2020 ) , and ProphetNet ( Yan et al. , 2020 ) . In contrast , these approaches adopt Transformerbased sequence-to-sequence models to jointly pre-train for both the encoder and the decoder . While these PLMs can learn rich semantic patterns from raw text data and thereby enhance downstream NLP applications , many of them do not explicitly model domain-specific knowledge . As a result , they may not be as sufficient for capturing human-curated or domain-specific knowledge that is necessary for tasks in a certain domain , such as tasks in e-commerce scenarios . In order to overcome this limitation , several recent studies have proposed to enrich PLMs with explicit knowledge , including knowledge base ( KB ) ( Zhang et al. , 2019 ; Peters et al. , 2019 ; Xiong et al. , 2020 ; Wang et al. , 2019 ; 2020 ) , lexical relation ( Lauscher et al. , 2019 ; Wang et al. , 2020 ) , word sense ( Levine et al. , 2020 ) , part-of-speech tag ( Ke et al. , 2019 ) , and sentiment polarity ( Ke et al. , 2019 ; Tian et al. , 2020 ) . However , these methods only integrate domain-specific knowledge into the encoder , and the decoding process in many NLG tasks benefits little from these knowledge . 1Our code is available at https : //github.com/ICLR21Anonymous/knowledge_pretrain . To mitigate this problem , we propose a Knowledge-injected Pre-trained Language model that is suitable for both Natural Language Understanding and Generation ( K-PLUG ) . Different from existing knowledge-injected PLMs , K-PLUG integrates knowledge into pre-training for both the encoder and the decoder , and thus K-PLUG can be adopted to both downstream knowledge-driven NLU and NLG tasks . We verify the performance of the proposed method in various e-commerce scenarios . In the proposed K-PLUG , we formulate the learning of four types of domain-specific knowledge : e-commerce domain-specific knowledge-bases , aspects of product entities , categories of product entities , and unique selling propositions ( USPs ) ( Garrett , 1961 ) of product entities . Specifically , e-commerce KB stores standardized product attribute information , product aspects are features that play a crucial role in understanding product information , product categories are the backbones for constructing taxonomies for organization , and USPs are the essence of what differentiates a product from its competitors . K-PLUG learns these types of knowledge into a unified PLM , enhancing performances for various language understanding and generation tasks . To effectively learn these four types of valuable domain-specific knowledge in K-PLUG , we proposed five new pre-training objectives : knowledge-aware masked language model ( KMLM ) , knowledge-aware masked sequence-to-sequence ( KMS2S ) , product entity aspect boundary detection ( PEABD ) , product entity category classification ( PECC ) , and product entity aspect summary generation ( PEASG ) . Among these objectives , KMLM and KMS2S learn to predict the masked single and multiple tokens , respectively , that are associated with domain-specific knowledge rather than general information ; PEABD determines the boundaries between descriptions of different product aspects given full product information ; PECC identifies the product category that each product belongs to ; and PEASG generates a summary for each individual product aspect based on the entire product description . After pre-training K-PLUG , we fine-tune it on three domain-specific NLP tasks , namely , ecommerce knowledge base completion , abstractive product summarization , and multi-turn dialogue . The results show that K-PLUG significantly outperforms comparative models on all these tasks . Our main contributions can be summarized as follows : • We present K-PLUG that learns domain-specific knowledge for both the encoder and the decoder in a pre-training language model framework , which benefits both NLG and NLU tasks . • We formulate the learning of four types of domain-specific knowledge in e-commerce scenarios : e-commerce domain-specific knowledge-bases , aspects of product entities , categories of product entities , and unique selling propositions of product entities , which provide critical information for many applications in the domain of e-commerce . Specifically , five self-supervised objectives are proposed to learn these four types of knowledge into a unified PLM . • Our proposed model exhibits clear effectiveness in many downstream tasks in the ecommerce scenario , including e-commerce KB completion , abstractive product summarization , and multi-turn dialogue . 2 RELATED WORK . 2.1 PLMS IN GENERAL . Unsupervised pre-training language model has been successfully applied to many NLP tasks . ELMo ( Peters et al. , 2018 ) learns the contextual representations based on a bidirectional LM . GPT ( Radford et al. , 2018 ) predicts tokens based on the context on the left-hand side . BERT ( Devlin et al. , 2019 ) adopts a bi-directional LM to predict the masked tokens . XLNet ( Yang et al. , 2019 ) predicts masked tokens in a permuted order through an autoregressive method . MASS ( Song et al. , 2019 ) pre-trains the sequence-to-sequence LM to recover a span of masked tokens . UniLM ( Dong et al. , 2019 ) combines bidirectional , unidirectional , and sequence-to-sequence LMs . T5 ( Raffel et al. , 2019 ) and BART ( Lewis et al. , 2020 ) present denoising sequence-to-sequence pre-training . PEGASUS ( Zhang et al. , 2020 ) pre-trains with gap-sentence generation objective . While humancurated or domain-specific knowledge is essential for downstream knowledge-driven tasks , these methods do not explicitly consider external knowledge like our proposed K-PLUG . ties , categories of product entities , and : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : unique selling propositions of product entities . Pre-training objectives include knowledge-aware masked language model ( KMLM ) , knowledge-aware masked sequence-to-sequence ( KMS2S ) , product entity aspect boundary detection ( PEABD ) , product entity category classification ( PECC ) , and product entity aspect summary generation ( PEASG ) . 2.2 INJECTING KNOWLEDGE INTO PLMS . Recent work investigates how to incorporate knowledge into PLMs for NLU . ERNIE ( Sun et al. , 2019 ) enhances language representation with the entity/phrase-level masking . ERNIE ( Zhang et al. , 2019 ) identifies and links entity mentions in texts to their corresponding entities in KB . Similar to ERNIE ( Zhang et al. , 2019 ) , KnowBERT ( Peters et al. , 2019 ) injects KBs into PLM . Xiong et al . ( 2020 ) leverages an entity replacement pre-training objective to learn better representations for entities . KEPLER ( Wang et al. , 2019 ) adopts the knowledge embedding objective in the pretraining . Besides , SKEP ( Tian et al. , 2020 ) , SenseBERT ( Levine et al. , 2020 ) , SentiLR ( Ke et al. , 2019 ) , and K-ADAPTER ( Wang et al. , 2020 ) propose to integrate sentiment knowledge , word sense , sentiment polarity , and lexical relation into PLM , respectively . However , most of these studies are focused on integrating knowledge for language understanding task , work of utilizing domainspecific knowledge for pre-training for language generation tasks are limited . Inspired by these work , we construct K-PLUG that learns domain-specific knowledge into a PLM for both NLU and NLG tasks . 3 KNOWLEDGE-INJECTED PRE-TRAINING . In this section , we explain the data used to pre-train K-PLUG , its model architecture , and our pretraining objectives . 3.1 DATA PREPARATION . We collect the pre-training data from a mainstream Chinese e-commerce platform2 , which contains approximately 25 million textual product descriptions and covers 40 product categories . With an average length of 405 tokens , these product descriptions constitute a corpus with a size of 10B Chinese characters . Each product description consists of information on 10.7 product aspects on average , and each product aspect is accompanied with a summary highlighting its prominent features , as shown in Figure 1 ( a ) . Additionally , the e-commerce KB and USPs ( further explained below ) used in our pre-training data are as specified by the e-commerce platform and its online stores . 2https : //www.jd.com/ 3.2 MODEL ARCHITECTURE . K-PLUG adopts the standard sequence-to-sequence Transformer architecture ( Vaswani et al. , 2017 ) , consisting of a 6-layer encoder and a 6-layer decoder as Song et al . ( 2019 ) . We set the size of hidden vectors as 768 , and the number of self-attention heads as 12 . More details about the experimental settings are in the appendix . 3.3 KNOWLEDGE FORMULATION AND PRE-TRAINING OBJECTIVES . We formulate the learning of four types of knowledge in a unified PLM : e-commerce KB , aspects of product entities , categories of product entities , and USPs of product entities . Specifically , ecommerce KB stores standardized product attribute information , e.g. , ( Material : Cotton ) and ( Collar Type : Pointed Collar ) . It provides details about the products ( Logan IV et al. , 2017 ) . Aspects of product entities are features of a product , such as the sound quality of a stereo speaker , etc . ( Li et al. , 2020 ) . Categories of product entities such as Clothing and Food are widely used by e-commerce platforms to organize their products so to present structured offerings to their customers ( Luo et al. , 2020 ; Dong et al. , 2020 ) USPs of product entities are the essence of what differentiates a product from its competitors ( Garrett , 1961 ) . For example , a stereo speaker ’ s USP exhibiting its supreme sound quality could be “ crystal clear stereo sound ” . An effective USP immediately motivates the purchasing behavior of potential buyers . We propose and evaluate five novel self-supervised pre-training objectives to learn the abovementioned four types of knowledge in the K-PLUG model ( see Figure 1 ) . Knowledge-aware Masked Language Model ( KMLM ) . Inspired by BERT ( Devlin et al. , 2019 ) , we adopt the masked language model ( MLM ) to train the Transformer encoder as one of our pre-training objectives , which learns to predict the masked tokens in the source sequence ( e.g. , “ The company is [ MASK ] at the foot of a hill. ” ) . Similar to BERT , we mask 15 % of all tokens in a text sequence ; 80 % of the masked tokens are replaced with the [ MASK ] token , 10 % with a random token , and 10 % left unchanged . Particularly , given an original text sequence x = ( x1 , ... , xm , ... , xM ) with M tokens , a masked sequence is produced by masking xm through one of the three ways explained above , e.g. , replacing xm with [ MASK ] to create x̃ = ( x1 , ... , [ MASK ] , ... , xM ) . MLM aims to model the conditional likelihood P ( xm|x̃ ) , and the loss function is : LMLM = logP ( xm|x̃ ) ( 1 ) The major difference from BERT is that our KMLM prioritizes knowledge tokens , which contain knowledge regarding product attributes and USPs , when selecting positions to mask and , in the case that the knowledge tokens make up less than 15 % of all tokens , randomly picks non-knowledge tokens to complete the masking .

Summary: This paper proposes pretraining language model for e-commerce domain. Specifically, the authors design five pretraining objectives to incorporate various domain knowledge into the the models with an encoder-decoder architecture. When further finetuned on language understanding and generation tasks in the e-commerce domain, the proposed models named K-PLUG outperforms the existing baseline models including those pretrained on general domains. The paper is generally easy to follow. Designs of the pretraining objectives are reasonable and empirically effective. Experiments are solid and convincing.