Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: Summary: This paper proposes pretraining language model for e-commerce domain. Specifically, the authors design five pretraining objectives to incorporate various domain knowledge into the the models with an encoder-decoder architecture. When further finetuned on language understanding and generation tasks in the e-commerce domain, the proposed models named K-PLUG outperforms the existing baseline models including those pretrained on general domains. The paper is generally easy to follow. Designs of the pretraining objectives are reasonable and empirically effective. Experiments are solid and convincing.

Pros:
The paper demonstrates a good practice on how to pretrain a language model for a specific domain by injecting a variety of structured and unstructured domain knowledge. Experiments verify that the knowledge-aware pretraining is indeed effective when you want to boost the performances of the tasks in specific domain. The experiments are well designed and quite comprehensive.

Concerns:
1. The paper is generally lack of novelty and inspiration, however. Most of the pretraining objectives have been tackled more or less in previous work. It is not surprising that when pretraining on the large amount of domain-specific data, the performance on the downstream tasks will improve. It will be more inspiring if the authors can give some high-level principles. Is it possible to design a more general framework that can work well when shifting to a new domain? What are the principles of designing pretrianing objectives with regard to domain knowledge and is there any way to evaluate them without need of expensive pretraining and finetuning?
2. The presentation of paper could be improved, especially for section 3. For example, from the notation it is unclear which training objectives are defined on encoder and which are on both encoder and decoder. Also in figure 1, there seems no clear distinction between the encoder and the decoder. 

Additional questions: 
1. How the five objectives are weighted in the pretraining? 
2. What are the “knowledge-tokens” and “non-knowledge” tokens in KMLM and KMS2S? Are the results sensitive to them?

review_title: Good practice on pretraining language models for specific domain

final_score: rating: 4: Ok but not good enough - rejection
review_content: The paper proposes K-PLUG, a procedure for pre-training an encoder-decoder transformer with domain knowledge.
A main claimed contribution that domain-specific knowledge is incorporated both in the encoder and the decoder (as opposed to focusing on the encoder side).
The case study is e-commerce, where the authors incorporate information from a knowledge base about products.

More concretely, the authors propose to apply multiple pre-training objectives adapted to the knowledge source:
1. Token masking - similar to BERT, where the masked tokens correspond to `"knowledge tokens" rather than randomly selected tokens.
2. Encoder-decoder masking - Again, it is proposed to mask segments of text that cover `"knowledge tokens".
3. Product Entity Aspect Boundary Detection: tag mentions of `product aspects' in the text. 
4. Product Entity Category Classification: Associating a given text with the product category
5. Product Entity Aspect Summary Generation: adapting previous work, this objective generates a summary from the description of a `"product entity aspect".

The approach is evaluated on several language understanding and generation tasks that pertain to the product knowledge source, including product knowledge base completion, abstractive product summarization, and multi-turn dialogue. 

PROS:
- Integrating knowledge in transformer-based model is a topic of interest
- The experimental evaluation is solid. The authors compare with previous works, and present sensible ablation results.

CONS:
- Some details of the approach are under-specified/formulized. The results may therefore not be reproducible. (see comments)
- The paper is not self-contained (see comments)
- It is not clear if and how the approach extends to other types of knowledge, outside of the e-commerce and products domain.

Detailed comments:

- The major difference from BERT is that our KMLM prioritizes knowledge tokens - what are the `"knowledge tokens"? is it based on lexical match? does it only include `"content" words (with no prepositions, for example)? is ambiguity accounted for? is there a preliminary step of entity linking?

- What is a product `"aspect" exactly? how are `"attributes" and `"features" different?

- The baseline method JAVE is missing a citation

- Table 1: why not report also precision and recall, in addition to F1? there seems to be enough space.

- Table 2: note that the bold-faced results are not the best for all columns.

- The evaluation tasks are very briefly described. Perhaps examples can be given? with some explanation of how the proposed approach helps?

- Some terms are under-specified: e.g., `"intents" in Sec. 4.2.3. 

- 'For the retrieval-based approach, we concatenate the dialogue context and use [SEP] token to separate context and response. 
The [CLS] representation is fed into the output layer for classification.' - what is the context? 

review_title: An interesting work should be described better

final_score: rating: 5: Marginally below acceptance threshold
review_content: 
Overview:

The authors introduce K-PLUG, a knowledge-injected language model with domain-specific knowledge for both NLU and NLG tasks. They evaluate their method in e-commerce scenarios. They use five pre-training objectives, including domain-specific knowledge-bases, entities aspects, entities categories, and entities selling propositions. 


Reasons to accept:
* The authors present K-PLUG that pre-trained on Chinese e-commerce data, which contains approximately 25 million textual product descriptions.

* The authors evaluate three different downstream tasks, e-commerce KB completion, abstractive product summarization, and multi-turn dialogue, and show good results compared to its baselines. 

Reasons to reject:
* The paper is not easy-to-follow with too many abbreviations.

* The main concern of this work is: the comparison to existing works is too weak. As mentioned in Section 2.2, there are many related works also working on injecting knowledge or retrieving knowledge to improve language model pretraining. For example, KEPLER, SKEP, SenseBERT, and K-ADAPTER. However, the authors do not compare their training objectives/strategies to theirs, it is very hard for us to make a conclusion that which is better. It is not an apple-to-apple comparison. The only baselines to compare within this paper is C-PLUG and E-PLUG.

* The dataset used to pre-train the model contains expensive human annotation, which is not scalable. For example, the category labels (used for Product Entity Category Classification) and the summary (used for Product Entity Aspect Summary Generation). In other words, this kind of makes the pre-training process like a "multi-task" learning approach with human annotations, which in my opinion is hard to collect more data. 

* The ablation study results that are shown in Table 5 basically suggest that many of the used objective functions are not important and could be ignored as the difference with and without is really marginal (I am sure some of them are not significantly better and could be the same if the authors tune a bit the hyper-parameters).

review_title: K-PLUG Review

final_score: rating: 6: Marginally above acceptance threshold
review_content: 
#### Summary

This paper introduces K-PLUG, an in-domain transformer-style pretrained language model. The domain on focus is e-commerce.

To enforce the PLM to be domain-aware, the authors identify several key types of knowledge for e-commerce: knowledge-bases, product aspects, product categories and product unique selling propositions (USP). They design specific pretraining objectives based on each type of knowledge.

Experiments on several downstream tasks exhibited the usefulness of the proposed model.

#### Strength

- The idea of domain-specific PLM is likely to be useful in practice but not widely explored. Specifically, in the e-commerce field people often have heavy needs for automated text processing and also a large amount of data, making this line of work more useful in practice.
- The pretraining objectives are properly designed, as they capture domain knowledge and have supervisions from the large-scale pretraining dataset. The experiment results also justified the design.

#### Weakness

- The proposed method (objectives) is hard to transfer to other domains. More importantly, it requires highly-structured and large-scale in-domain data for pretraining, which might not be available in other domains.
- Some downstream tasks are too similar to pretraining tasks. For instance, Abstractive Product Summarization looks very similar to Product Entity Aspect Summary Generation (PEASG) in pretraining, given that the image modality is ignored. I understand that if pretraining tasks are similar to downstream tasks, the performance on downstream tasks will be better. However, such similarity limits the ability of the experiments on downstream tasks to show the usefulness of the pretrained model. In other words, when the tasks are too similar, it becomes less clear whether the performance gain comes from the pretraining method or merely a larger training set. Also, have you tried directly using the pretraining dataset in downstream tasks (it looks possible for E-commerce KB Completion and Abstractive Product Summarization)?

#### Questions & Suggestions

- In Figure 1a, in the pretraining data aspect descriptions, the knowledge-base values and USPs are highlighted. How are these labeled? Are they directly available from the platform?
- For the pretraining dataset and downstream task datasets, each of them is said to be collected from "a Chinese e-commerce platform". Are they from the same or different platforms? It might be more clear to explicitly provide the platform names.
- In the formulation of KMLM and KMS2S (page 4) you mentioned "knowledge tokens", what are "knowledge tokens"? Are they knowledge-base tokens and/or USP tokens?
- For multi-turn dialog, since it is highly user-oriented, it would be better to have human evaluations.

#### Typos

- Page 5, PEASG, Line 6: "a aspect"

review_title: A well-designed and useful PLM for e-commerce; less clear if applicable to other fields

Meta-review: This paper proposes a new method for pre-training of language models in the e-commerce domain. It introduces five objectives for pre-training by incorporating domain knowledge into the model.

Pros • The paper is generally easy to follow. • Design of the pre-training objectives is reasonable. • Experimental results are solid and convincing. • A useful method is proposed, and its effectiveness has been verified in the e-commence domain.

Cons • Novelty of the work might not be enough. • Presentations can be improved.
• It is not clear whether the proposed approach can be applied to other domains which may not have enough structured data.

The authors have made several things clearer in the rebuttal. They have also added new experimental results. However, the overall quality of the paper does not reach the level of ICLR from the viewpoint of novelty, significance, and clarity.


Decision: Reject