# Analyzing the Effect of Masking Length Distribution of MLM: An Evaluation Framework and Case Study on Chinese MRC Datasets

Authors: Changchang Zeng; Shaobo Li
Journal: Wireless Communications and Mobile Computing
 (2021)

---

## Abstract

Machine reading comprehension (MRC) is a challenging natural language processing (NLP) task. It has a wide application potential in the fields of question answering robots, human-computer interactions in mobile virtual reality systems, etc. Recently, the emergence of pretrained models (PTMs) has brought this research field into a new era, in which the training objective plays a key role. The masked language model (MLM) is a self-supervised training objective widely used in various PTMs. With the development of training objectives, many variants of MLM have been proposed, such as whole word masking, entity masking, phrase masking, and span masking. In different MLMs, the length of the masked tokens is different. Similarly, in different machine reading comprehension tasks, the length of the answer is also different, and the answer is often a word, phrase, or sentence. Thus, in MRC tasks with different answer lengths, whether the length of MLM is related to performance is a question worth studying. If this hypothesis is true, it can guide us on how to pretrain the MLM with a relatively suitable mask length distribution for MRC tasks. In this paper, we try to uncover how much of MLM’s success in the machine reading comprehension tasks comes from the correlation between masking length distribution and answer length in the MRC dataset. In order to address this issue, herein, (1) we propose four MRC tasks with different answer length distributions, namely, the short span extraction task, long span extraction task, short multiple-choice cloze task, and long multiple-choice cloze task; (2) four Chinese MRC datasets are created for these tasks; (3) we also have pretrained four masked language models according to the answer length distributions of these datasets; and (4) ablation experiments are conducted on the datasets to verify our hypothesis. The experimental results demonstrate that our hypothesis is true. On four different machine reading comprehension datasets, the performance of the model with correlation length distribution surpasses the model without correlation.


---

## Body

## 1. Introduction

In the field of natural language processing (NLP), machine reading comprehension (MRC) is a challenging task and has received extensive attention. According to the definition of Burges, machine reading comprehension refers to the following: “A machine comprehends a passage of text if, for any question regarding that text that can be answered correctly by a majority of native speakers, that machine can provide a string which those speakers would agree both answers that question, and does not contain information irrelevant to that question [1].”Generally, MRC tasks can be roughly divided into four categories based on the answer form: cloze test, multiple choice, span extraction, and free answering [2, 3]. Most of the early reading comprehension systems were based on retrieval technology; that is, we search in the article according to the questions and find the relevant sentences as the answers. However, information retrieval mainly depends on keyword matching, and in many cases, the answers found by relying solely on text matching are not related to the questions.With the development of machine learning (especially deep learning) and the release of large-scale datasets, the efficiency and quality of the MRC model have been greatly improved. In some benchmark datasets, the accuracy of the MRC model has exceeded the human performance [4]. In recent years, pretrained language models (PTMs) have brought revolutionary changes to the field of MRC. Among them, the most representative pretrained model is the BERT proposed by Google in 2018 [5]. BERT uses unsupervised learning to pretrain on a large-scale corpus and creatively uses MLM and NSP subtasks to enhance the language ability of the model [5]. After the author released the code and pretrained models, BERT was immediately used by researchers in various NLP tasks, and the previous SOTA results were refreshed frequently and significantly.Recently, many efforts have been devoted to improve pretrained models, and various pretrained models have been proposed, such as BERT-wwm [6], ERNIE 1.0 [7], ERNIE 2.0 [8], SpanBERT [9], and MacBERT [10]. We can see that all of them have improved the masked language model (MLM) of the BERT model in different ways. However, the BERT model itself (the paradigm of the pretraining process, transformer-based model, and fine-tuning process) has not been significantly modified. This shows the importance of MLM. MLM is a self-supervised training objective of predicting missing tokens in a sequence from placeholders, which is widely used in various PTMs [11]. With the development of training objectives, many variants of MLM have been proposed, such as whole word masking [6], entity masking [7, 8], phrase masking [7, 8], and span masking [9].In different MRC tasks, the length of the answer text is often different, and the answer is either a word, a phrase, or a sentence. Similarly, in different variants of MLM, the text length of the mask is also different. For example, the whole word masking improves the MLM objective of BERT by using the whole word instead of the word piece [6]; the span masking performs the replacement at the span level and not for each token individually [9]; the entity masking masks entities that are usually composed of multiple words, while the phrase masking masks an entire phrase composed of multiple words as a conceptual unit [7, 8].How to choose a masking scheme for MRC tasks with different answer lengths has become a question worth studying. At the same time, it also makes us wonder whether the length of MLM is related to their performance in MRC tasks with different answer lengths. If this hypothesis is true, maybe it can guide us on how to pretrain an MLM with a relatively suitable mask length distribution for various MRC tasks.However, for different variants of MLM, there are many inconsistencies in their corpora, training methods, evaluation tasks, and benchmark datasets. Therefore, it is difficult to perform ablation experiments on the existing MRC datasets and these publicly released pretrained models to quantitatively measure the performance improvements brought about by different masking schemes.To address the above issues, we design a set of controlled experiments to verify our hypothesis. In summary, our main contributions are as follows:(1)
Four MRC tasks with different answer length distributions are proposed, including the short span extraction task, long span extraction task, short cloze task, and long cloze task(2)
We create MRC datasets for these four tasks and statistically analyze the answer word length distribution on these four datasets(3)
Using uniform hyperparameters, we trained MLMs with different masking length distributions(4)
We conducted ablation experiments on the above dataset to verify our hypothesis. The experiment result shows that the consistency of the masking length distribution and the answer length distribution does affect the performance of the model. On four different machine reading comprehension datasets, the performance of the model with correlation length distribution surpasses the model without correlation


## 2. Related Works

### 2.1. Existing Masked Language Models (MLMs)

Recently, many efforts have been devoted to improve masked language models (MLMs). In this section, we briefly introduce several existing MLMs, including word piece masking, whole word masking, entity masking, phrase masking, span masking, andN-gram masking.


#### 2.1.1. Word Piece Masking

Word piece masking is the MLM used in the original version of BERT [5], where the “WordPiece tokenizer” is used in the data preprocessing to split the input sequence into subwords, which is very effective in dealing with out-of-vocabulary (OOV) words. In Chinese text tokenization, when a sentence is tokenized with a “WordPiece tokenizer,” it will be split into Chinese characters. Then, tokens are selected randomly for masking, and 15% of the tokens will be randomly selected. Among the selected tokens, for each word, it has an 80% probability of being replaced with MASK, 10% will be replaced with a random token, and 10% will remain unchanged. It should be noted that each token is masked independently according to the above probability, rather than all selected tokens being masked at the same time.


#### 2.1.2. Whole Word Masking

Whole word masking (WWM) is an upgraded version of BERT [5] released by Google, which mitigates the drawbacks of masking partial WordPiece tokens in the original BERT [6]. In the whole word masking, if a subword of a complete word is masked, the other parts of the same word will also be masked; that is, the whole word will be masked at the same time. In the Chinese version of BERT released by Google, Chinese is segmented at the granularity of characters, and the Chinese word segmentation (CWS) [12] is not considered. Therefore, Cui et al. applied the whole word masking to Chinese [6] and masked the whole word instead of masking Chinese characters.


#### 2.1.3.N-Gram Masking

N-gram In reference [10], Cui et al. believed that the idea of N-gram masking was first proposed by Devlin et al. [5]. Devlin et al.’s paper [5] did not include the phrase “N-gram masking,” but according to their model name on the SQuAD leaderboard, the academic community usually admits that they were the first to propose this concept [10]. In N-gram masking, a sequence of N words is treated as a whole unity. During pretraining of MLM, all words in the same unit are masked, instead of masking only one word or character. N-gram masking is used in many advanced pretraining models. For example, MacBERT [10] uses an N-gram masking scheme for selecting candidate tokens for masking, with a percentage of 40%, 30%, 20%, and 10% for the word-level unigram to 4-gram. To a certain extent, the span masking, entity-level masking, and phrase-level masking can be regarded as special cases of the N-gram masking scheme [5, 9].


#### 2.1.4. ExplicitlyN-Gram Masking

ExplicitlyN-gram masking is an explicit N-gram masking scheme, in which the N-gram is replaced by a single MASK symbol [13]. When predicting masked tokens, explicit N-gram identities are directly used instead of token sequences. In addition, explicitly N-gram masking uses a generator model to sample reasonable N-gram identities as an optional N-gram mask and predicts it in both a coarse-grained manner and a fine-grained manner to achieve comprehensive relation modeling. Explicitly N-gram masking is proposed by the Baidu team in 2021 [13].


#### 2.1.5. Entity-Level Masking

Entities usually contain important information in the sentences, such as a person, location, organization, and product. Unlike selecting random tokens for masking, entity-level masking masks the whole named entities, which are usually composed of multiple words [7, 8]. Before masking, the text needs to be segmented using named entity recognition tools. In entity-level masking, the MLM implicitly learned the information about longer semantic dependency, such as the relationship between entities.


#### 2.1.6. Phrase-Level Masking

A phrase is a small group of words or characters as a conceptual unit. Phrase-level masking masks the whole phrase which is composed of several words [7, 8], and it is similar to the N-gram masking scheme [5, 9, 10]. For English, vocabulary analysis and chunking tools are used to get the boundaries of phrases in sentences, and we use some language-related segmentation tools to get word/phrase information in other languages (such as Chinese) [7, 8]. In this way, the prior knowledge of phrases is considered to be learned implicitly during the training procedure, such as syntactic and semantic information.


#### 2.1.7. Span Masking

Span masking was proposed in the SpanBERT [9], in which contiguous random spans are masked, rather than individual tokens. The process of span masking is, first, iteratively selecting contiguous random spans until the 15% masking budget is spent. In each iteration, the length of the span is selected according to the geometric distribution. Then, randomly select the starting point of the span. Third, replace all the tokens in the same selected span with MASK, random or original tags according to the 80%-10%-10% rule in BERT [5], where the span constitutes the unit. Therefore, it forces the model to use only the context in which the span occurs to predict the entire span.


#### 2.1.8. Multilevel Masking

Multilevel masking uses multiple schemes at the same time. For example, the knowledge masking proposed in the ERNIE 1.0 [7] masking can be regarded as a kind of multilevel masking scheme, which uses both phrase-level masking and entity-level masking. The knowledge masking treats a phrase or entity as a unit, which is usually composed of several words. All words in the same unit are masked, instead of masking only one word or character. The knowledge masking does not directly add knowledge embedding but is considered to be learning information about knowledge, such as entity attributes and event types, to guide word embedding learning [7].


#### 2.1.9. Dynamic Masking

Static masking is used in the MLM of the original BERT, and the masking is performed only once during the data preprocessing before MLM training, which means that the same words are masked in the input sequence provided to the model on each epoch. In order to avoid masking the same words multiple times and make full use of the input sequence, dynamic masking is proposed. In the dynamic masking process, the “dupe_factor” is defined, and the input sequence will be the duplicated “dupe_factor”; then, the same sequence will have different masks [14]. Before providing the input sequence to the model each time, the masking operation will be performed repeatedly. Therefore, the model will see different masking versions of the same sequence. Dynamic masking is adopted by many pretrained models, such as RoBERTa [14].


### 2.2. Interpretability of Masked Language Models

With various advanced MLMs, many pretrained language models have achieved state-of-the-art performance when adapted to MRC tasks. The black box nature of MLM and related pretrained models has inspired many works trying to understand them.Many efforts have been devoted to uncover whether the MLM calculates various types of structured information by probing analysis or evaluating the performance of simple classifiers on representations [15–20]. Popular methods also include analyzing self-attention weights and evaluating the performance of classifiers with different representations as inputs [19]. A possible explanation for the success of masked language model (MLM) training is that these models have learned to represent semantic information or syntactic information [21].


#### 2.2.1. Semantic Information

With the MLM probing study, Ettinger applied a set of diagnostic methods derived from human language experiments to the BERT model and found that BERT has a certain understanding of semantic roles [19, 22].Tenney et al. used a set of detection tasks derived from traditional NLP pipelines to quantify the encoding position of specific types of semantic information, and the experimental results show that BERT encodes information about entity types, relationships, semantic roles, and prototype roles [18, 19].


#### 2.2.2. Syntactic Information

Through the probes of MLM, Goldberg assessed the extent to which the BERT model captures English syntactic phenomena and found that the BERT models perform remarkably well on the syntactic test cases. The experimental results show that BERT considers subject-predicate agreement when completing the cloze task, even for meaningless sentences and sentences with participle clauses between the subject and the verb [19, 23].Wu et al. proposed a perturbation masking technique to evaluate the impact of one word on the prediction of another word in MLM. They concluded that BERT “naturally” learns some syntactic information, although it is not very similar to linguistically annotated resources [19, 24].


#### 2.2.3. Distributional Information

Most recently, Sinha et al. [21] surprisingly found that most of MLM’s high performance can in fact be explained by the “distributional prior” rather than its ability to replicate “the types of syntactic and semantic abstractions traditionally believed necessary for language processing [18].” In other words, they found that the success of MLM in downstream tasks is almost entirely because they can model high-order word co-occurrence statistics. To prove this, they pretrained MLMs on sentences with a randomly shuffled word order and showed that after fine-tuning many downstream tasks, these models can still achieve high accuracy, including tasks designed specifically to be challenging for models that ignore the word order. According to some parametric syntactic probes, these models perform surprisingly well, which indicates possible deficiencies when testing the representation for syntactic information [21].


## 2.1. Existing Masked Language Models (MLMs)

Recently, many efforts have been devoted to improve masked language models (MLMs). In this section, we briefly introduce several existing MLMs, including word piece masking, whole word masking, entity masking, phrase masking, span masking, andN-gram masking.


### 2.1.1. Word Piece Masking

Word piece masking is the MLM used in the original version of BERT [5], where the “WordPiece tokenizer” is used in the data preprocessing to split the input sequence into subwords, which is very effective in dealing with out-of-vocabulary (OOV) words. In Chinese text tokenization, when a sentence is tokenized with a “WordPiece tokenizer,” it will be split into Chinese characters. Then, tokens are selected randomly for masking, and 15% of the tokens will be randomly selected. Among the selected tokens, for each word, it has an 80% probability of being replaced with MASK, 10% will be replaced with a random token, and 10% will remain unchanged. It should be noted that each token is masked independently according to the above probability, rather than all selected tokens being masked at the same time.


### 2.1.2. Whole Word Masking

Whole word masking (WWM) is an upgraded version of BERT [5] released by Google, which mitigates the drawbacks of masking partial WordPiece tokens in the original BERT [6]. In the whole word masking, if a subword of a complete word is masked, the other parts of the same word will also be masked; that is, the whole word will be masked at the same time. In the Chinese version of BERT released by Google, Chinese is segmented at the granularity of characters, and the Chinese word segmentation (CWS) [12] is not considered. Therefore, Cui et al. applied the whole word masking to Chinese [6] and masked the whole word instead of masking Chinese characters.


### 2.1.3.N-Gram Masking

N-gram In reference [10], Cui et al. believed that the idea of N-gram masking was first proposed by Devlin et al. [5]. Devlin et al.’s paper [5] did not include the phrase “N-gram masking,” but according to their model name on the SQuAD leaderboard, the academic community usually admits that they were the first to propose this concept [10]. In N-gram masking, a sequence of N words is treated as a whole unity. During pretraining of MLM, all words in the same unit are masked, instead of masking only one word or character. N-gram masking is used in many advanced pretraining models. For example, MacBERT [10] uses an N-gram masking scheme for selecting candidate tokens for masking, with a percentage of 40%, 30%, 20%, and 10% for the word-level unigram to 4-gram. To a certain extent, the span masking, entity-level masking, and phrase-level masking can be regarded as special cases of the N-gram masking scheme [5, 9].


### 2.1.4. ExplicitlyN-Gram Masking

ExplicitlyN-gram masking is an explicit N-gram masking scheme, in which the N-gram is replaced by a single MASK symbol [13]. When predicting masked tokens, explicit N-gram identities are directly used instead of token sequences. In addition, explicitly N-gram masking uses a generator model to sample reasonable N-gram identities as an optional N-gram mask and predicts it in both a coarse-grained manner and a fine-grained manner to achieve comprehensive relation modeling. Explicitly N-gram masking is proposed by the Baidu team in 2021 [13].


### 2.1.5. Entity-Level Masking

Entities usually contain important information in the sentences, such as a person, location, organization, and product. Unlike selecting random tokens for masking, entity-level masking masks the whole named entities, which are usually composed of multiple words [7, 8]. Before masking, the text needs to be segmented using named entity recognition tools. In entity-level masking, the MLM implicitly learned the information about longer semantic dependency, such as the relationship between entities.


### 2.1.6. Phrase-Level Masking

A phrase is a small group of words or characters as a conceptual unit. Phrase-level masking masks the whole phrase which is composed of several words [7, 8], and it is similar to the N-gram masking scheme [5, 9, 10]. For English, vocabulary analysis and chunking tools are used to get the boundaries of phrases in sentences, and we use some language-related segmentation tools to get word/phrase information in other languages (such as Chinese) [7, 8]. In this way, the prior knowledge of phrases is considered to be learned implicitly during the training procedure, such as syntactic and semantic information.


### 2.1.7. Span Masking

Span masking was proposed in the SpanBERT [9], in which contiguous random spans are masked, rather than individual tokens. The process of span masking is, first, iteratively selecting contiguous random spans until the 15% masking budget is spent. In each iteration, the length of the span is selected according to the geometric distribution. Then, randomly select the starting point of the span. Third, replace all the tokens in the same selected span with MASK, random or original tags according to the 80%-10%-10% rule in BERT [5], where the span constitutes the unit. Therefore, it forces the model to use only the context in which the span occurs to predict the entire span.


### 2.1.8. Multilevel Masking

Multilevel masking uses multiple schemes at the same time. For example, the knowledge masking proposed in the ERNIE 1.0 [7] masking can be regarded as a kind of multilevel masking scheme, which uses both phrase-level masking and entity-level masking. The knowledge masking treats a phrase or entity as a unit, which is usually composed of several words. All words in the same unit are masked, instead of masking only one word or character. The knowledge masking does not directly add knowledge embedding but is considered to be learning information about knowledge, such as entity attributes and event types, to guide word embedding learning [7].


### 2.1.9. Dynamic Masking

Static masking is used in the MLM of the original BERT, and the masking is performed only once during the data preprocessing before MLM training, which means that the same words are masked in the input sequence provided to the model on each epoch. In order to avoid masking the same words multiple times and make full use of the input sequence, dynamic masking is proposed. In the dynamic masking process, the “dupe_factor” is defined, and the input sequence will be the duplicated “dupe_factor”; then, the same sequence will have different masks [14]. Before providing the input sequence to the model each time, the masking operation will be performed repeatedly. Therefore, the model will see different masking versions of the same sequence. Dynamic masking is adopted by many pretrained models, such as RoBERTa [14].


## 2.1.1. Word Piece Masking

Word piece masking is the MLM used in the original version of BERT [5], where the “WordPiece tokenizer” is used in the data preprocessing to split the input sequence into subwords, which is very effective in dealing with out-of-vocabulary (OOV) words. In Chinese text tokenization, when a sentence is tokenized with a “WordPiece tokenizer,” it will be split into Chinese characters. Then, tokens are selected randomly for masking, and 15% of the tokens will be randomly selected. Among the selected tokens, for each word, it has an 80% probability of being replaced with MASK, 10% will be replaced with a random token, and 10% will remain unchanged. It should be noted that each token is masked independently according to the above probability, rather than all selected tokens being masked at the same time.


## 2.1.2. Whole Word Masking

Whole word masking (WWM) is an upgraded version of BERT [5] released by Google, which mitigates the drawbacks of masking partial WordPiece tokens in the original BERT [6]. In the whole word masking, if a subword of a complete word is masked, the other parts of the same word will also be masked; that is, the whole word will be masked at the same time. In the Chinese version of BERT released by Google, Chinese is segmented at the granularity of characters, and the Chinese word segmentation (CWS) [12] is not considered. Therefore, Cui et al. applied the whole word masking to Chinese [6] and masked the whole word instead of masking Chinese characters.


## 2.1.3.N-Gram Masking

N-gram In reference [10], Cui et al. believed that the idea of N-gram masking was first proposed by Devlin et al. [5]. Devlin et al.’s paper [5] did not include the phrase “N-gram masking,” but according to their model name on the SQuAD leaderboard, the academic community usually admits that they were the first to propose this concept [10]. In N-gram masking, a sequence of N words is treated as a whole unity. During pretraining of MLM, all words in the same unit are masked, instead of masking only one word or character. N-gram masking is used in many advanced pretraining models. For example, MacBERT [10] uses an N-gram masking scheme for selecting candidate tokens for masking, with a percentage of 40%, 30%, 20%, and 10% for the word-level unigram to 4-gram. To a certain extent, the span masking, entity-level masking, and phrase-level masking can be regarded as special cases of the N-gram masking scheme [5, 9].


## 2.1.4. ExplicitlyN-Gram Masking

ExplicitlyN-gram masking is an explicit N-gram masking scheme, in which the N-gram is replaced by a single MASK symbol [13]. When predicting masked tokens, explicit N-gram identities are directly used instead of token sequences. In addition, explicitly N-gram masking uses a generator model to sample reasonable N-gram identities as an optional N-gram mask and predicts it in both a coarse-grained manner and a fine-grained manner to achieve comprehensive relation modeling. Explicitly N-gram masking is proposed by the Baidu team in 2021 [13].


## 2.1.5. Entity-Level Masking

Entities usually contain important information in the sentences, such as a person, location, organization, and product. Unlike selecting random tokens for masking, entity-level masking masks the whole named entities, which are usually composed of multiple words [7, 8]. Before masking, the text needs to be segmented using named entity recognition tools. In entity-level masking, the MLM implicitly learned the information about longer semantic dependency, such as the relationship between entities.


## 2.1.6. Phrase-Level Masking

A phrase is a small group of words or characters as a conceptual unit. Phrase-level masking masks the whole phrase which is composed of several words [7, 8], and it is similar to the N-gram masking scheme [5, 9, 10]. For English, vocabulary analysis and chunking tools are used to get the boundaries of phrases in sentences, and we use some language-related segmentation tools to get word/phrase information in other languages (such as Chinese) [7, 8]. In this way, the prior knowledge of phrases is considered to be learned implicitly during the training procedure, such as syntactic and semantic information.


## 2.1.7. Span Masking

Span masking was proposed in the SpanBERT [9], in which contiguous random spans are masked, rather than individual tokens. The process of span masking is, first, iteratively selecting contiguous random spans until the 15% masking budget is spent. In each iteration, the length of the span is selected according to the geometric distribution. Then, randomly select the starting point of the span. Third, replace all the tokens in the same selected span with MASK, random or original tags according to the 80%-10%-10% rule in BERT [5], where the span constitutes the unit. Therefore, it forces the model to use only the context in which the span occurs to predict the entire span.


## 2.1.8. Multilevel Masking

Multilevel masking uses multiple schemes at the same time. For example, the knowledge masking proposed in the ERNIE 1.0 [7] masking can be regarded as a kind of multilevel masking scheme, which uses both phrase-level masking and entity-level masking. The knowledge masking treats a phrase or entity as a unit, which is usually composed of several words. All words in the same unit are masked, instead of masking only one word or character. The knowledge masking does not directly add knowledge embedding but is considered to be learning information about knowledge, such as entity attributes and event types, to guide word embedding learning [7].


## 2.1.9. Dynamic Masking

Static masking is used in the MLM of the original BERT, and the masking is performed only once during the data preprocessing before MLM training, which means that the same words are masked in the input sequence provided to the model on each epoch. In order to avoid masking the same words multiple times and make full use of the input sequence, dynamic masking is proposed. In the dynamic masking process, the “dupe_factor” is defined, and the input sequence will be the duplicated “dupe_factor”; then, the same sequence will have different masks [14]. Before providing the input sequence to the model each time, the masking operation will be performed repeatedly. Therefore, the model will see different masking versions of the same sequence. Dynamic masking is adopted by many pretrained models, such as RoBERTa [14].


## 2.2. Interpretability of Masked Language Models

With various advanced MLMs, many pretrained language models have achieved state-of-the-art performance when adapted to MRC tasks. The black box nature of MLM and related pretrained models has inspired many works trying to understand them.Many efforts have been devoted to uncover whether the MLM calculates various types of structured information by probing analysis or evaluating the performance of simple classifiers on representations [15–20]. Popular methods also include analyzing self-attention weights and evaluating the performance of classifiers with different representations as inputs [19]. A possible explanation for the success of masked language model (MLM) training is that these models have learned to represent semantic information or syntactic information [21].


### 2.2.1. Semantic Information

With the MLM probing study, Ettinger applied a set of diagnostic methods derived from human language experiments to the BERT model and found that BERT has a certain understanding of semantic roles [19, 22].Tenney et al. used a set of detection tasks derived from traditional NLP pipelines to quantify the encoding position of specific types of semantic information, and the experimental results show that BERT encodes information about entity types, relationships, semantic roles, and prototype roles [18, 19].


### 2.2.2. Syntactic Information

Through the probes of MLM, Goldberg assessed the extent to which the BERT model captures English syntactic phenomena and found that the BERT models perform remarkably well on the syntactic test cases. The experimental results show that BERT considers subject-predicate agreement when completing the cloze task, even for meaningless sentences and sentences with participle clauses between the subject and the verb [19, 23].Wu et al. proposed a perturbation masking technique to evaluate the impact of one word on the prediction of another word in MLM. They concluded that BERT “naturally” learns some syntactic information, although it is not very similar to linguistically annotated resources [19, 24].


### 2.2.3. Distributional Information

Most recently, Sinha et al. [21] surprisingly found that most of MLM’s high performance can in fact be explained by the “distributional prior” rather than its ability to replicate “the types of syntactic and semantic abstractions traditionally believed necessary for language processing [18].” In other words, they found that the success of MLM in downstream tasks is almost entirely because they can model high-order word co-occurrence statistics. To prove this, they pretrained MLMs on sentences with a randomly shuffled word order and showed that after fine-tuning many downstream tasks, these models can still achieve high accuracy, including tasks designed specifically to be challenging for models that ignore the word order. According to some parametric syntactic probes, these models perform surprisingly well, which indicates possible deficiencies when testing the representation for syntactic information [21].


## 2.2.1. Semantic Information

With the MLM probing study, Ettinger applied a set of diagnostic methods derived from human language experiments to the BERT model and found that BERT has a certain understanding of semantic roles [19, 22].Tenney et al. used a set of detection tasks derived from traditional NLP pipelines to quantify the encoding position of specific types of semantic information, and the experimental results show that BERT encodes information about entity types, relationships, semantic roles, and prototype roles [18, 19].


## 2.2.2. Syntactic Information

Through the probes of MLM, Goldberg assessed the extent to which the BERT model captures English syntactic phenomena and found that the BERT models perform remarkably well on the syntactic test cases. The experimental results show that BERT considers subject-predicate agreement when completing the cloze task, even for meaningless sentences and sentences with participle clauses between the subject and the verb [19, 23].Wu et al. proposed a perturbation masking technique to evaluate the impact of one word on the prediction of another word in MLM. They concluded that BERT “naturally” learns some syntactic information, although it is not very similar to linguistically annotated resources [19, 24].


## 2.2.3. Distributional Information

Most recently, Sinha et al. [21] surprisingly found that most of MLM’s high performance can in fact be explained by the “distributional prior” rather than its ability to replicate “the types of syntactic and semantic abstractions traditionally believed necessary for language processing [18].” In other words, they found that the success of MLM in downstream tasks is almost entirely because they can model high-order word co-occurrence statistics. To prove this, they pretrained MLMs on sentences with a randomly shuffled word order and showed that after fine-tuning many downstream tasks, these models can still achieve high accuracy, including tasks designed specifically to be challenging for models that ignore the word order. According to some parametric syntactic probes, these models perform surprisingly well, which indicates possible deficiencies when testing the representation for syntactic information [21].


## 3. Motivation and Approach

Firstly, as described in Section2.1, most of the existing MLMs adopt different mask rules to improve their performance. Some take a word as a mask unit; some take phrases, entities, or spans as mask units; and others adopt multilevel masking schemes. As we can see, the length of the masked text is one of the basic variables in the above masking schemes. However, at present, there is a lack of quantitative research and analysis on the performance of MLM with different masking lengths. The whole word masking [6] only masks words, and entity and phrase masking schemes [7, 8] mask only entities or phrases. Span masking [9] simply uses geometric distribution in the process of selecting spans. N-gram masking in MacBERT [10] also just masks different N-grams in a fixed probability. However, for MRC tasks with different answer lengths, if different masking lengths of MLM are used, will the performance achieved be different? There is no relevant research yet. In addition, some MLMs use a multilevel masking scheme, such as the knowledge masking in ERNIE [7, 8]. However, what is the optimal proportion of masking schemes of different levels? It is also a question worth studying. If there is a correlation between performances of MLM and masking lengths in MRC tasks with different answer lengths, then we can choose an appropriate length for an MLM according to the length distribution in the MRC dataset.Secondly, from the interpretability of MLM described in Section2.2, we can see that the theoretical analysis of the MLM is very challenging. There are many empirical studies trying to understand why MLMs are so effective, and one possible explanation for the impressive performance of MLMs is that these models have learned semantic information and syntactic information. A lot of work has been devoted to revealing whether MLM calculates various types of structured information [15–20, 22–24].However, the most recent studies have pointed out that the success of MLM may actually come from the word distribution information it learns to a large extent, and they found that the success of MLM in downstream tasks is almost entirely because they can model high-order word co-occurrence statistics [21].Inspired by this research, we wonder whether the distribution of masking length will also affect the performance of MLM in MRC tasks with different answer lengths. In this work, we try to uncover how much of MLM’s success comes from the correlation between masking length distribution and answer length in the MRC dataset. We treat the distribution of answer lengths in the MRC dataset and the masking length of MLMs as latent variables and treat the performance of different MLMs on downstream MRC tasks with different answer distributions as functions of latent variables. Assuming that the distribution of the answer length in the MRC dataset is correlated with the masking length of the MLM, then pretraining on a large corpus allows MLM to learn the hidden information of different lengths. Therefore, in the downstream MRC task, the MLM whose masking length is closest to the answer length distribution in the MRC dataset should achieve better performance.The key starting point of our research work is to propose an evaluation framework to quantitatively verify whether masking schemes of different lengths will affect the results of the MLM in MRC tasks with different answer lengths. However, using the existing pretrained models and MRC dataset to create a verification framework is challenging because there are too many different factors affecting performance. Since there are many inconsistencies in the pretraining corpus, pretraining methods, downstream tasks, and evaluation datasets used by different pretrained models, therefore, it is difficult to conduct ablation experiments on different masking schemes.To address the above issues, we first design four MRC tasks and construct the related datasets with different answer lengths. Next, using unified hyperparameters, we retrain several MLMs with different masking lengths according to the answer lengths of the above MRC datasets. Then, ablation experiments are carried out, and we evaluate the performance of different MLMs on the above MRC dataset. The key points of our experiment are as follows.


### 3.1. New MRC Tasks and Datasets

When designing the MRC tasks and datasets, we integrate the mainstream MRC tasks, namely, the cloze test, multiple choice, span extraction, and free answering [2, 3], and we adopt two kinds of MRC tasks, including the span extraction tasks and two multiple-choice cloze tasks. The answer length of these two tasks can also be divided into two categories: long answer and short answer. Finally, we construct four MRC datasets, namely, the short span dataset, long span dataset, short cloze dataset, and long cloze dataset. In addition, because the Chinese corpus is composed of Chinese characters, there is no well-marked word boundary, which is conducive to eliminate the influence of word boundary information in the pretrained model. Therefore, we chose to create Chinese MRC datasets.


### 3.2. Training MLM from Scratch

Existing MLMs usually integrate a variety of improvements, such as MLM in the SpanBERT [9] which uses both span masking and SBO pretraining tasks [9] at the same time. In order to eliminate the influence of the prior knowledge embedded in the pretrained model, in this experiment, we do not directly use the existing MLMs, but we conducted MLM training from scratch by ourselves, thereby eliminating the interference variables.


### 3.3. Unified Pretraining Corpora

When training MLMs with different masking lengths, we use the same pretraining corpora to eliminate the impact of word distribution in different corpora.


### 3.4. Answer Length Distribution of Datasets

In this article, in order to quantitatively verify whether masking schemes of different lengths will affect the performance of the MLM, we have counted the length distributions of different datasets.


### 3.5. Masking Length Distribution of MLMs

In the process of training different MLMs, we use the weighted average answer length distribution in the dataset as the MLM mask length to quantitatively verify whether masking schemes of different lengths will affect the results of the MLM.


### 3.6. Unified Masking Ratios

During the experiment, we fixed the masking ratio to be the same as the original version of BERT. That is, select 15% of the text in the paragraph, and 80% of the selected text are replaced byMASK, 10% are replaced by random tokens, 10% are replaced by original tokens, and 10% remain unchanged. We perform this replacement at the sequence level; that is, each time, all tokens in a sequence are replaced with “mask” or random tokens or remain unchanged.


### 3.7. Unified Pretraining Hyperparameters

In order to eliminate the influencing factors, we use the same model hyperparameters in the pretraining of different MLMs.


## 3.1. New MRC Tasks and Datasets

When designing the MRC tasks and datasets, we integrate the mainstream MRC tasks, namely, the cloze test, multiple choice, span extraction, and free answering [2, 3], and we adopt two kinds of MRC tasks, including the span extraction tasks and two multiple-choice cloze tasks. The answer length of these two tasks can also be divided into two categories: long answer and short answer. Finally, we construct four MRC datasets, namely, the short span dataset, long span dataset, short cloze dataset, and long cloze dataset. In addition, because the Chinese corpus is composed of Chinese characters, there is no well-marked word boundary, which is conducive to eliminate the influence of word boundary information in the pretrained model. Therefore, we chose to create Chinese MRC datasets.


## 3.2. Training MLM from Scratch

Existing MLMs usually integrate a variety of improvements, such as MLM in the SpanBERT [9] which uses both span masking and SBO pretraining tasks [9] at the same time. In order to eliminate the influence of the prior knowledge embedded in the pretrained model, in this experiment, we do not directly use the existing MLMs, but we conducted MLM training from scratch by ourselves, thereby eliminating the interference variables.


## 3.3. Unified Pretraining Corpora

When training MLMs with different masking lengths, we use the same pretraining corpora to eliminate the impact of word distribution in different corpora.


## 3.4. Answer Length Distribution of Datasets

In this article, in order to quantitatively verify whether masking schemes of different lengths will affect the performance of the MLM, we have counted the length distributions of different datasets.


## 3.5. Masking Length Distribution of MLMs

In the process of training different MLMs, we use the weighted average answer length distribution in the dataset as the MLM mask length to quantitatively verify whether masking schemes of different lengths will affect the results of the MLM.


## 3.6. Unified Masking Ratios

During the experiment, we fixed the masking ratio to be the same as the original version of BERT. That is, select 15% of the text in the paragraph, and 80% of the selected text are replaced byMASK, 10% are replaced by random tokens, 10% are replaced by original tokens, and 10% remain unchanged. We perform this replacement at the sequence level; that is, each time, all tokens in a sequence are replaced with “mask” or random tokens or remain unchanged.


## 3.7. Unified Pretraining Hyperparameters

In order to eliminate the influencing factors, we use the same model hyperparameters in the pretraining of different MLMs.


## 4. Proposed MRC Tasks

According to the style of the answers and questions, MRC tasks can be roughly divided into four categories: cloze test, multiple choice, span extraction, and free answering [2, 3]. When designing MRC tasks and datasets required for ablation experiments, we integrate the main characteristics of these MRC tasks, and we adopt two kinds of MRC tasks, including the span extraction tasks and two multiple-choice cloze tasks. The answer length of these two tasks can also be divided into two categories: long answer and short answer. Finally, the four MRC tasks are as follows:
(1)
Span extraction tasks with short answers(2)
Span extraction tasks with long answers(3)
Multiple-choice cloze tasks with short answers(4)
Multiple-choice cloze tasks with long answersWe believe these tasks are representative of most of the current MRC tasks. Among them, the number of tokens in the short answer of span extraction tasks is set to be greater than 3 and less than 7, and the size of the long answer is greater than 6 and less than 10; the number of tokens in the short answer of multiple-choice cloze tasks is greater than 6 and less than 15, and the size of the long answer is greater than 16 and less than 30.In the following subsections, we briefly introduce the definitions of typical MRC tasks and the two types of MRC tasks we used in the experiment.


### 4.1. Typical MRC Tasks

Generally, the definition of a typical MRC task is given below.Definition 1.
A typical machine reading comprehension task could be formulated as a supervised learning problem. Given the training examplesp,q,a, where p is a passage and q is a question, the goal of a typical machine reading comprehension task is to learn a predictor f which takes the passage p and a corresponding question q as inputs and gives the answer a as output, which could be formulated as the following formula [2–4]:
(1)a=fp,q,and it is necessary that a majority of native speakers would agree what the question q does regarding that text p, and the answer a is a correct one which does not contain information irrelevant to that question.


### 4.2. Span Extraction Tasks with Different Answer Lengths

In order to quantitatively verify whether masking schemes with different lengths will affect the performance of MLM, we propose two span extraction tasks with different answer lengths for Chinese machine reading comprehension. Table1 shows an example of the proposed span extraction task.Table 1
An example of the proposed span extraction task.
Context:传染病的医学治疗属于传染病医学领域，在某些情况下传播学的研究属于流行病学领域。一般来说，感染最初由初级保健医生或内科专家诊断。例如，“简单”肺炎通常由内科医师或肺科医师(肺科医师)治疗。因此，传染病专家的工作需要与病人和全科医生以及实验室科学家、免疫学家、细菌学家和其他专家合作。Context:The medical treatment of infectious diseases belongs to the field of infectious disease medicine. In some cases, the research of communication belongs to the field of epidemiology. In general, infection is initially diagnosed by a primary care physician or medical expert. For example, “simple” pneumonia is usually treated by a physician or pulmonary physician (pulmonary physician). Therefore, the work of infectious disease experts requires cooperation with patients and general practitioners, as well as laboratory scientists, immunologists, bacteriologists, and other experts.Question:疾病传播可以归入哪些研究领域的范畴?Question:What research areas can disease transmission fall into?Answer:流行病学Answer:The field of epidemiologyThe definition of the span extraction task is as follows.Definition 2.
Given a serial of training samples, each sample contains a passage about a public service event, a corresponding question, and the answer to this question. The answer should be a span which is directly extracted from the passage. The goal of the span extraction machine reading comprehension task is to train the machine so that it can find the correct answers in the given passage. The task can be simplified by predicting the start and end pointers of the right answer in the given passage.


### 4.3. Multiple-Choice Cloze Tasks with Different Answer Lengths

We also proposed two multiple-choice cloze tasks with different answer lengths for Chinese machine reading comprehension. The form of our multiple-choice cloze tasks is similar to the CMRC2019 task [25], but redundant fake answers are removed. Table 2 shows an example of the proposed multiple-choice cloze task.Table 2
An example of the proposed multiple-choice cloze task.
Passage:2001年初，苹果开始销售带有CD-RW驱动器的电脑，并强调Mac电脑的DVD播放能力，将DVD-ROM和DVD-RAM驱动器作为标准配置。[BLANK1]，苹果在可写CD技术方面“迟到了”，但他认为，Mac电脑可以成为一个“数字中心”，连接并促成一种“新兴的数字生活方式”。苹果随后将对其iTunes音乐播放器软件进行更新，同时推出一项颇具争议的“翻录、混合、刻录”广告活动，[BLANK2]。[BLANK3]，iPod是苹果第一款成功的手持设备。[BLANK4]，例如失败的Power Macg4cube、面向教育的EMAC和面向专业人士的钛(后来的铝)PowerBook G4笔记本电脑。直到1989年苹果公司发布了他们的第一台便携式电脑，[BLANK5]。[BLANK6]，它很快在1991年被第一条PowerBook线取代。Passage:In early 2001, Apple began selling computers with CD-RW drives, emphasized the DVD playback capability of MAC computers, and took DVD-ROM and DVD-RAM drives as standard configurations. [BLANK1] Apple was “late” in the technology of writable CD, but he believed that the MAC could become a “Digital Center” to connect and promote an “emerging digital lifestyle.” Apple will then update its iTunes music player software, and launch a controversial “rip, mix, and burn” advertising campaign, [BLANK2], [BLANK3], iPod is Apple’s first successful handheld device [BLANK4], such as the failed Power Macg4cube, the EMAC for education, and the titanium (later aluminum) PowerBook G4 laptop for professionals. Until 1989, Apple released their first portable computer, [BLANK5]. [BLANK6], which was soon replaced by the first PowerBook line in 1991.Options:A: “一些人认为这鼓励了媒体盗版”B: “伴随着iPod的发布”C: “一种小型便携式设备”D: “史蒂夫•乔布斯承认”E: “Macintosh便携式电脑”F: “苹果继续推出产品”Options:A: “some people think this encourages media piracy”B: “with the release of iPod”C: “a small portable device”D: “Steve Jobs admitted that”E: “Macintosh laptop”F: “Apple continues to launch products”Answer:D, F, E, C, A, BAnswer:D, F, E, C, A, BThe definition of the multiple-choice cloze task is as follows.Definition 3.
Generally, the reading comprehension task can be described as a tripleP,Q,A, where P represents Passage, Q represents Question, and A represents Answer. Specifically, for the multiple-choice cloze-style reading comprehension task, we select several sentences in the passages and replace them with special marks (for example, BLANK), forming an incomplete passage. The selected sentences form a candidate list, and the machine should fill in the blanks with these candidate sentences to form a complete passage [2–4, 25].


## 4.1. Typical MRC Tasks

Generally, the definition of a typical MRC task is given below.Definition 1.
A typical machine reading comprehension task could be formulated as a supervised learning problem. Given the training examplesp,q,a, where p is a passage and q is a question, the goal of a typical machine reading comprehension task is to learn a predictor f which takes the passage p and a corresponding question q as inputs and gives the answer a as output, which could be formulated as the following formula [2–4]:
(1)a=fp,q,and it is necessary that a majority of native speakers would agree what the question q does regarding that text p, and the answer a is a correct one which does not contain information irrelevant to that question.


## 4.2. Span Extraction Tasks with Different Answer Lengths

In order to quantitatively verify whether masking schemes with different lengths will affect the performance of MLM, we propose two span extraction tasks with different answer lengths for Chinese machine reading comprehension. Table1 shows an example of the proposed span extraction task.Table 1
An example of the proposed span extraction task.
Context:传染病的医学治疗属于传染病医学领域，在某些情况下传播学的研究属于流行病学领域。一般来说，感染最初由初级保健医生或内科专家诊断。例如，“简单”肺炎通常由内科医师或肺科医师(肺科医师)治疗。因此，传染病专家的工作需要与病人和全科医生以及实验室科学家、免疫学家、细菌学家和其他专家合作。Context:The medical treatment of infectious diseases belongs to the field of infectious disease medicine. In some cases, the research of communication belongs to the field of epidemiology. In general, infection is initially diagnosed by a primary care physician or medical expert. For example, “simple” pneumonia is usually treated by a physician or pulmonary physician (pulmonary physician). Therefore, the work of infectious disease experts requires cooperation with patients and general practitioners, as well as laboratory scientists, immunologists, bacteriologists, and other experts.Question:疾病传播可以归入哪些研究领域的范畴?Question:What research areas can disease transmission fall into?Answer:流行病学Answer:The field of epidemiologyThe definition of the span extraction task is as follows.Definition 2.
Given a serial of training samples, each sample contains a passage about a public service event, a corresponding question, and the answer to this question. The answer should be a span which is directly extracted from the passage. The goal of the span extraction machine reading comprehension task is to train the machine so that it can find the correct answers in the given passage. The task can be simplified by predicting the start and end pointers of the right answer in the given passage.


## 4.3. Multiple-Choice Cloze Tasks with Different Answer Lengths

We also proposed two multiple-choice cloze tasks with different answer lengths for Chinese machine reading comprehension. The form of our multiple-choice cloze tasks is similar to the CMRC2019 task [25], but redundant fake answers are removed. Table 2 shows an example of the proposed multiple-choice cloze task.Table 2
An example of the proposed multiple-choice cloze task.
Passage:2001年初，苹果开始销售带有CD-RW驱动器的电脑，并强调Mac电脑的DVD播放能力，将DVD-ROM和DVD-RAM驱动器作为标准配置。[BLANK1]，苹果在可写CD技术方面“迟到了”，但他认为，Mac电脑可以成为一个“数字中心”，连接并促成一种“新兴的数字生活方式”。苹果随后将对其iTunes音乐播放器软件进行更新，同时推出一项颇具争议的“翻录、混合、刻录”广告活动，[BLANK2]。[BLANK3]，iPod是苹果第一款成功的手持设备。[BLANK4]，例如失败的Power Macg4cube、面向教育的EMAC和面向专业人士的钛(后来的铝)PowerBook G4笔记本电脑。直到1989年苹果公司发布了他们的第一台便携式电脑，[BLANK5]。[BLANK6]，它很快在1991年被第一条PowerBook线取代。Passage:In early 2001, Apple began selling computers with CD-RW drives, emphasized the DVD playback capability of MAC computers, and took DVD-ROM and DVD-RAM drives as standard configurations. [BLANK1] Apple was “late” in the technology of writable CD, but he believed that the MAC could become a “Digital Center” to connect and promote an “emerging digital lifestyle.” Apple will then update its iTunes music player software, and launch a controversial “rip, mix, and burn” advertising campaign, [BLANK2], [BLANK3], iPod is Apple’s first successful handheld device [BLANK4], such as the failed Power Macg4cube, the EMAC for education, and the titanium (later aluminum) PowerBook G4 laptop for professionals. Until 1989, Apple released their first portable computer, [BLANK5]. [BLANK6], which was soon replaced by the first PowerBook line in 1991.Options:A: “一些人认为这鼓励了媒体盗版”B: “伴随着iPod的发布”C: “一种小型便携式设备”D: “史蒂夫•乔布斯承认”E: “Macintosh便携式电脑”F: “苹果继续推出产品”Options:A: “some people think this encourages media piracy”B: “with the release of iPod”C: “a small portable device”D: “Steve Jobs admitted that”E: “Macintosh laptop”F: “Apple continues to launch products”Answer:D, F, E, C, A, BAnswer:D, F, E, C, A, BThe definition of the multiple-choice cloze task is as follows.Definition 3.
Generally, the reading comprehension task can be described as a tripleP,Q,A, where P represents Passage, Q represents Question, and A represents Answer. Specifically, for the multiple-choice cloze-style reading comprehension task, we select several sentences in the passages and replace them with special marks (for example, BLANK), forming an incomplete passage. The selected sentences form a candidate list, and the machine should fill in the blanks with these candidate sentences to form a complete passage [2–4, 25].


## 5. Evaluation Metrics

In this paper, we use F1 and EM to measure the performance of the pretrained model in the span extraction tasks.


### 5.1. F1 Score

F1 is a commonly used MRC task evaluation metric. The equation of F1 for a single question is as follows:(2)F1=2×Precision×RecallPrecision+Recall,where P denotes the token-level Precision for a single question and R denotes the Recall for a single question [2–4].


### 5.2. Precision

Precision represents the percentage of the maximum span overlap between the tokens in the correct answer and the tokens in the predicted answer. In order to calculate Precision, we first need to obtain the true positive (TP), false positive (FP), true negative (TN), and false negative (FN), as shown in Figure1.Figure 1
The true positive (TP), false positive (FP), true negative (TN), and false negative (FN).As shown in Figure1, for a single question in the proposed dataset, the true positive (TP) is equal to the maximum common span (MCS) between the predicted answer and the correct answer. The false positive (FP) indicates the span not in the correct answer but in the predicted answer, while the false negative (FN) indicates the span not in the predicted answer but in the correct answer [2–4]. The Precision of a single question is calculated as follows:
(3)Precision=TPTP+FP.


### 5.3. Recall

Recall represents the percentage of correct answers that have been correctly predicted in the question [2–4]. According to the above definitions of the true positive (TP), false positive (FP), and false negative (FN), the Recall of a single answer is calculated as follows:
(4)Recall=TPTP+FN,where Recall represents the recall rate of a single problem, NumTP represents the number of true positive (TP) tokens, and NumFN represents the number of false negative (FN) tokens.


### 5.4. Exact Match

Exact Match represents the percentage of questions where the answer generated by the system exactly matches the correct answer, which means that every word is the same. Exact Match is usually abbreviated as EM. In the span extraction MRC task, the answer to the question is a sentence, and some words in the predicted answer may be included in the correct answer, while other words are not included in the correct answer [2–4]. For example, if the MRC task contains N questions, each question corresponds to a correct answer. The answer can be a word, a phrase, or a sentence, and the number of predicted answers exactly the same as the correct answer is M. Exact Match can be calculated as follows:
(5)ExactMatch=MN.


### 5.5. Accuracy

In this paper, we use Accuracy to measure the performance of the pretrained model in the multiple-choice cloze tasks. Accuracy is defined as the ratio of the number of correctly predicted samples to the total number of samples for a given test dataset.For example, suppose that an MRC task containsN questions; each question corresponds to one correct answer; the answers can be a word, a phrase, or a sentence; and the number of questions that the system answers correctly is M. The equation for the Accuracy is as follows:
(6)Accuracy=MN.In addition, in order to make the assessment more reliable, following the evaluation method of CMRC2019 [25], we adopt two metrics to evaluate the systems on our datasets, which are Question-level Accuracy (QAC) and Passage-level Accuracy (PAC).The Question-level Accuracy (QAC) is the ratio between the correct prediction and the total blanks, which can be calculated by the following formula [25]:
(7)QAC=#CorrectlyAnsweredQuestions#TotalQuestions.Similar to the QAC, Passage-level Accuracy (PAC) measures how many passages have been correctly answered. We only count the passages where all blanks have been correctly predicted [25].Passage-level Accuracy (PAC) is used to measure how many passages are answered exactly correctly. Similar to the Exact Match, only paragraphs where all blanks are correctly predicted are considered to be exactly correctly predicted samples. Passage-level Accuracy (PAC) can be calculated by the following formula [25]:
(8)PAC=#ExactlyCorrectParagraphs#TotalParagraphs.


## 5.1. F1 Score

F1 is a commonly used MRC task evaluation metric. The equation of F1 for a single question is as follows:(2)F1=2×Precision×RecallPrecision+Recall,where P denotes the token-level Precision for a single question and R denotes the Recall for a single question [2–4].


## 5.2. Precision

Precision represents the percentage of the maximum span overlap between the tokens in the correct answer and the tokens in the predicted answer. In order to calculate Precision, we first need to obtain the true positive (TP), false positive (FP), true negative (TN), and false negative (FN), as shown in Figure1.Figure 1
The true positive (TP), false positive (FP), true negative (TN), and false negative (FN).As shown in Figure1, for a single question in the proposed dataset, the true positive (TP) is equal to the maximum common span (MCS) between the predicted answer and the correct answer. The false positive (FP) indicates the span not in the correct answer but in the predicted answer, while the false negative (FN) indicates the span not in the predicted answer but in the correct answer [2–4]. The Precision of a single question is calculated as follows:
(3)Precision=TPTP+FP.


## 5.3. Recall

Recall represents the percentage of correct answers that have been correctly predicted in the question [2–4]. According to the above definitions of the true positive (TP), false positive (FP), and false negative (FN), the Recall of a single answer is calculated as follows:
(4)Recall=TPTP+FN,where Recall represents the recall rate of a single problem, NumTP represents the number of true positive (TP) tokens, and NumFN represents the number of false negative (FN) tokens.


## 5.4. Exact Match

Exact Match represents the percentage of questions where the answer generated by the system exactly matches the correct answer, which means that every word is the same. Exact Match is usually abbreviated as EM. In the span extraction MRC task, the answer to the question is a sentence, and some words in the predicted answer may be included in the correct answer, while other words are not included in the correct answer [2–4]. For example, if the MRC task contains N questions, each question corresponds to a correct answer. The answer can be a word, a phrase, or a sentence, and the number of predicted answers exactly the same as the correct answer is M. Exact Match can be calculated as follows:
(5)ExactMatch=MN.


## 5.5. Accuracy

In this paper, we use Accuracy to measure the performance of the pretrained model in the multiple-choice cloze tasks. Accuracy is defined as the ratio of the number of correctly predicted samples to the total number of samples for a given test dataset.For example, suppose that an MRC task containsN questions; each question corresponds to one correct answer; the answers can be a word, a phrase, or a sentence; and the number of questions that the system answers correctly is M. The equation for the Accuracy is as follows:
(6)Accuracy=MN.In addition, in order to make the assessment more reliable, following the evaluation method of CMRC2019 [25], we adopt two metrics to evaluate the systems on our datasets, which are Question-level Accuracy (QAC) and Passage-level Accuracy (PAC).The Question-level Accuracy (QAC) is the ratio between the correct prediction and the total blanks, which can be calculated by the following formula [25]:
(7)QAC=#CorrectlyAnsweredQuestions#TotalQuestions.Similar to the QAC, Passage-level Accuracy (PAC) measures how many passages have been correctly answered. We only count the passages where all blanks have been correctly predicted [25].Passage-level Accuracy (PAC) is used to measure how many passages are answered exactly correctly. Similar to the Exact Match, only paragraphs where all blanks are correctly predicted are considered to be exactly correctly predicted samples. Passage-level Accuracy (PAC) can be calculated by the following formula [25]:
(8)PAC=#ExactlyCorrectParagraphs#TotalParagraphs.


## 6. Dataset Construction

As mentioned above, in order to eliminate the influence of interference factors on the experiment as much as possible, we designed four MRC tasks: short span extraction task, long span extraction task, short multiple-choice cloze task, and long multiple-choice cloze task. In this section, we further construct four Chinese MRC datasets for these MRC tasks. Unlike the English text, a feature of the Chinese text is that there are no obvious spaces to mark word boundaries, so the influence of word boundary information on the results can be further eliminated. So in this article, we use Chinese as the language of the dataset. Below, we will briefly introduce the construction methods of these MRC datasets.


### 6.1. Span Extraction Dataset with Different Answer Lengths

The corpus of our span extraction datasets comes from the paragraphs in the Chinese SQuAD dataset [26]. The Stanford Question Answering Dataset (SQuAD) [27] is one of the most popular machine reading comprehension datasets, containing more than 100,000 questions generated by humans, and the answer to each question is a span of text in a related context [20]. Since its release in 2016, SQuAD1.1 has quickly become the most widely used MRC dataset. Now, it has been updated to SQuAD2.0 [4, 28].The Chinese SQuAD dataset [26] is translated from the original SQuAD through machine translation and manual correction, including SQuAD1.1 [27] and SQuAD2.0 [28]. Because some translations cannot find the answers in the original text (the answer translation and document translation are different), the amount of data is reduced compared to the original English version of SQuAD. After data cleaning, the Chinese SQuAD dataset contains 125,892 questions and 36,100 paragraphs, and the number of unanswerable questions is 49,443 [26]. Among them, each paragraph includes a number of different contexts, and each context includes multiple question-and-answer pairs. Then, we divided the paragraphs in the Chinese SQuAD dataset according to the length of the answer and obtained the long span extraction dataset and the short span extraction dataset, where the number of tokens in the short answer of span extraction tasks is set to be greater than 3 and less than 7, and the size of the long answer is greater than 6 and less than 10. The statistics of our span extraction datasets is shown in the sections below.


### 6.2. Multiple-Choice Cloze Dataset with Different Answer Lengths

The corpus source of our multiple-choice cloze dataset is the NLPCC 2017 corpus [29]. The cleaned NLPCC 2017 corpus contains 50,000 news articles with summaries, and the average number of tokens in an article is 1036 [29].We first divide the above corpus into several paragraphs and then divide each paragraph into sentences using commas, periods, semicolons, exclamation marks, and question marks as the dividing point. Then, when constructing the multiple-choice cloze dataset with short answers, for each paragraph, we randomly select 9 sentences as candidate long answers, and the number of tokens in these sentences is greater than 6 and less than 15. When constructing the multiple-choice cloze dataset with long answers, for each paragraph, we also randomly select 9 sentences as candidate long answers, and the number of tokens in these sentences is greater than 16 and less than 30. After selecting the candidate answers, we randomly shuffle the order of the answers to obtain candidate options in the form of multiple choices. The statistics of our multiple-choice cloze datasets is shown in sections below.


### 6.3. Dataset Analysis

In this subsection, we analyze the paragraphs, questions, and answers in the proposed datasets. Specifically, we explore (1) the statistics of the data size and (2) the length distribution of the answer lengths in the train set, development set, and test set of the proposed datasets. As we can see, the statistics of the proposed span extraction datasets are given in Table3, and Table 4 shows the statistics of the proposed multiple-choice cloze datasets.Table 3
Statistics of the proposed span extraction datasets.
Short span extractionLong span extractionTrain setDev setTest setTrain setDev setTest setParagraph #600150200600150200Passage #15,26527702484775611631565Question #31,3906774830912,05323765812Max tokens in a context #512512512512512512Max answer tokens #666999Min answer tokens #444777Table 4
Statistics of the proposed multiple-choice cloze datasets.
Short multiple-choice clozeLong multiple-choice clozeTrain setDev setTest setTrain setDev setTest setPassages #450010001000450010001000Blanks #40,5009000900040,50090009000Max tokens in a passage #100010001000100010001000Max answer tokens #141414292929Min answer tokens #777171717Options #999999


### 6.4. Distribution of Answer Lengths

We have separately counted the distribution of answer lengths in these four datasets. Tables5 and 6 show the answer length distributions of the train set, development set, and test set. For example, in the short span extraction dataset, there are 16,171 answers that have 4 tokens in the training set, and 3344 answers that have 4 tokens in the development set, while 4147 in the test set.Table 5
The answer length distributions of the proposed span extraction datasets.
(a)Short span datasetTokens #Train #PP (%)Dev #PP (%)Test #PP (%)416,17151.52%334449.37%414749.91%5856627.29%182826.99%223026.84%6665321.19%160223.65%193223.25%Total31,390100.00%6774100.00%8309100.00%(b)Long span datasetTokens #Train #PP (%)Dev #PP (%)Test #PP (%)7504041.82%81934.47%207135.63%8419234.78%98041.25%239041.12%9282123.40%57724.28%135123.25%Total12,053100.00%2376100.00%5812100.00%Table 6
The answer length distributions of the proposed multiple-choice cloze datasets.
(a)Short cloze datasetTokens #Train #PP (%)Dev #PP (%)Test #PP (%)7442810.93%97410.82%101611.29%8463011.43%102211.36%103711.52%9502012.40%116612.96%107711.97%10526012.99%121213.47%115412.82%11538913.31%114912.77%125313.92%12542713.40%120513.39%120013.33%13530813.11%117113.01%113612.62%14503812.44%110112.23%112712.52%Total40,500100.00%9000100.00%9000100.00%(b)Long cloze datasetTokens #Train #PP (%)Dev #PP (%)Test #PP (%)17520912.86%113212.58%117413.04%18491912.15%111412.38%107211.91%19436710.78%99911.10%98210.91%2039839.83%8919.90%8759.72%2136378.98%7628.47%7708.56%2231877.87%7147.93%7157.94%2329807.36%6557.28%6026.69%2426276.49%5766.40%5906.56%2522755.62%5466.07%5706.33%2621625.34%4675.19%5045.60%2718854.65%4084.53%4274.74%2817764.39%3854.28%3964.40%2914933.69%3513.90%3233.59%Total40,500100.00%9000100.00%9000100.00%Based on the data in the above table, we have also given the illustration of answer length distribution ratios in different MRC datasets. For example, it can be seen from Figure2(a). The blue squares represent the proportion of answers with length 4, the red squares represent the proportion of answers with length 5, and the green squares represent the proportion of answers with length 5.Figure 2
The illustration of answer length distribution ratios in different MRC datasets. (a) The short span dataset. (b) The long span dataset. (c) The short cloze dataset. (d) The long cloze dataset.
(a)(b)(c)(d)


### 6.5. Dataset Comparison

The statistics of the proposed dataset have been given in the previous section. In this section, we compare the proposed dataset with the other MRC datasets. The comparison of the number of questions is shown in Table7. In contrast to prior MRC datasets, the question size of the proposed dataset is at a medium level.Table 7
The number of questions of each MRC dataset.
DatasetQuestion #Train question #Dev question #Test question #Percentage of the train setSQuAD2.0151,054130,31911,873886286.27%SQuAD1.1107,70287,59910,570953381.33%TQA26,26015,1545309579757.71%MovieQA21,40614,1662844439666.18%MCScript13,93997311411279769.81%DREAM10,19761162040204159.98%ARC-E51972251570237643.31%WikiQA3047211829663369.51%ARC-C25901119299117243.20%ProPara488391544380.12%Short span46,47331,3906774830967.54%Long span20,24112,0532376581259.55%Short cloze58,50040,5009000900069.23%Long cloze58,50040,5009000900069.23%Next, the statistics of the context size are given in Table8. As we can see, in contrast to prior MRC datasets, the context size of the proposed dataset is also at a medium level.Table 8
The number of contexts of each MRC dataset.
DatasetContext #Train context #Dev context #Test context #Unit of the contextCoQA83997199500700PassageCLOTH71315513805813PassageDREAM6444386912881287DialogueQangaroo-M25081620342546PassageTQA1076666200210LessonMovieQA54836277109MovieSQuAD1.15364424846ArticleSQuAD2.05054423528ArticleShort span20,51915,26527702484PassageLong span10,484775611631565PassageShort cloze58,50040,50090009000PassageLong cloze58,50040,50090009000PassageWe also compared the question style, answer style, source of corpora, and generation method of each dataset, as shown in Table9.Table 9
Analysis of each MRC dataset.
DatasetQuestion styleAnswer styleCorpus sourceGeneration methodDREAMNaturalFree-formExamCrowdsourcingMCScriptNaturalFree-formNarrative textsCrowdsourcingMovieQANaturalFree-formMovie plotCrowdsourcingOpenBookQANaturalFree-formElementary-level science factsCrowdsourcingQangaroo-MSynthesisSpansWikipedia articlesCrowdsourcingQangaroo-WSynthesisSpansPaper abstractsCrowdsourcingSQuAD1.1NaturalSpansWikipediaCrowdsourcingSQUAD2.0NaturalSpansWikipediaCrowdsourcingWikiQANaturalFree-formRelevant Wikipedia pagesCrowdsourcingTQANaturalFree-formMiddle school curriculaExpertsShort spanNaturalSpansWikipediaCrowdsourcingLong spanNaturalSpansWikipediaCrowdsourcingShort clozeClozeMultiple choiceWikipedia/newsAutomatedLong clozeClozeMultiple choiceWikipedia/newsAutomated


## 6.1. Span Extraction Dataset with Different Answer Lengths

The corpus of our span extraction datasets comes from the paragraphs in the Chinese SQuAD dataset [26]. The Stanford Question Answering Dataset (SQuAD) [27] is one of the most popular machine reading comprehension datasets, containing more than 100,000 questions generated by humans, and the answer to each question is a span of text in a related context [20]. Since its release in 2016, SQuAD1.1 has quickly become the most widely used MRC dataset. Now, it has been updated to SQuAD2.0 [4, 28].The Chinese SQuAD dataset [26] is translated from the original SQuAD through machine translation and manual correction, including SQuAD1.1 [27] and SQuAD2.0 [28]. Because some translations cannot find the answers in the original text (the answer translation and document translation are different), the amount of data is reduced compared to the original English version of SQuAD. After data cleaning, the Chinese SQuAD dataset contains 125,892 questions and 36,100 paragraphs, and the number of unanswerable questions is 49,443 [26]. Among them, each paragraph includes a number of different contexts, and each context includes multiple question-and-answer pairs. Then, we divided the paragraphs in the Chinese SQuAD dataset according to the length of the answer and obtained the long span extraction dataset and the short span extraction dataset, where the number of tokens in the short answer of span extraction tasks is set to be greater than 3 and less than 7, and the size of the long answer is greater than 6 and less than 10. The statistics of our span extraction datasets is shown in the sections below.


## 6.2. Multiple-Choice Cloze Dataset with Different Answer Lengths

The corpus source of our multiple-choice cloze dataset is the NLPCC 2017 corpus [29]. The cleaned NLPCC 2017 corpus contains 50,000 news articles with summaries, and the average number of tokens in an article is 1036 [29].We first divide the above corpus into several paragraphs and then divide each paragraph into sentences using commas, periods, semicolons, exclamation marks, and question marks as the dividing point. Then, when constructing the multiple-choice cloze dataset with short answers, for each paragraph, we randomly select 9 sentences as candidate long answers, and the number of tokens in these sentences is greater than 6 and less than 15. When constructing the multiple-choice cloze dataset with long answers, for each paragraph, we also randomly select 9 sentences as candidate long answers, and the number of tokens in these sentences is greater than 16 and less than 30. After selecting the candidate answers, we randomly shuffle the order of the answers to obtain candidate options in the form of multiple choices. The statistics of our multiple-choice cloze datasets is shown in sections below.


## 6.3. Dataset Analysis

In this subsection, we analyze the paragraphs, questions, and answers in the proposed datasets. Specifically, we explore (1) the statistics of the data size and (2) the length distribution of the answer lengths in the train set, development set, and test set of the proposed datasets. As we can see, the statistics of the proposed span extraction datasets are given in Table3, and Table 4 shows the statistics of the proposed multiple-choice cloze datasets.Table 3
Statistics of the proposed span extraction datasets.
Short span extractionLong span extractionTrain setDev setTest setTrain setDev setTest setParagraph #600150200600150200Passage #15,26527702484775611631565Question #31,3906774830912,05323765812Max tokens in a context #512512512512512512Max answer tokens #666999Min answer tokens #444777Table 4
Statistics of the proposed multiple-choice cloze datasets.
Short multiple-choice clozeLong multiple-choice clozeTrain setDev setTest setTrain setDev setTest setPassages #450010001000450010001000Blanks #40,5009000900040,50090009000Max tokens in a passage #100010001000100010001000Max answer tokens #141414292929Min answer tokens #777171717Options #999999


## 6.4. Distribution of Answer Lengths

We have separately counted the distribution of answer lengths in these four datasets. Tables5 and 6 show the answer length distributions of the train set, development set, and test set. For example, in the short span extraction dataset, there are 16,171 answers that have 4 tokens in the training set, and 3344 answers that have 4 tokens in the development set, while 4147 in the test set.Table 5
The answer length distributions of the proposed span extraction datasets.
(a)Short span datasetTokens #Train #PP (%)Dev #PP (%)Test #PP (%)416,17151.52%334449.37%414749.91%5856627.29%182826.99%223026.84%6665321.19%160223.65%193223.25%Total31,390100.00%6774100.00%8309100.00%(b)Long span datasetTokens #Train #PP (%)Dev #PP (%)Test #PP (%)7504041.82%81934.47%207135.63%8419234.78%98041.25%239041.12%9282123.40%57724.28%135123.25%Total12,053100.00%2376100.00%5812100.00%Table 6
The answer length distributions of the proposed multiple-choice cloze datasets.
(a)Short cloze datasetTokens #Train #PP (%)Dev #PP (%)Test #PP (%)7442810.93%97410.82%101611.29%8463011.43%102211.36%103711.52%9502012.40%116612.96%107711.97%10526012.99%121213.47%115412.82%11538913.31%114912.77%125313.92%12542713.40%120513.39%120013.33%13530813.11%117113.01%113612.62%14503812.44%110112.23%112712.52%Total40,500100.00%9000100.00%9000100.00%(b)Long cloze datasetTokens #Train #PP (%)Dev #PP (%)Test #PP (%)17520912.86%113212.58%117413.04%18491912.15%111412.38%107211.91%19436710.78%99911.10%98210.91%2039839.83%8919.90%8759.72%2136378.98%7628.47%7708.56%2231877.87%7147.93%7157.94%2329807.36%6557.28%6026.69%2426276.49%5766.40%5906.56%2522755.62%5466.07%5706.33%2621625.34%4675.19%5045.60%2718854.65%4084.53%4274.74%2817764.39%3854.28%3964.40%2914933.69%3513.90%3233.59%Total40,500100.00%9000100.00%9000100.00%Based on the data in the above table, we have also given the illustration of answer length distribution ratios in different MRC datasets. For example, it can be seen from Figure2(a). The blue squares represent the proportion of answers with length 4, the red squares represent the proportion of answers with length 5, and the green squares represent the proportion of answers with length 5.Figure 2
The illustration of answer length distribution ratios in different MRC datasets. (a) The short span dataset. (b) The long span dataset. (c) The short cloze dataset. (d) The long cloze dataset.
(a)(b)(c)(d)


## 6.5. Dataset Comparison

The statistics of the proposed dataset have been given in the previous section. In this section, we compare the proposed dataset with the other MRC datasets. The comparison of the number of questions is shown in Table7. In contrast to prior MRC datasets, the question size of the proposed dataset is at a medium level.Table 7
The number of questions of each MRC dataset.
DatasetQuestion #Train question #Dev question #Test question #Percentage of the train setSQuAD2.0151,054130,31911,873886286.27%SQuAD1.1107,70287,59910,570953381.33%TQA26,26015,1545309579757.71%MovieQA21,40614,1662844439666.18%MCScript13,93997311411279769.81%DREAM10,19761162040204159.98%ARC-E51972251570237643.31%WikiQA3047211829663369.51%ARC-C25901119299117243.20%ProPara488391544380.12%Short span46,47331,3906774830967.54%Long span20,24112,0532376581259.55%Short cloze58,50040,5009000900069.23%Long cloze58,50040,5009000900069.23%Next, the statistics of the context size are given in Table8. As we can see, in contrast to prior MRC datasets, the context size of the proposed dataset is also at a medium level.Table 8
The number of contexts of each MRC dataset.
DatasetContext #Train context #Dev context #Test context #Unit of the contextCoQA83997199500700PassageCLOTH71315513805813PassageDREAM6444386912881287DialogueQangaroo-M25081620342546PassageTQA1076666200210LessonMovieQA54836277109MovieSQuAD1.15364424846ArticleSQuAD2.05054423528ArticleShort span20,51915,26527702484PassageLong span10,484775611631565PassageShort cloze58,50040,50090009000PassageLong cloze58,50040,50090009000PassageWe also compared the question style, answer style, source of corpora, and generation method of each dataset, as shown in Table9.Table 9
Analysis of each MRC dataset.
DatasetQuestion styleAnswer styleCorpus sourceGeneration methodDREAMNaturalFree-formExamCrowdsourcingMCScriptNaturalFree-formNarrative textsCrowdsourcingMovieQANaturalFree-formMovie plotCrowdsourcingOpenBookQANaturalFree-formElementary-level science factsCrowdsourcingQangaroo-MSynthesisSpansWikipedia articlesCrowdsourcingQangaroo-WSynthesisSpansPaper abstractsCrowdsourcingSQuAD1.1NaturalSpansWikipediaCrowdsourcingSQUAD2.0NaturalSpansWikipediaCrowdsourcingWikiQANaturalFree-formRelevant Wikipedia pagesCrowdsourcingTQANaturalFree-formMiddle school curriculaExpertsShort spanNaturalSpansWikipediaCrowdsourcingLong spanNaturalSpansWikipediaCrowdsourcingShort clozeClozeMultiple choiceWikipedia/newsAutomatedLong clozeClozeMultiple choiceWikipedia/newsAutomated


## 7. MLMs with Different Masking Lengths

In order to quantitatively verify whether masking schemes of different lengths will affect the performance of the MLM, in the previous section, we have proposed MRC tasks and constructed MRC datasets with different answer lengths. In this section, as shown in Figure3, we use the above datasets and tasks to propose an evaluation framework for masked language models (MLMs) with different masking lengths. However, existing MLMs usually integrate various improvements. To eliminate the influence of the prior knowledge embedded in the existing MLMs, in this experiment, we do not directly use the existing MLMs but conduct MLM training from scratch by ourselves. We trained four different MLMs, namely, short span MLM, long span MLM, short cloze MLM, and long cloze MLM. When training our MLMs, we used different masking lengths according to the average distribution of answer lengths in the proposed four MRC datasets.Figure 3
Illustration of the masked language models (MLMs) with different masking lengths according to the average distribution of answer lengths in the proposed four MRC datasets.


### 7.1. Masking Schemes

The key point of our mask scheme used in our experiment is that the probability distribution of different masking lengths is equal to the proportional distribution of different answer lengths in the corresponding dataset. For example, as shown in Figure3, in the short multiple-choice cloze dataset, the answer length distribution is shown in Figure 3(e). Suppose that the total number of answers with length l in the dataset is Xl, and the length of l ranges from min to max. Then, we can calculate the actual distribution ratio of the length of each answer as shown in the following formula:
(9)Dl=Xl∑l=minmaxXl.Among them,Dl is the distribution ratio of the number of answers with length l to the total number of answers.Then, we treat the distribution ratios as the probabilities and use them as the probabilities of different lengths being selected in the MLM. The pie chart of this probability distribution is shown in Figure3(a).When training the MLM for the dataset, we takeDl as the selected probability of different masking lengths; that is, in the MLM, if the number of samples with masking length of l is Ml and the value range of l is also set to min to max, then the masked probability of each masking length is Pl, which we set as follows:
(10)Pl=Ml.Then, we usePl as the probability to randomly select masked sequences with the length of l from the training corpus as training samples and set the total number of selected training samples to T. Then, the final number of masked samples with length l is Nl:
(11)Nl=PlT.Next, we get the probability distributions of the mask lengths for four different datasets:(12)Plss=Xlss∑l=minmaxXlss,Plsc=Xlsc∑l=minmaxXlsc,Plls=Xlls∑l=minmaxXlls,Pllc=Xllc∑l=minmaxXllc,where the Plss is the selected probability of the masked sequences with the length of l in the short span dataset, and thePlsc, Plls, and Pllcare the probabilities of the masked sequences with the length of l in the short cloze dataset, long span dataset, and long cloze dataset, respectively.In the MLM training process, first, we duplicate the input sequence 10 times and then choose different ways to mask it. We use iterative sampling to mask the sequence. In each iteration, we will randomly select the current mask length according to the above probability distribution, such asl. Then, we randomly select a sequence with l consecutive tokens from the paragraphs. This process will be cycled until the masking budget has been spent. Following BERT, the masking budget is set to 15%, which means that 15% of the text in the paragraph will be selected.Then, for each selected sequence, we also replace it with a proportion of 80%-10%-10%. As shown in Figure3, in the following span masking scheme in SpanBERT [9], we perform this replacement at the sequence level, rather than separately for each token; i.e., each selected sequence has an 80% probability of being replaced with “mask” and 10% probability of being replaced with random tokens, and 10% remains the same.Finally, the actual number of masked samples of the masked sequence of lengthl is as follows:
(13)Real_Nl=80%×Nl.The number of masked samples which are randomly replaced with other texts is as follows:(14)Random_Nl=10%×Nl.The number of masked samples whose text in the masked part remains unchanged is as follows:(15)Remain_Nl=10%×Nl.We use dynamic masking [14] to avoid masking the same sequences for each paragraph in every epoch. Following RoBERTa, we duplicate the input sequence 10 times so that each sequence is masked in 10 different ways.


### 7.2. Input Sequences

Before feeding the training data into the model, we need to preprocess the data. A preprocessed input sample is a sequence composed of both the question and the reference context. A separation token (denoted asSEP) is used to separate the question and context. It will be added between the question and the context, as well as the end of the context. In addition to SEP, there are 4 special tokens in the input sequence:CLS. This is used to identify the beginning of the sequence. In tasks such as classification, it is usually necessary to use the output of the CLS position in the last layer.UNK. The out-of-vocabulary (OOV) words will be replaced by this token.PAD. In the zero-padding mask, for sentences shorter than the maximum length, we will have to fill PAD to make up for the length.MASK. In some training objectives, such as the masked language model (MLM), some input tokens are randomly replaced with the MASK token (being masked), and the model is required to predict the masked tokens.After that, the question-and-context pair is tokenized. A commonly used tokenization method is the BERT tokenizer. In this baseline, we use “–vocab_path” to specify the Chinese vocabulary path. Then, we use this vocabulary to tokenize the question-and-context pair. Finally, each token is converted into a unique index according to the index of the corresponding Chinese character in the vocabulary.


### 7.3. Tokenization

We use the WordPiece tokenizer. The WordPiece tokenizer follows the subword tokenization scheme. The tokenizer first checks whether the word is in the vocabulary. If so, then it will be used as a token. If the word is not in the vocabulary, then the word will be split into subwords, and the tokenizer will constantly check that the split subword appears in the vocabulary after each split. Once a subword is found in the vocabulary, we use it as a token. The WordPiece tokenizer is very effective when dealing with out-of-vocabulary (OOV) words. Because there is no subword and no space between words in Chinese, we cannot apply the WordPiece tokenizer to the Chinese text directly. Thus, when tokenizing the Chinese text with the WordPiece tokenization, following the Chinese BERT, we add spaces around all Chinese characters, and the input Chinese text will be split into Chinese characters, so all Chinese tokens (subwords) in the vocabulary are single Chinese characters.


### 7.4. Embeddings

In the embedding layer, the input indices are transformed into corresponding vector representations, which are usually obtained by adding three distinct representations, namely, the following:Token embeddings (usually with shape (1, max length, hidden size)). Each input index is transformed into a multidimensional word embedding, which is randomly initialized from a standard normal distribution with 0 mean and unit variance.Position embeddings (usually with shape (1, max length, hidden size)). It is used to indicate the position of the token, which is a learned embedding vector. This is different from the normal transformer in BERT, which has a preset value.Finally, these embeddings are summed element-wise to produce a single vector representation and fed into the transformer encoders.


## 7.1. Masking Schemes

The key point of our mask scheme used in our experiment is that the probability distribution of different masking lengths is equal to the proportional distribution of different answer lengths in the corresponding dataset. For example, as shown in Figure3, in the short multiple-choice cloze dataset, the answer length distribution is shown in Figure 3(e). Suppose that the total number of answers with length l in the dataset is Xl, and the length of l ranges from min to max. Then, we can calculate the actual distribution ratio of the length of each answer as shown in the following formula:
(9)Dl=Xl∑l=minmaxXl.Among them,Dl is the distribution ratio of the number of answers with length l to the total number of answers.Then, we treat the distribution ratios as the probabilities and use them as the probabilities of different lengths being selected in the MLM. The pie chart of this probability distribution is shown in Figure3(a).When training the MLM for the dataset, we takeDl as the selected probability of different masking lengths; that is, in the MLM, if the number of samples with masking length of l is Ml and the value range of l is also set to min to max, then the masked probability of each masking length is Pl, which we set as follows:
(10)Pl=Ml.Then, we usePl as the probability to randomly select masked sequences with the length of l from the training corpus as training samples and set the total number of selected training samples to T. Then, the final number of masked samples with length l is Nl:
(11)Nl=PlT.Next, we get the probability distributions of the mask lengths for four different datasets:(12)Plss=Xlss∑l=minmaxXlss,Plsc=Xlsc∑l=minmaxXlsc,Plls=Xlls∑l=minmaxXlls,Pllc=Xllc∑l=minmaxXllc,where the Plss is the selected probability of the masked sequences with the length of l in the short span dataset, and thePlsc, Plls, and Pllcare the probabilities of the masked sequences with the length of l in the short cloze dataset, long span dataset, and long cloze dataset, respectively.In the MLM training process, first, we duplicate the input sequence 10 times and then choose different ways to mask it. We use iterative sampling to mask the sequence. In each iteration, we will randomly select the current mask length according to the above probability distribution, such asl. Then, we randomly select a sequence with l consecutive tokens from the paragraphs. This process will be cycled until the masking budget has been spent. Following BERT, the masking budget is set to 15%, which means that 15% of the text in the paragraph will be selected.Then, for each selected sequence, we also replace it with a proportion of 80%-10%-10%. As shown in Figure3, in the following span masking scheme in SpanBERT [9], we perform this replacement at the sequence level, rather than separately for each token; i.e., each selected sequence has an 80% probability of being replaced with “mask” and 10% probability of being replaced with random tokens, and 10% remains the same.Finally, the actual number of masked samples of the masked sequence of lengthl is as follows:
(13)Real_Nl=80%×Nl.The number of masked samples which are randomly replaced with other texts is as follows:(14)Random_Nl=10%×Nl.The number of masked samples whose text in the masked part remains unchanged is as follows:(15)Remain_Nl=10%×Nl.We use dynamic masking [14] to avoid masking the same sequences for each paragraph in every epoch. Following RoBERTa, we duplicate the input sequence 10 times so that each sequence is masked in 10 different ways.


## 7.2. Input Sequences

Before feeding the training data into the model, we need to preprocess the data. A preprocessed input sample is a sequence composed of both the question and the reference context. A separation token (denoted asSEP) is used to separate the question and context. It will be added between the question and the context, as well as the end of the context. In addition to SEP, there are 4 special tokens in the input sequence:CLS. This is used to identify the beginning of the sequence. In tasks such as classification, it is usually necessary to use the output of the CLS position in the last layer.UNK. The out-of-vocabulary (OOV) words will be replaced by this token.PAD. In the zero-padding mask, for sentences shorter than the maximum length, we will have to fill PAD to make up for the length.MASK. In some training objectives, such as the masked language model (MLM), some input tokens are randomly replaced with the MASK token (being masked), and the model is required to predict the masked tokens.After that, the question-and-context pair is tokenized. A commonly used tokenization method is the BERT tokenizer. In this baseline, we use “–vocab_path” to specify the Chinese vocabulary path. Then, we use this vocabulary to tokenize the question-and-context pair. Finally, each token is converted into a unique index according to the index of the corresponding Chinese character in the vocabulary.


## 7.3. Tokenization

We use the WordPiece tokenizer. The WordPiece tokenizer follows the subword tokenization scheme. The tokenizer first checks whether the word is in the vocabulary. If so, then it will be used as a token. If the word is not in the vocabulary, then the word will be split into subwords, and the tokenizer will constantly check that the split subword appears in the vocabulary after each split. Once a subword is found in the vocabulary, we use it as a token. The WordPiece tokenizer is very effective when dealing with out-of-vocabulary (OOV) words. Because there is no subword and no space between words in Chinese, we cannot apply the WordPiece tokenizer to the Chinese text directly. Thus, when tokenizing the Chinese text with the WordPiece tokenization, following the Chinese BERT, we add spaces around all Chinese characters, and the input Chinese text will be split into Chinese characters, so all Chinese tokens (subwords) in the vocabulary are single Chinese characters.


## 7.4. Embeddings

In the embedding layer, the input indices are transformed into corresponding vector representations, which are usually obtained by adding three distinct representations, namely, the following:Token embeddings (usually with shape (1, max length, hidden size)). Each input index is transformed into a multidimensional word embedding, which is randomly initialized from a standard normal distribution with 0 mean and unit variance.Position embeddings (usually with shape (1, max length, hidden size)). It is used to indicate the position of the token, which is a learned embedding vector. This is different from the normal transformer in BERT, which has a preset value.Finally, these embeddings are summed element-wise to produce a single vector representation and fed into the transformer encoders.


## 8. Experiments

### 8.1. Pretraining Setup

Using the open-source framework of UER-pyUER‐py, we pretrain MLMs with different lengths on the Chinese corpus. Compared with the original BERT implementation, the main points in our implementation include the following:
(a)
The probability distribution of different masking lengths is equal to the proportional distribution of different answer lengths in the corresponding dataset(b)
We did not use the next sentence prediction (NSP) training objective but only the masked language model (MLM)(c)
We perform the mask replacement at the sequence level instead of performing this replacement separately for each token(d)
We use dynamic masking instead of static maskingOur implementation of MLM is trained on the cloud with Nvidia V100 GPU. We use a sequence of up to 512 tokens for pretraining. The learning rate is set to5e‐5, and the batch size is 64. The parameters of the Adam optimizer are fixed at β1=0.8 and β2=0.9. We use mixed precision training to reduce memory usage and accelerate pretraining.


### 8.2. Training Corpus

The effective pretraining of the MLM crucially relies on large-scale training data from various domains. Improving the diversity of data domains and increasing the amount of data can result in improved performance in downstream tasks [9, 30].In this work, we collect the Chinese training data from the Internet and then cleaned the data. For example, we clean the HTML mark, remove extra empty characters, and remove the picture mark. Finally, we collect a Chinese corpus for our MLM pretraining. The corpus contains Chinese texts in the following domains:(1)
Wikipedia that contains various Chinese Wikipedia documents(2)
Chinese academic papers on the WanFang database(3)
Chinese social text messages on Weibo(4)
Chinese articles on the WeChat official accounts(5)
Chinese news articles, including titles, keywords, descriptions, and texts. Categories include finance, real estate, stocks, home furnishing, education, technology, society, fashion, current affairs, sports, constellations, games, and entertainment


### 8.3. Fine-Tune and Evaluation

We fine-tune and evaluate our pretrained masked language models on four machine reading comprehension datasets, namely, the short span extraction dataset, long span extraction dataset, short multiple-choice cloze dataset, and long multiple-choice cloze dataset, and the details are given below.


#### 8.3.1. Span Extraction

The span extraction reading comprehension task is composed of passages, questions, and answers. This task requires the computers to answer relevant questions according to the passages. The answer to the question can be found in the passage; that is, the answer is a span (fragment) in the passage.This task can be simplified to predict a starting position and an ending position in a passage, and the answer is a text span between the start and the end.The process of using MLM to deal with span extraction reading comprehension tasks can be divided into three layers: input layer, transformer-based encoder layer, and output layer.(1) Input Layer. In the input layer, we preprocess the input passages and questions; first, we perform word piece tokenization, then splice the questions and passages; we insert CLS at the beginning of the input sequence and SEP at the end and at the dividing point between the question and the passage so as to finally get the input sequence.
(16)S=CLS,q1,⋯,qmSEP,p1,⋯,pn,SEP.It should be noted that if the lengthn of the input text is less than the maximum sequence length N, the padding token PAD needs to be spliced after the input sequence until it reaches the maximum sequence length n. In the following example, assume that the maximum sequence length of our model is n=10 and the current input sequence length is 7. Then, three padding tokens PAD are required after the input sequence.
(17)S=CLS,q1,⋯,qmSEP,p1,⋯,pn,SEPPADPADPAD.Conversely, if the length of the current input sequence is longer than the maximum sequence lengthN, the sequence needs to be sliced and divided into multiple subsequences. For example, assume that the maximum sequence length of our model is n=10 and the current input sequence length is 40. Then, the model can only process input sequences with a length of 10 tokens at one time and the sequence needs to be divided into 4 subsequences.In addition, it should be noted that we have to put the question at the beginning of the input sequence. Because if the question is divided into multiple subsequences, the question cannot be answered. If the passage is divided into multiple subsequences, the answers in the passage can be obtained through other sequences.(2) Encoder Layer Based on the Transformers. The input sequence will be converted into token embeddings, position embeddings, and segment embeddings. These three embeddings will be added to obtain the input vector. The input vector will pass through 12 encoding layers. In these encoding layers, with the help of a multihead self-attention mechanism, the model will fully learn the semantic association between passages and questions.(3) Output Layer. The output of the last layer of the transformer encoder passes through a full connection layer and predicts the probability PS of each position as the answer and the probability PE of the end position through Softmax.Then, we input the prediction probabilities and the ground truth positions into the cross-entropy loss function at the same time to obtain the loss of the model. Finally, the cross-entropy loss at the starting position and the loss at the ending position are averaged to obtain the final total cross-entropy loss of the model. The training objective is to minimize the total cross-entropy loss between the prediction probability and the ground truth position.(4) Answer Prediction and Evaluation. In the output layer, we select the starting position and ending position with the highest probability as the prediction answer. Finally, F1 and EM of the predicted answer are calculated according to the standard answer.


#### 8.3.2. Multiple-Choice Cloze

Following the method in CMRC [25], in the sentence cloze-style reading comprehension task, we select several sentences in the passages and replace them with special marks (for example, BLANK) to form an incomplete passage. The selected sentences will form the candidate list, and the computer is required to fill in the blanks with the right candidate sentences.(1) Input Sequence. The input sequence is composed of an answer option and the passage (with blanks), and then the semantic representation of the context is obtained through transformer encoder layers. Finally, the probability of each blank corresponding to an option is output. It should be noted that the two components of the input sequence are an answer option and an incomplete passage with multiple blanks. Because there are 9 blanks (corresponding to 9 different options) in each passage in our dataset, therefore, we need to enter 9 different sequences, and each sequence contains an option.For example, assume that the current answer options are as follows:a1,⋯,ai,⋯,am, where ai represents the i-th word in the answer option text. Let the input paragraph with blanks be p1,⋯,blank1,⋯,blanki,⋯,pn, where pn represents the n-th word of the input passage and blanki represents the i-th masked answer. The input sequence S can be expressed as follows:
(18)S=CLS,a1,⋯,am,SEP,p1,⋯,blank1,⋯,blanki,⋯,pn,SEP,where CLS represents the special token at the beginning of the input sequence and SEP represents the segmentation token and the end token of the input sequence (following BERT).It should be noted that if the lengthn of the input sequence is less than the maximum sequence length N, the padding token PAD needs to be spliced after the input sequence until it reaches the maximum sequence length N. In the following example, assume that the maximum sequence length is n=10 and the input sequence length is 9. Then, one padding token PAD is required after the input sequence.
(19)S=CLS,a1,⋯,am,SEP,p1,⋯,blank1,⋯,blanki,⋯,pn,SEP,PAD.Conversely, if the input sequence lengthn is longer than the maximum length N, it needs to be truncated into multiple input sequences. Here, we usually put the answer option in the front so that the answer options will not be truncated.(2) Embeddings. This section describes how to preprocess the input sequence to get the corresponding input representation. The input representation is composed of the sum of the token embedding, segment embedding, and position embedding. For example, assume that these three embeddings are Etoken, Esegment, and Eposition, respectively, and the input representation E corresponding to the input sequence can be calculated by the following formula:
(20)E=Etoken+Esegment+Eposition.In the formula,Etoken represents the token embedding, Esegment represents the segment embedding, and Eposition represents the position embedding; the size of the three embeddings isM∗d, whereMrepresents the maximum length of the sequence, which is 512 in this paper, and d represents the dimension of the word vector, which is 768 in this article.(3) Transformer Encoders. In transformer encoders, the input embeddings pass through 12 encoder layers and use the self-attention mechanism to fully learn the semantic representation between each word in the input sequence.
(21)Ei=Encoder_layerEi−1,where Ei−1 indicates the output vector of the i-th encoder layer, and E0 is specified to be equal to the input embedding. Finally, after 12 encoder layers, the output vector of the encoder is as follows:
(22)Eo=Encoder_layerE11,where Eo indicates the output vector of the last encoder layer.(4) The Pooled Layer. In this layer, the output of the encoder layer is fed into a pooled layer to get pooled output.
(23)Pooled_output=Eo∗W+b,where W denotes the weight matrix of the pooled layer and brepresents the bias vector.(5) The Output Layer. In the final output result, we do not need the output of each token in the input sequence, but we only need the output sequence where the current blank is located. Therefore, for the pooled outputs of the other positions except for blanks, we will remove them from the total output and then splice the remaining pooled outputs of these blank positions to obtain the output O. Among them, for the output Oi denoting the i-th blank, we use the Softmax function to calculate the confidence probability that the current blank position matches the current option.
(24)Pi=SoftmaxOi=eOi∑jeOj.Finally, after obtaining the prediction probabilityPi corresponding to the class label of the current sequence, the cross-entropy loss between the correct answer Tiand the prediction probability Pi is calculated.
(25)Loss=CrossEntropyLossPi,Ti.The training objective is to minimize the total cross-entropy loss between the prediction probability and the standard answer sequence.(6) Answer Prediction and Evaluation. When predicting the answer, we choose the blank with the highest probability as the position where the answer option I should be filled. Finally, according to the standard answer, the PAC and QAC of the predicted answer are calculated.


## 8.1. Pretraining Setup

Using the open-source framework of UER-pyUER‐py, we pretrain MLMs with different lengths on the Chinese corpus. Compared with the original BERT implementation, the main points in our implementation include the following:
(a)
The probability distribution of different masking lengths is equal to the proportional distribution of different answer lengths in the corresponding dataset(b)
We did not use the next sentence prediction (NSP) training objective but only the masked language model (MLM)(c)
We perform the mask replacement at the sequence level instead of performing this replacement separately for each token(d)
We use dynamic masking instead of static maskingOur implementation of MLM is trained on the cloud with Nvidia V100 GPU. We use a sequence of up to 512 tokens for pretraining. The learning rate is set to5e‐5, and the batch size is 64. The parameters of the Adam optimizer are fixed at β1=0.8 and β2=0.9. We use mixed precision training to reduce memory usage and accelerate pretraining.


## 8.2. Training Corpus

The effective pretraining of the MLM crucially relies on large-scale training data from various domains. Improving the diversity of data domains and increasing the amount of data can result in improved performance in downstream tasks [9, 30].In this work, we collect the Chinese training data from the Internet and then cleaned the data. For example, we clean the HTML mark, remove extra empty characters, and remove the picture mark. Finally, we collect a Chinese corpus for our MLM pretraining. The corpus contains Chinese texts in the following domains:(1)
Wikipedia that contains various Chinese Wikipedia documents(2)
Chinese academic papers on the WanFang database(3)
Chinese social text messages on Weibo(4)
Chinese articles on the WeChat official accounts(5)
Chinese news articles, including titles, keywords, descriptions, and texts. Categories include finance, real estate, stocks, home furnishing, education, technology, society, fashion, current affairs, sports, constellations, games, and entertainment


## 8.3. Fine-Tune and Evaluation

We fine-tune and evaluate our pretrained masked language models on four machine reading comprehension datasets, namely, the short span extraction dataset, long span extraction dataset, short multiple-choice cloze dataset, and long multiple-choice cloze dataset, and the details are given below.


### 8.3.1. Span Extraction

The span extraction reading comprehension task is composed of passages, questions, and answers. This task requires the computers to answer relevant questions according to the passages. The answer to the question can be found in the passage; that is, the answer is a span (fragment) in the passage.This task can be simplified to predict a starting position and an ending position in a passage, and the answer is a text span between the start and the end.The process of using MLM to deal with span extraction reading comprehension tasks can be divided into three layers: input layer, transformer-based encoder layer, and output layer.(1) Input Layer. In the input layer, we preprocess the input passages and questions; first, we perform word piece tokenization, then splice the questions and passages; we insert CLS at the beginning of the input sequence and SEP at the end and at the dividing point between the question and the passage so as to finally get the input sequence.
(16)S=CLS,q1,⋯,qmSEP,p1,⋯,pn,SEP.It should be noted that if the lengthn of the input text is less than the maximum sequence length N, the padding token PAD needs to be spliced after the input sequence until it reaches the maximum sequence length n. In the following example, assume that the maximum sequence length of our model is n=10 and the current input sequence length is 7. Then, three padding tokens PAD are required after the input sequence.
(17)S=CLS,q1,⋯,qmSEP,p1,⋯,pn,SEPPADPADPAD.Conversely, if the length of the current input sequence is longer than the maximum sequence lengthN, the sequence needs to be sliced and divided into multiple subsequences. For example, assume that the maximum sequence length of our model is n=10 and the current input sequence length is 40. Then, the model can only process input sequences with a length of 10 tokens at one time and the sequence needs to be divided into 4 subsequences.In addition, it should be noted that we have to put the question at the beginning of the input sequence. Because if the question is divided into multiple subsequences, the question cannot be answered. If the passage is divided into multiple subsequences, the answers in the passage can be obtained through other sequences.(2) Encoder Layer Based on the Transformers. The input sequence will be converted into token embeddings, position embeddings, and segment embeddings. These three embeddings will be added to obtain the input vector. The input vector will pass through 12 encoding layers. In these encoding layers, with the help of a multihead self-attention mechanism, the model will fully learn the semantic association between passages and questions.(3) Output Layer. The output of the last layer of the transformer encoder passes through a full connection layer and predicts the probability PS of each position as the answer and the probability PE of the end position through Softmax.Then, we input the prediction probabilities and the ground truth positions into the cross-entropy loss function at the same time to obtain the loss of the model. Finally, the cross-entropy loss at the starting position and the loss at the ending position are averaged to obtain the final total cross-entropy loss of the model. The training objective is to minimize the total cross-entropy loss between the prediction probability and the ground truth position.(4) Answer Prediction and Evaluation. In the output layer, we select the starting position and ending position with the highest probability as the prediction answer. Finally, F1 and EM of the predicted answer are calculated according to the standard answer.


### 8.3.2. Multiple-Choice Cloze

Following the method in CMRC [25], in the sentence cloze-style reading comprehension task, we select several sentences in the passages and replace them with special marks (for example, BLANK) to form an incomplete passage. The selected sentences will form the candidate list, and the computer is required to fill in the blanks with the right candidate sentences.(1) Input Sequence. The input sequence is composed of an answer option and the passage (with blanks), and then the semantic representation of the context is obtained through transformer encoder layers. Finally, the probability of each blank corresponding to an option is output. It should be noted that the two components of the input sequence are an answer option and an incomplete passage with multiple blanks. Because there are 9 blanks (corresponding to 9 different options) in each passage in our dataset, therefore, we need to enter 9 different sequences, and each sequence contains an option.For example, assume that the current answer options are as follows:a1,⋯,ai,⋯,am, where ai represents the i-th word in the answer option text. Let the input paragraph with blanks be p1,⋯,blank1,⋯,blanki,⋯,pn, where pn represents the n-th word of the input passage and blanki represents the i-th masked answer. The input sequence S can be expressed as follows:
(18)S=CLS,a1,⋯,am,SEP,p1,⋯,blank1,⋯,blanki,⋯,pn,SEP,where CLS represents the special token at the beginning of the input sequence and SEP represents the segmentation token and the end token of the input sequence (following BERT).It should be noted that if the lengthn of the input sequence is less than the maximum sequence length N, the padding token PAD needs to be spliced after the input sequence until it reaches the maximum sequence length N. In the following example, assume that the maximum sequence length is n=10 and the input sequence length is 9. Then, one padding token PAD is required after the input sequence.
(19)S=CLS,a1,⋯,am,SEP,p1,⋯,blank1,⋯,blanki,⋯,pn,SEP,PAD.Conversely, if the input sequence lengthn is longer than the maximum length N, it needs to be truncated into multiple input sequences. Here, we usually put the answer option in the front so that the answer options will not be truncated.(2) Embeddings. This section describes how to preprocess the input sequence to get the corresponding input representation. The input representation is composed of the sum of the token embedding, segment embedding, and position embedding. For example, assume that these three embeddings are Etoken, Esegment, and Eposition, respectively, and the input representation E corresponding to the input sequence can be calculated by the following formula:
(20)E=Etoken+Esegment+Eposition.In the formula,Etoken represents the token embedding, Esegment represents the segment embedding, and Eposition represents the position embedding; the size of the three embeddings isM∗d, whereMrepresents the maximum length of the sequence, which is 512 in this paper, and d represents the dimension of the word vector, which is 768 in this article.(3) Transformer Encoders. In transformer encoders, the input embeddings pass through 12 encoder layers and use the self-attention mechanism to fully learn the semantic representation between each word in the input sequence.
(21)Ei=Encoder_layerEi−1,where Ei−1 indicates the output vector of the i-th encoder layer, and E0 is specified to be equal to the input embedding. Finally, after 12 encoder layers, the output vector of the encoder is as follows:
(22)Eo=Encoder_layerE11,where Eo indicates the output vector of the last encoder layer.(4) The Pooled Layer. In this layer, the output of the encoder layer is fed into a pooled layer to get pooled output.
(23)Pooled_output=Eo∗W+b,where W denotes the weight matrix of the pooled layer and brepresents the bias vector.(5) The Output Layer. In the final output result, we do not need the output of each token in the input sequence, but we only need the output sequence where the current blank is located. Therefore, for the pooled outputs of the other positions except for blanks, we will remove them from the total output and then splice the remaining pooled outputs of these blank positions to obtain the output O. Among them, for the output Oi denoting the i-th blank, we use the Softmax function to calculate the confidence probability that the current blank position matches the current option.
(24)Pi=SoftmaxOi=eOi∑jeOj.Finally, after obtaining the prediction probabilityPi corresponding to the class label of the current sequence, the cross-entropy loss between the correct answer Tiand the prediction probability Pi is calculated.
(25)Loss=CrossEntropyLossPi,Ti.The training objective is to minimize the total cross-entropy loss between the prediction probability and the standard answer sequence.(6) Answer Prediction and Evaluation. When predicting the answer, we choose the blank with the highest probability as the position where the answer option I should be filled. Finally, according to the standard answer, the PAC and QAC of the predicted answer are calculated.


## 8.3.1. Span Extraction

The span extraction reading comprehension task is composed of passages, questions, and answers. This task requires the computers to answer relevant questions according to the passages. The answer to the question can be found in the passage; that is, the answer is a span (fragment) in the passage.This task can be simplified to predict a starting position and an ending position in a passage, and the answer is a text span between the start and the end.The process of using MLM to deal with span extraction reading comprehension tasks can be divided into three layers: input layer, transformer-based encoder layer, and output layer.(1) Input Layer. In the input layer, we preprocess the input passages and questions; first, we perform word piece tokenization, then splice the questions and passages; we insert CLS at the beginning of the input sequence and SEP at the end and at the dividing point between the question and the passage so as to finally get the input sequence.
(16)S=CLS,q1,⋯,qmSEP,p1,⋯,pn,SEP.It should be noted that if the lengthn of the input text is less than the maximum sequence length N, the padding token PAD needs to be spliced after the input sequence until it reaches the maximum sequence length n. In the following example, assume that the maximum sequence length of our model is n=10 and the current input sequence length is 7. Then, three padding tokens PAD are required after the input sequence.
(17)S=CLS,q1,⋯,qmSEP,p1,⋯,pn,SEPPADPADPAD.Conversely, if the length of the current input sequence is longer than the maximum sequence lengthN, the sequence needs to be sliced and divided into multiple subsequences. For example, assume that the maximum sequence length of our model is n=10 and the current input sequence length is 40. Then, the model can only process input sequences with a length of 10 tokens at one time and the sequence needs to be divided into 4 subsequences.In addition, it should be noted that we have to put the question at the beginning of the input sequence. Because if the question is divided into multiple subsequences, the question cannot be answered. If the passage is divided into multiple subsequences, the answers in the passage can be obtained through other sequences.(2) Encoder Layer Based on the Transformers. The input sequence will be converted into token embeddings, position embeddings, and segment embeddings. These three embeddings will be added to obtain the input vector. The input vector will pass through 12 encoding layers. In these encoding layers, with the help of a multihead self-attention mechanism, the model will fully learn the semantic association between passages and questions.(3) Output Layer. The output of the last layer of the transformer encoder passes through a full connection layer and predicts the probability PS of each position as the answer and the probability PE of the end position through Softmax.Then, we input the prediction probabilities and the ground truth positions into the cross-entropy loss function at the same time to obtain the loss of the model. Finally, the cross-entropy loss at the starting position and the loss at the ending position are averaged to obtain the final total cross-entropy loss of the model. The training objective is to minimize the total cross-entropy loss between the prediction probability and the ground truth position.(4) Answer Prediction and Evaluation. In the output layer, we select the starting position and ending position with the highest probability as the prediction answer. Finally, F1 and EM of the predicted answer are calculated according to the standard answer.


## 8.3.2. Multiple-Choice Cloze

Following the method in CMRC [25], in the sentence cloze-style reading comprehension task, we select several sentences in the passages and replace them with special marks (for example, BLANK) to form an incomplete passage. The selected sentences will form the candidate list, and the computer is required to fill in the blanks with the right candidate sentences.(1) Input Sequence. The input sequence is composed of an answer option and the passage (with blanks), and then the semantic representation of the context is obtained through transformer encoder layers. Finally, the probability of each blank corresponding to an option is output. It should be noted that the two components of the input sequence are an answer option and an incomplete passage with multiple blanks. Because there are 9 blanks (corresponding to 9 different options) in each passage in our dataset, therefore, we need to enter 9 different sequences, and each sequence contains an option.For example, assume that the current answer options are as follows:a1,⋯,ai,⋯,am, where ai represents the i-th word in the answer option text. Let the input paragraph with blanks be p1,⋯,blank1,⋯,blanki,⋯,pn, where pn represents the n-th word of the input passage and blanki represents the i-th masked answer. The input sequence S can be expressed as follows:
(18)S=CLS,a1,⋯,am,SEP,p1,⋯,blank1,⋯,blanki,⋯,pn,SEP,where CLS represents the special token at the beginning of the input sequence and SEP represents the segmentation token and the end token of the input sequence (following BERT).It should be noted that if the lengthn of the input sequence is less than the maximum sequence length N, the padding token PAD needs to be spliced after the input sequence until it reaches the maximum sequence length N. In the following example, assume that the maximum sequence length is n=10 and the input sequence length is 9. Then, one padding token PAD is required after the input sequence.
(19)S=CLS,a1,⋯,am,SEP,p1,⋯,blank1,⋯,blanki,⋯,pn,SEP,PAD.Conversely, if the input sequence lengthn is longer than the maximum length N, it needs to be truncated into multiple input sequences. Here, we usually put the answer option in the front so that the answer options will not be truncated.(2) Embeddings. This section describes how to preprocess the input sequence to get the corresponding input representation. The input representation is composed of the sum of the token embedding, segment embedding, and position embedding. For example, assume that these three embeddings are Etoken, Esegment, and Eposition, respectively, and the input representation E corresponding to the input sequence can be calculated by the following formula:
(20)E=Etoken+Esegment+Eposition.In the formula,Etoken represents the token embedding, Esegment represents the segment embedding, and Eposition represents the position embedding; the size of the three embeddings isM∗d, whereMrepresents the maximum length of the sequence, which is 512 in this paper, and d represents the dimension of the word vector, which is 768 in this article.(3) Transformer Encoders. In transformer encoders, the input embeddings pass through 12 encoder layers and use the self-attention mechanism to fully learn the semantic representation between each word in the input sequence.
(21)Ei=Encoder_layerEi−1,where Ei−1 indicates the output vector of the i-th encoder layer, and E0 is specified to be equal to the input embedding. Finally, after 12 encoder layers, the output vector of the encoder is as follows:
(22)Eo=Encoder_layerE11,where Eo indicates the output vector of the last encoder layer.(4) The Pooled Layer. In this layer, the output of the encoder layer is fed into a pooled layer to get pooled output.
(23)Pooled_output=Eo∗W+b,where W denotes the weight matrix of the pooled layer and brepresents the bias vector.(5) The Output Layer. In the final output result, we do not need the output of each token in the input sequence, but we only need the output sequence where the current blank is located. Therefore, for the pooled outputs of the other positions except for blanks, we will remove them from the total output and then splice the remaining pooled outputs of these blank positions to obtain the output O. Among them, for the output Oi denoting the i-th blank, we use the Softmax function to calculate the confidence probability that the current blank position matches the current option.
(24)Pi=SoftmaxOi=eOi∑jeOj.Finally, after obtaining the prediction probabilityPi corresponding to the class label of the current sequence, the cross-entropy loss between the correct answer Tiand the prediction probability Pi is calculated.
(25)Loss=CrossEntropyLossPi,Ti.The training objective is to minimize the total cross-entropy loss between the prediction probability and the standard answer sequence.(6) Answer Prediction and Evaluation. When predicting the answer, we choose the blank with the highest probability as the position where the answer option I should be filled. Finally, according to the standard answer, the PAC and QAC of the predicted answer are calculated.


## 9. Result Analysis

### 9.1. Human Performance

In order to evaluate the human performance on our datasets, we invited 10 college students to answer questions in the datasets manually. Finally, we got the answers in the test sets of the four datasets, respectively. Then, we calculated F1 and EM to roughly evaluate the human performance on the proposed long span dataset and short span dataset, and we also calculated PAC and QAC on the proposed cloze datasets.


### 9.2. Model Performance

The evaluation results on the pretrained models on different MRC datasets are presented in Table10. For fair comparison, these models are all fine-tuned with the same hyperparameters and without any data augmentation. We fine-tuned three different runs and report the mean results. The pretrained language models based on the corresponding MLMs constantly outperform other pretrained language models on the corresponding datasets by an obvious margin.Table 10
The evaluation results of pretrained models and human performance on span datasets.
ModelsShort span datasetLong span datasetF1EMF1EMHuman performance93.5685.3490.4782.56PLM with short span MLM25.0323.9313.0910.79PLM with long span MLM24.7423.3113.2811.06As shown in Table10, the pretrained language model based on the long span MLMs performs better on the long span dataset compared to the other three pretrained language models, though there still exists a large gap between this model and human performance. At the same time, the pretrained language model based on the short span MLMs performs better on the short span dataset compared to the other pretrained language models.As shown in Table11, pretrained with long cloze MLMs, the long pretrained model outperforms other models on the long cloze dataset. As for the short cloze dataset, the pretrained language model based on the short cloze MLMs achieves a score increase over other models, demonstrating the effectiveness of the proposed MLMs.Table 11
The evaluation results of pretrained models and human performance on cloze datasets.
ModelsShort cloze datasetLong cloze datasetQACPACQACPACHuman performance97.4592.8795.2490.71PLM with short cloze MLM64.6317.1065.8217.60PLM with long cloze MLM63.2815.2067.0218.60In summary, the experimental results demonstrate that our hypothesis is true. The length of MLM is indeed related to their performance in MRC tasks with different answer lengths. It can guide us on how to pretrain an MLM with a relatively suitable mask length distribution for various MRC tasks.


## 9.1. Human Performance

In order to evaluate the human performance on our datasets, we invited 10 college students to answer questions in the datasets manually. Finally, we got the answers in the test sets of the four datasets, respectively. Then, we calculated F1 and EM to roughly evaluate the human performance on the proposed long span dataset and short span dataset, and we also calculated PAC and QAC on the proposed cloze datasets.


## 9.2. Model Performance

The evaluation results on the pretrained models on different MRC datasets are presented in Table10. For fair comparison, these models are all fine-tuned with the same hyperparameters and without any data augmentation. We fine-tuned three different runs and report the mean results. The pretrained language models based on the corresponding MLMs constantly outperform other pretrained language models on the corresponding datasets by an obvious margin.Table 10
The evaluation results of pretrained models and human performance on span datasets.
ModelsShort span datasetLong span datasetF1EMF1EMHuman performance93.5685.3490.4782.56PLM with short span MLM25.0323.9313.0910.79PLM with long span MLM24.7423.3113.2811.06As shown in Table10, the pretrained language model based on the long span MLMs performs better on the long span dataset compared to the other three pretrained language models, though there still exists a large gap between this model and human performance. At the same time, the pretrained language model based on the short span MLMs performs better on the short span dataset compared to the other pretrained language models.As shown in Table11, pretrained with long cloze MLMs, the long pretrained model outperforms other models on the long cloze dataset. As for the short cloze dataset, the pretrained language model based on the short cloze MLMs achieves a score increase over other models, demonstrating the effectiveness of the proposed MLMs.Table 11
The evaluation results of pretrained models and human performance on cloze datasets.
ModelsShort cloze datasetLong cloze datasetQACPACQACPACHuman performance97.4592.8795.2490.71PLM with short cloze MLM64.6317.1065.8217.60PLM with long cloze MLM63.2815.2067.0218.60In summary, the experimental results demonstrate that our hypothesis is true. The length of MLM is indeed related to their performance in MRC tasks with different answer lengths. It can guide us on how to pretrain an MLM with a relatively suitable mask length distribution for various MRC tasks.


## 10. Conclusions

In this paper, we propose an evaluation framework to quantitatively verify whether masking schemes of different lengths will affect the results of the MLM in MRC tasks with different answer lengths. In order to address this issue, herein, (1) we propose four MRC tasks with different answer length distributions, namely, the short span extraction task, long span extraction task, short multiple-choice cloze task, and long multiple-choice cloze task; (2) four Chinese MRC datasets are created for these tasks; (3) we also have pretrained four masked language models according to the answer length distributions of these datasets; and (4) ablation experiments are conducted on the datasets to verify our hypothesis. The experimental results demonstrate that our hypothesis is true. On four different machine reading comprehension datasets, the performance of the model with correlation length distribution surpasses the model without correlation. It can guide us on how to pretrain an MLM with a relatively suitable mask length distribution for various MRC tasks. However, as a case study, we must also be conservative in the strength of our conclusions since more comprehensive future research and experiments are needed.



---

*Source: 5375334-2021-11-23.xml*