SlowGuess/NeurIPS_2023_OCR huggingface dataset:
From Cloze to Comprehension: Retrofitting Pre-trained Masked Language Models to Pre-trained Machine Reader*
Weiwen Xu $^{12\dagger}$  Xin Li $^{2\dagger}$  Wenxuan Zhang $^{2}$  Meng Zhou $^{3\dagger}$
Wai Lam<sup>1</sup> Luo Si<sup>2</sup> Lidong Bing<sup>2</sup>
<sup>1</sup>The Chinese University of Hong Kong
$^{2}$ DAMO Academy, Alibaba Group
$^{3}$ Carnegie Mellon University
{wxxu,w1am}@se.cuhk.edu.hk mengzhou@andrew.cmu.edu
{xinting.lx,saike.zwx,luo.si,l.bing}@alibaba-inc.com
We present Pre-trained Machine Reader (PMR), a novel method for retrofitting pre-trained masked language models (MLMs) to pre-trained machine reading comprehension (MRC) models without acquiring labeled data. PMR can resolve the discrepancy between model pre-training and downstream fine-tuning of existing MLMs. To build the proposed PMR, we constructed a large volume of general-purpose and high-quality MRC-style training data by using Wikipedia hyperlinks and designed a Wiki Anchor Extraction task to guide the MRC-style pretraining. Apart from its simplicity, PMR effectively solves extraction tasks, such as Extractive Question Answering and Named Entity Recognition. PMR shows tremendous improvements over existing approaches, especially in low-resource scenarios. When applied to the sequence classification task in the MRC formulation, PMR enables the extraction of high-quality rationales to explain the classification process, thereby providing greater prediction explainability. PMR also has the potential to serve as a unified model for tackling various extraction and classification tasks in the MRC formulation.[2]
Span extraction, such as Extractive Question Answering (EQA) and Named Entity Recognition (NER), is a sub-topic of natural language understanding (NLU) with the goal of detecting token spans from the input text according to specific requirements like task labels or questions [45, 54]. Discriminative methods were used to execute such tasks and achieved state-of-the-art performance. As shown in the left part of Figure 1, these works tailored a task-specific fine-tuning head on top of pre-trained language models (PLMs) to perform sequence tagging or machine reading comprehension (MRC) [12, 36, 27]. The base PLMs are usually selected from pre-trained masked language models (MLM), such as RoBERTa [36] or BART [29] due to their comprehensive bi-directional modeling for the input text in the encoder. However, given the disparate nature of the learning objectives and different model architectures of MLM pre-training and task-specific fine-tuning, the discriminative
*This work was supported by Alibaba Group through Alibaba Research Intern Program. The work described in this paper was also partially supported by a grant from the Research Grant Council of the Hong Kong Special Administrative Region, China (Project Code: 14200620).† This work was done when Weiwen Xu and Meng Zhou interned at Alibaba DAMO Academy.‡ Xin Li is the corresponding author.
2The code, data, and checkpoints are released at https://github.com/DAMO-NLP-SG/PMR
Figure 1: Comparison among three fine-tuning strategies for NER and EQA, namely, vanilla discriminative fine-tuning, generative fine-tuning, and fine-tuning by using the proposed PMR.
methods are less effective for adapting MLMs to downstream tasks when there is limited fine-tuning data available, leading to poor low-resource performance [6].
As shown in the middle part of Figure 1, generative fine-tuning is a popular solution to mitigate the gap between pre-training and fine-tuning [49, 50, 34]. This solution achieves remarkable few-shot performance in various span extraction tasks [10, 6, 38]. Specifically, generative methods formulate the downstream tasks as a language modeling problem in which PLMs generate response words for a given prompt (i.e., a task-specific template) as task prediction. Despite its success, tackling extraction tasks in a generative manner leads to several disadvantages. First, if it is used to generate the label token (e.g., "person" for PER entities) for a candidate span, the generative method needs to enumerate all possible span candidates to query PLMs [10]. This requirement can be computationally expensive for tasks with a long input text, such as EQA. Second, if the desired predictions are target spans (e.g., the "answer" in the EQA task), generative methods usually need to explore a large search space to generate span tokens. Moreover, it is also challenging to accurately generate structured outputs, e.g., the span-label pairs in the NER task, with PLMs originally trained on unstructured natural language texts. These limitations impede PLMs from effectively learning extraction patterns from increased volumes of training data. As a result, even instruction-tuned large language models like ChatGPT<sup>3</sup> are less effective than discriminative methods with smaller MLMs on extraction tasks [41, 43, 60, 31].
To bridge the gap between pre-training and fine-tuning without suffering from the aforementioned disadvantages, we propose a novel Pre-trained Machine Reader (PMR) as a retrofit of pre-trained MLM for more effective span extraction. As shown in the right part of Figure 1, PMR resembles common MRC models and introduces an MRC head on top of MLMs. But PMR is further enhanced by a comprehensive continual pre-training stage with large-scale MRC-style data. By maintaining the same MRC-style learning objective and model architecture as the continual pre-training during fine-tuning, PMR facilitates effective knowledge transfer in a discriminative manner and thus demonstrates great potential in both low-resource and rich-resource scenarios. Given that MRC has been proven as a universal paradigm [32, 33, 63, 23], our PMR can be directly applied to a broad range of span extraction tasks without additional task design.
To establish PMR, we constructed a large volume of general-purpose and high-quality MRC-style training data based on Wikipedia anchors (i.e., hyperlinked texts). As shown in Figure 2, for each Wikipedia anchor, we composed a pair of correlated articles. One side of the pair is the Wikipedia article that contains detailed descriptions of the hyperlinked entity, which we defined as the definition
<sup>3</sup>https://chat.openai.com
Figure 2: Construction of MRC-style data by using Wikipedia anchors.
article. The other side of the pair is the article that mentions the specific anchor text, which we defined as the mention article. We composed an MRC-style training instance in which the anchor is the answer, the surrounding passage of the anchor in the mention article is the context, and the definition of the anchor entity in the definition article is the query. Based on the above data, we then introduced a novel Wiki Anchor Extraction (WAE) problem as the pre-training task of PMR. In this task, PMR determines whether the context and the query are relevant. If so, PMR extracts the answer from the context that satisfies the query description.
We evaluated PMR on two representative span extraction tasks: NER and EQA. The results show that PMR consistently obtains better extraction performance compared with the vanilla MLM and surpasses the best baselines by large margins under almost all few-shot settings (up to 6.3 F1 on EQA and 16.3 F1 on NER). Additionally, we observe that sequence classification can be viewed as a special case of extraction tasks in our MRC formulation. In this scenario, it is surprising that PMR can identify high-quality rationale phrases from input text as the justifications for classification decisions. Furthermore, PMR has the potential to serve as a unified model for addressing various extraction and classification tasks in the MRC formulation.
In summary, our contributions are as follows. Firstly, we constructed a large volume of general-purpose and high-quality MRC-style training data to retrofit MLMs to PMRs. Secondly, by unifying pre-training and fine-tuning as the same discriminative MRC process, the proposed PMR obtains state-of-the-art results under all few-shot NER settings and three out of four few-shot EQA settings. Thirdly, with a unified MRC architecture for solving extraction and classification tasks, PMR also shows promising potential in explaining the sequence classification predictions and unifying NLU tasks.
2 PMR
This section describes PMR from the perspectives of model pre-training and downstream fine-tuning. For pre-training, we first introduce the proposed model with the training objective of WAE and then describe the curation procedure of WAE pre-training data from Wikipedia. For fine-tuning, we present how PMR can seamlessly be applied to various extraction tasks and solve them in a unified MRC paradigm.
2.1 Pre-training of PMR
PMR receives MRC-style data in the format of  $(Q, C, \{A^k\}_{k=1}^K)$ , where  $Q$  is a natural language query and  $C$  is the input context that contains the answers  $\{A^k\}_{k=1}^K$  to the query. Each answer is a consecutive token span in the context, and zero ( $K = 0$ ) or multiple ( $K > 1$ ) answers may exist.
Model Architecture. PMR has two components: an MLM encoder and an extractor (Figure 3). The encoder receives the concatenation of query  $Q$  and context  $C$  as input  $X$  and represents each input token as hidden states  $H$ .
$$
X = \left[ \left[ \mathrm {C L S} \right], Q, [ \mathrm {S E P} ], C, [ \mathrm {S E P} ] \right] \tag {1}
$$
$$
H = \mathbf {M L M} (X) \in \mathbb {R} ^ {M \times d}
$$
Figure 3: Model architecture of PMR. “-” indicates illegal candidate spans.
where [CLS] and [SEP] are special tokens inserted into the sequence,  $M$  is the sequence length, and  $d$  is the dimension of the hidden states. The encoder  $\mathbf{MLM}(\cdot)$  denotes any pre-trained text encoder for retrofitting, e.g. RoBERTa.
The extractor receives the hidden states of any two tokens and predicts the probability score that tells if the span between the two tokens should be output as an answer. We applied the general way to compute the score matrix  $S$  [39]:
$$
S = \operatorname {s i g m o i d} \left(\mathbf {F F N} (H) ^ {T} H\right) \in \mathbb {R} ^ {M \times M} \tag {2}
$$
where  $\mathbf{FFN}$  is the feed-forward network [57], and  $S_{i,j}$  is the probability to extract the span  $X_{i:j}$  as output. The general way avoids creating a large  $\mathbb{R}^{M\times M\times 2d}$ -shape tensor of the concatenation way [32], achieving higher training efficiency with fewer resources.
Training Objective. PMR is pre-trained with the WAE task, which checks the existence of answers in the context and extracts the answers if they exist. For the first goal, PMR determines whether the context contains spans that can answer the query:
$$
L _ {c l s} = \mathbf {C E} \left(S _ {1, 1}, Y ^ {c l s}\right) \tag {3}
$$
where  $\mathbf{CE}$  is the cross-entropy loss and  $S_{1,1}$  at the [CLS] token denotes the query-context relevance score. If  $Y^{cls} = 1$ , the query and the context are relevant (i.e. answers exist). This task mimics the downstream situation in which there may be no span to be extracted in the context (e.g. NER) and encourages the model to learn through the semantic relevance of two pieces of text to recognize the unextractable examples.
Secondly, the model is expected to extract all correct spans from the context as answers, which can be implemented by predicting the answer positions:
$$
L _ {e x t} = \sum_ {N <   i \leq j <   M} \mathbf {C E} \left(S _ {i, j}, Y _ {i, j} ^ {e x t}\right) \tag {4}
$$
where  $Y_{i,j}^{ext} = 1$  indicates that  $X_{i:j}$  is an answer to  $Q$ , and  $N$  is the positional offset of the context in  $X$ . Note that only  $X_{i:j}$  with  $N < i \leq j < M$  are legal answer span candidates (i.e., spans from the context). MRC-NER [32] predicted the start and end probabilities as two additional objectives. However, we find that these objectives are redundant for our matrix-based objective and incompatible with multi-span extraction.
The overall pre-training objective  $L_{wae}$  is:
$$
L _ {w a e} = L _ {c l s} + L _ {e x t} \tag {5}
$$
Data Preparation. MLM training can be easily scaled to millions of raw texts with a self-supervised learning objective [12]. In contrast, training PMR in the MRC paradigm requires labeled triplets (query, context, and answers) as supervision signals, which is expensive to prepare for large-scale pre-training. To address this limitation, we automated the construction of general-purpose and high-quality MRC-style training data by using Wikipedia anchors.
As illustrated in Figure 2, a Wikipedia anchor hyperlinks two Wikipedia articles: the definition article that provides detailed descriptions of the anchor entity "Silicon", and the mention article where the anchor is mentioned. We leveraged the large scale of such hyperlink relations in Wikipedia as the distant supervision to automatically construct the MRC triplets. Specifically, we regarded an anchor as the MRC answer for the following context and query pair. The sentences surrounding the anchor in the mention article serve as the MRC context. The sentences from the first section of the definition article, which usually composes the most representative summary for the anchor entity [7], comprise the query. The query provides a precise definition of the anchor entity, and thus serves as a good guide for PMR to extract answers (i.e., anchor text) from the context.
Concretely, we considered sentences within a window size  $W$  of the anchor as the MRC context and used the first  $T$  sentences from the definition article as the query. Note that the context may cover multiple mentions of the same anchor entity. In this case, we treated all mentions as valid answers (i.e.,  $K > 1$ ) to avoid confusing the model training. More importantly, the preceding scenario naturally resembles multi-span extraction tasks like NER. To prevent information leakage, we anonymized the anchor entity in the query by using "it" to substitute text spans that overlapped more than  $50\%$  with the anchor entity name. We did not use the "[MASK]" token because it does not exist in the data of downstream tasks.
In addition to the above answerable query and context pairs prepared through hyperlink relation, we introduced unanswerable examples by pairing a context with an irrelevant query (i.e., query and context pairs without the hyperlink association). The unanswerable examples are designed to help the model learn the ability to identify passage-level relevance and avoid extracting any answer (i.e.,  $K = 0$ ) for such examples.
2.2 Fine-tuning PMR for Extraction Tasks
We unified downstream span extraction tasks in our MRC formulation, which typically falls into two categories: (1) span extraction with pre-defined labels (e.g., NER) in which each task label is treated as a query to search the corresponding answers in the input text (context) and (2) span extraction with natural questions (e.g., EQA) in which the question is treated as the query for answer extraction from the given passage (context). Then, in the output space, we tackled span extraction problems by predicting the probability  $S_{i,j}$  of context span  $X_{i:j}$  being the answer. The detailed formulation and examples are provided in Appendix A.2.
Implementation. We used the definition articles of the entities that appear as anchors in at least 10 other articles to construct the query. As mentioned in Sec. 2, we prepared 10 answerable query and context pairs for each anchor entity. Then, we paired the query with 10 irrelevant contexts to formulate unanswerable MRC examples. The resulting pre-training corpus consists of 18 million MRC examples (6.4 billion words). We also tried various advanced data construction strategies, such as relevance-driven and diversity-driven ones, to construct query and context pairs. However, no significant performance gain is observed. A detailed comparison is provided in Appendix A.4.
The encoder of PMR is initialized with RoBERTa, a popular MLM with competitive downstream performance. The extractor is randomly initialized, introducing additional 1.6M parameters. In terms of the pre-training efficiency, with four A100 GPUs, it only takes 36 and 89 hours to complete 3-epoch training of PMR for base-sized and large-sized models, respectively. Additional data preprocessing details and hyper-parameter settings can be found in Appendix A.3.
Downstream Extraction Tasks. We evaluated two extraction tasks: EQA and NER.
EQA: We evaluated PMR on MRQA benchmark [15]. For the few-shot setting, we used the few-shot MRQA datasets sampled by Splinter [46]. Although BioASQ and TbQA are originally used for OOD
Model	Size	Unified	SQuAD	TriviaQA	NQ	NewsQA	SearchQA	HotpotQA	BioASQ	TbQA	Avg.
RoBERTa	125M	X	7.74.3	7.54.4	17.33.3	1.40.8	6.92.7	10.52.5	16.77.1	3.32.1	8.9
RBT-Post	125M	X	8.81.2	11.93.4	15.25.6	2.61.2	12.42.5	7.03.5	22.85.1	5.31.5	10.8
SpanBERT	110M	X	18.26.7	11.62.1	19.63.0	7.64.1	13.36.0	12.55.5	15.94.4	7.52.9	13.3
Splinter	125M	✓	54.66.4	18.94.1	27.44.6	20.82.7	26.33.9	24.05.0	28.24.9	19.44.6	27.5
FewshotBART	139M	✓	55.52.0	50.51.0	46.72.3	38.90.7	39.80.04	45.11.5	49.40.02	19.91.3	43.2
PMRbase	125M	✓	46.54.0	47.73.7	32.63.0	26.22.7	50.13.2	32.92.9	49.14.1	27.93.3	39.1
PMRlarge	355M	✓	60.34.0	56.23.1	43.61.7	30.13.7	58.25.0	46.14.7	54.23.4	31.01.8	47.5
RoBERTa	125M	X	18.25.1	10.51.8	22.90.7	3.21.7	13.51.8	10.41.9	23.36.6	4.30.9	13.3
RBT-Post	125M	X	12.73.7	13.42.5	20.73.0	2.90.6	13.42.5	10.80.8	26.05.2	5.31.2	13.2
SpanBERT	110M	X	25.67.7	15.16.4	25.11.6	7.24.6	14.68.5	13.23.5	25.13.3	7.62.3	16.7
Splinter	125M	✓	59.22.1	28.93.1	33.62.4	27.53.2	34.81.8	34.73.9	36.53.2	27.64.3	35.4
FewshotBART	139M	✓	56.82.1	52.50.7	50.11.1	40.41.5	41.80.02	47.91.4	52.30.02	22.72.3	45.6
PMRbase	125M	✓	61.02.9	55.62.7	41.63.6	31.22.7	58.02.8	43.50.8	58.93.1	35.13.5	48.1
PMRlarge	355M	✓	70.03.2	66.32.5	48.53.5	36.62.1	64.82.2	52.92.5	62.92.4	36.43.2	54.8
RoBERTa	125M	X	43.07.1	19.12.9	30.11.9	16.73.8	27.82.5	27.33.9	46.11.4	8.21.1	27.3
RBT-Post	125M	X	11.01.0	26.90.9	24.64.9	4.81.3	27.81.9	13.51.8	30.21.6	8.91.3	18.5
SpanBERT	110M	X	55.83.7	26.32.1	36.01.9	29.57.3	26.34.3	36.63.4	52.23.2	20.95.1	35.5
Splinter	125M	✓	72.71.0	44.73.9	46.30.8	43.51.3	47.24.5	54.71.4	63.24.1	42.62.5	51.9
FewshotBART	139M	✓	68.00.3	50.11.8	53.90.9	47.91.2	58.11.4	54.80.8	68.51.0	29.72.4	53.9
PMRbase	125M	✓	73.10.9	63.61.9	51.91.7	46.90.6	67.51.2	56.41.5	79.21.3	42.72.2	60.2
PMRlarge	355M	✓	81.71.2	70.30.5	57.42.6	52.31.4	70.01.1	65.91.0	78.80.5	45.11.2	65.2
RoBERTa	125M	X	73.80.8	46.80.9	54.21.1	47.51.1	54.31.2	61.81.3	84.11.1	35.82.0	57.3
RBT-Post	125M	X	73.00.4	49.91.2	48.12.6	46.80.9	54.81.0	59.70.8	86.40.3	37.31.3	57.0
SpanBERT	110M	X	77.80.9	50.34.0	57.50.9	49.32.0	60.12.2	67.41.6	89.30.6	42.31.9	61.8
Splinter	125M	✓	82.80.8	64.80.9	65.50.5	57.30.8	67.31.3	70.30.8	91.01.0	54.51.5	69.2
FewshotBART	139M	✓	76.70.8	52.83.2	58.71.4	56.81.6	69.51.0	63.10.8	91.30.5	47.91.9	64.6
PMRbase	125M	✓	82.80.2	69.50.6	66.61.1	59.20.3	74.60.6	69.90.3	94.40.5	54.51.0	71.4
PMRlarge	355M	✓	87.60.7	73.70.8	71.81.2	64.40.9	76.00.9	74.70.9	94.70.2	60.81.6	75.5
Table 1: EQA results (F1) in four few-shot settings. In each setting, we reported the mean and standard deviation over five splits of training data. The results of  $\mathrm{PMR}_{\text{large}}$  are written in blue for demonstration purposes because it has more parameters than others (all base-sized.) The Size column indicates the parameter size of models. The Unified column indicates if the model bridges the gaps between pre-training and fine-tuning  $(\checkmark)$  or not  $(\times)$ . The results of the best base-sized models are written in bold.
evaluation, Splinter [46] constructed few-shot training sets for both datasets by sampling examples from their original dev sets. We followed FewshotQA [6] to build the dev set that has the same size as the training set for model selection. For the full-resource experiments, we followed MRQA [15] in both in-domain and OOD evaluations.
NER: We evaluated PMR on two flat NER datasets, namely, CoNLL and WNUT, and two nested NER datasets, namely, ACE04 and ACE05 [54, 11, 42, 58]. For the few-shot setting, we constructed five splits of K-shot training data, where  $\mathrm{K}\in \{4,8,32,64\}$  sentences are sampled for each tag [38]. We also constructed the dev set that has the same size as the training set.
Baselines. We compared PMR with (1) vanilla MLMs: RoBERTa [36] and SpanBERT [20] with a randomly-initialized extractor, (2) RBT-Post, a continually pre-trained RoBERTa using our Wikipedia data but with the MLM objective; and (3) models bridging the gaps between pre-training and fine-tuning, namely, Splinter [46], FewshotBART [6], EntLM [40], T5-v1.1 [44], and UIE [38], where the latter four are generative methods.
Few-shot Results. The few-shot results of EQA and NER are presented in Tables 1 and 2. The poor few-shot capability of RoBERTa suggests that fine-tuning an MRC extractor from scratch with limited data is extremely challenging. Our PMR achieves notably better results on all few-shot levels of the two tasks than RoBERTa, obtaining an average improvement of 34.8 F1 and 18.6 F1 on 32-shot EQA and 4-shot NER, respectively. Other models that bridge the gaps between pre-training and fine-tuning also perform much better than RoBERTa and RBT-Post, which is consistent with our findings. For EQA, although FewshotBART benefits from a larger output space of the generative
Model	Size	Unified	CoNLL	WNUT	ACE04	ACE05	Avg.	CoNLL	WNUT	ACE04	ACE05	Avg.
RoBERTa	125M	✘	32.47.2	29.73.8	47.54.5	48.13.7	39.4	47.27.4	35.12.0	63.72.4	58.82.7	51.2
RBT-Post	125M	✘	31.87.8	28.73.6	48.74.3	44.34.7	38.4	43.77.1	35.04.8	61.82.2	58.32.6	49.7
EntLM	125M	✘	shot	67.62.2	27.21.8	-	-	-	8-shot	71.30.7	33.30.8	-	-	-
UIE	220M	✓	52.03.3	28.33.1	45.22.3	41.33.0	41.7	66.52.0	39.31.6	52.41.8	51.81.2	52.5
\( PMR_{base} \)	125M	✓	65.14.2	40.83.1	65.32.5	60.72.9	58.0	73.93.2	41.13.9	70.71.8	68.01.3	63.4
\( PMR_{large} \)	355M	✓	65.74.5	40.53.3	65.23.5	66.12.8	59.4	70.34.4	46.42.5	71.71.5	69.92.1	64.6
RoBERTa	125M	✘	77.81.8	47.91.6	76.80.3	74.40.6	69.2	80.81.1	50.61.1	78.71.3	77.90.6	72.0
RBT-Post	125M	✘	77.10.7	45.81.9	75.70.7	73.60.9	68.1	80.91.0	49.82.8	79.21.4	77.51.1	71.9
EntLM	125M	✘	78.90.9	42.40.9	-	-	-	64-shot	82.11.1	46.21.1	-	-	-
UIE	220M	✓	79.61.1	46.21.2	68.00.5	66.30.8	65.0	83.20.8	48.41.2	74.70.6	72.20.4	69.6
\( PMR_{base} \)	125M	✓	81.71.2	50.31.4	79.01.1	76.91.3	72.0	84.40.9	51.52.2	81.60.8	79.50.5	74.3
\( PMR_{large} \)	355M	✓	83.20.6	50.41.6	79.81.2	77.21.5	72.7	84.51.7	53.12.8	81.21.2	79.71.1	74.6
Table 2: NER results (F1) in four few-shot settings. EntLM is not applicable for nested NER tasks.
Size	SQuAD	BioASQ	DROP	DuoRC	RACE	RE	TbQA	Avg.
T5-v1.1	800M	93.9	72.8	47.3	63.9	57.5	87.1	61.5	65.0
RoBERTa	355M	94.2	65.8	54.8	58.6	49.0	88.1	54.7	61.8
PMR	355M	94.5	71.4	62.7	64.1	53.6	88.2	57.5	66.3
model to achieve better transferability in an extremely low-resource EQA setting (i.e., 16 shot), processing such large output is far more complicated than the discriminative MRC and thus is prone to overfitting. MRC-based PMR demonstrates higher effectiveness in learning extraction capability from more training examples than FewshotBART, consequently yielding better performance on EQA when at least 32 examples are provided. Note that the compared baselines for EQA and NER are slightly different due to their different applicability. For example, UIE mainly emphasizes a structured prediction and is not applicable to complicated extraction tasks like EQA. EntLM, which aims to generate label tokens, is also not applicable to EQA. The findings further reveal that PMR can work reasonably well as a zero-shot learner (Appendix A.5).
OOD Generalization. Domain generalization is another common low-resource scenario in which the knowledge can be transferred from a resource-rich domain to a resource-poor domain. We evaluated the domain generalization capability of the proposed PMR on the MRQA benchmark. The MRQA benchmark provides meaningful OOD datasets that are mostly converted from other tasks (e.g., multi-choice QA) and substantially differ from SQuAD in terms of text domain.