SlowGuess/NAACL_2022_OCR huggingface dataset:
James Y. Huang, Bangzheng Li* Jiashu Xu* and Muhao Chen
{huangjam,bangzhen,jiashuxu,muhaoche}@usc.edu
Semantic typing aims at classifying tokens or spans of interest in a textual context into semantic categories such as relations, entity types, and event types. The inferred labels of semantic categories meaningfully interpret how machines understand components of text. In this paper, we present UNIST, a unified framework for semantic typing that captures label semantics by projecting both inputs and labels into a joint semantic embedding space. To formulate different lexical and relational semantic typing tasks as a unified task, we incorporate task descriptions to be jointly encoded with the input, allowing UNIST to be adapted to different tasks without introducing task-specific model components. UNIST optimizes a margin ranking loss such that the semantic relatedness of the input and labels is reflected from their embedding similarity. Our experiments demonstrate that UNIST achieves strong performance across three semantic typing tasks: entity typing, relation classification and event typing. Meanwhile, UNIST effectively transfers semantic knowledge of labels and substantially improves generalizability on inferring rarely seen and unseen types. In addition, multiple semantic typing tasks can be jointly trained within the unified framework, leading to a single compact multi-tasking model that performs comparably to dedicated single-task models, while offering even better transferability. $^{1}$
Semantic typing is a group of fundamental natural language understanding problems that aim at classifying tokens (or spans) of interest into semantic categories. This includes a wide range of long-standing NLP problems such as entity typing, relation classification, and event typing. Inferring the types of entities, relations or events mentioned
is not only crucial to the structural perception of human language, but also plays an important role in many downstream tasks such as entity linking (Onoe and Durrett, 2020), information extraction (Zhong and Chen, 2021) and question answering (Yavuz et al., 2016).
Most traditional methods tackle semantic typing problems by training task-specific multi-class classifiers with token or sentence representations from language models to predict a probability distribution over a pre-defined set of classes (Dai et al., 2021; Yamada et al., 2020). However, this approach comes with several limitations. First, these models simply convert labels into indices, thus completely ignoring the rich semantics carried by the label text itself. For example, given "Currently Ritek is the largest producer of OLEDs.", knowing what the entity type company means would naturally simplify the inference of "Ritek" is a company in this context. Second, models trained as classifiers do not generalize well to class labels that are rarely seen or unseen in the training data, as these models rely on the abundance of annotated examples to associate semantics to label indices. In particular, since these classifiers are limited by the pre-defined label set, they cannot infer any unseen labels unless being re-trained or incorporated with label mapping rules. As a result, these models struggle to handle more fine-grained semantic typing tasks in real-world scenarios (Choi et al., 2018; Chen et al., 2020) where any free-form textual labels may be used to represent the types, many of which may also be unseen during training.
In contrast to the aforementioned traditional paradigm for semantic typing, several studies have explored alternative approaches such as prompt-based learning (Schick and Schütze, 2021; Ding et al., 2021) and indirect supervision from NLI models (Yin et al., 2019; Sainz et al., 2021) to make more efficient use of label semantics. However, these methods usually require hand-crafted
1Our code and pre-trained models are available at https: //github.com/luka-group/UniST.
Human Language Technologies, pages 2642 - 2654
Figure 1: UNIST projects the input sentence with task descriptions (in blue) and marked token span of interest (with enclosing special tokens), and candidate labels into a shared semantic embedding space. In training, it optimizes a margin ranking loss such that positive labels are closer to the input sentence than negative labels. During inference, UNIST simply ranks candidate labels based on the similarity between input and label embeddings.
templates or mapping between labels and language model vocabulary that do not scale well to diverse, free-form labels across various semantic typing tasks. Instead, we seek a generalizable approach that captures label semantics while requiring minimal effort to be adapted to a different task.
In this paper, we propose UNIST, a unified framework for semantic typing that projects context sentences and candidate labels into a shared semantic embedding space. UNIST provides a unified solution to two major categories of semantic typing tasks, namely lexical typing (e.g., entity typing, event typing) and relational typing (relation classification). By optimizing a margin ranking loss, our model captures label semantics such that positive labels are encoded closer to their respective context sentences than negative labels by at least a certain similarity margin. Depending on the task requirement, either top- $k$  candidate labels or any candidate labels with similarity above a certain threshold are given as the final predictions. Furthermore, we add a task description to the end of the context sentences to specify the task and token (spans) of interest, and use a single model for encoding both context sentences and labels. This simple technique allows us to unify different semantic typing tasks without introducing separate task-specific model components or learning objectives, while differentiating among distinct task prediction processes during inference. UNIST demonstrates strong performance on three semantic typing benchmarks: UFET (Choi et al., 2018) for (ultra-fine) entity typing, TACRED (Zhang et al., 2017) for relation classification, and MAVEN (Wang et al., 2020) for event typing, even achieving comparable performance with a single model trained to solve all three tasks simultaneously.
The main contributions of this work are three-fold. First, the proposed UNIST framework converts distinct semantic typing tasks into a unified formulation, where both input and label semantics can be effectively captured in the same representation space. Second, we incorporate a model-agnostic task representation scheme to allow the model to differentiate among distinct tasks in training and inference without introducing additional task-specific model components. Third, UNIST demonstrates substantial improvements in both effectiveness and generalizability on entity typing, relation classification and event typing. In addition, our unified framework makes it possible to learn a single model for all three tasks, which performs comparably to dedicated models trained separately on each task.
In this section, we present the technical details of UNIST, our unified framework for semantic typing. We first provide a general definition of a semantic typing problem (§2.1), followed by a detailed description of our model (§2.2), training objective(§2.3), and inference (§2.4).
Given an input sentence  $s$  and a set of one or more token spans of interest  $E = \{e_1, \ldots, e_n\}$ ,  $e_i \subset s$ , the goal of semantic typing is to assign a set of one or more labels  $Y = \{y_1, \ldots, y_k\}$ ,  $Y \subset \mathcal{Y}$  to  $E$  that best describes the semantic category  $E$  belongs to in the context of  $s$ .  $\mathcal{Y}$  denotes the set of candidate labels, which may include a large number of free-form phrases (Choi et al., 2018) or ontological labels (Zhang et al., 2017). In this paper, we consider two categories of semantic typing tasks, lexical
Task	Input Format
Entity Typing	Currently <E> Ritek </E> is the largest producer of OLEDs.
Describe the type of Ritek.
Relation Classification	<HSUBJ> Herrera </SUBJ> 's wife <OBJ> Ramona </OBJ> died in 1991.
Describe the relationship between person Herrera and person Ramona.
Event Typing	The siege <T> began </T> on 15 September.
Describe the type of began.
Table 1: Input formats for different semantic typing tasks. The four pairs of special tokens marks entities, subjects, objects and triggers respectively.
typing of a single token span (e.g., entity or event typing), and relational typing between two token spans (relation classification).
Overview. As illustrated in Fig. 1, UNIST leverages a pre-trained language model (PLM) to project both input sentences and the candidate labels into a shared semantic embedding space, where the semantic relatedness between the input and label is reflected by their embedding similarity. This is accomplished by optimizing a margin ranking objective that pushes negative labels away from the input sentence while pulling the positive labels towards the input. This simple, unified paradigm allows our model to rank candidate labels based on the affinity of semantic representations with regard to the input during inference. Meanwhile, our model is not limited to a pre-defined label set, as any textual label, whether seen or unseen during training, can be ranked accordingly as long as the model captures its semantic representation. In order to specify the task at hand along with the tokens (or spans) we aim to classify, we add a task description to the end of the input sentence. This allows our framework to use unified representations from a single encoder for both inputs and labels, as well as support the inference of distinct semantic typing tasks without introducing task-specific model components.
Task Description. To highlight the tokens (or spans) we aim to type, we first enclose them with special marker tokens indicating their roles (entities, subjects, objects, or triggers). Next, we leverage the existing semantic knowledge in PLMs and add a natural language task description to the end of the input sentence to specify the task at hand along with tokens (or spans) of interest. The general format for lexical semantic typing is
Describe the type of <tokens>.
and that of relational semantic typing is
Describe the relationship between <subject> and <object>.
Examples of different input formats (including special tokens and task descriptions) can be found in Tab. 1. In addition, relational typing (relation classification) tasks may incorporate entity types from NER models alongside input sentences. Entity type information has been shown to benefit relation classification (Peng et al., 2020; Zhong and Chen, 2021; Zhou and Chen, 2021a), and can be easily incorporated into our task description, as shown in the given example.
Input Representation. We use a RoBERTa model (Liu et al., 2019) to jointly encode the input sentence and the task description. Given an input  $s$  and its task description  $d$ , we concatenate  $s$  and  $d$  into a single sequence, and obtain the hidden representation of the  $
$  token as the input sentence representation, denoted by  $\mathbf{u}$ :
$$
\mathbf {u} = f _ {\text {e n c o d e r}} ([ s, d ]).
$$
A traditional approach to semantic typing is to train classifiers on top of the representations of specific tokens of interest (Wang et al., 2021a; Yamada et al., 2020). In the case of relational typing where two entities are involved, their representations are usually concatenated, leading to dimension mismatch with lexical typing tasks and requiring a different task-specific module to handle. Instead, thanks to the introduction of task description, UNIST always uses the universal  $\langle s \rangle$  token representation for both inputs and labels, and across different semantic typing tasks.
Label Representation. Most semantic typing tasks provide textual labels in natural language from
which a language model can directly capture label semantics. Some relation classification datasets such as TACRED use extra identifiers per: and org: to distinguish same relation type with different subject types. For example, per:parent refers to the parent of a person, while org:parent represents the parent of an organization such as a company. In this case, we simply replace per: and org: with person and organization respectively. The label text is encoded by the exact same model used to encode the input sentence. Given the label  $y$ , we again take the  $
$  token representation as the label representation, denoted by  $v$ :
$$
\mathbf {v} = f _ {\text {e n c o d e r}} (y).
$$
2.3 Learning Objective
Let  $\mathcal{V}$  be the set of all candidates labels for a semantic typing task. Given an input  $[s,d]$  and the positive label set  $Y\subset \mathcal{V}$ , we first randomly sample a negative label  $y^{\prime}\in \mathcal{V}\backslash Y$  for each training instance. Then, we encode the input  $[s,d]$ , positive label  $y$  and negative label  $y^\prime$  into their respective semantic representations  $\mathbf{u}$ ,  $\mathbf{v}$ , and  $\mathbf{v}^{\prime}$ . UNIST optimizes a margin ranking loss such that positive labels, which are more semantically related to the input than negative labels, are also closer to the input in the embedding space. Specifically, the loss function for a single training instance is defined as:
$$
\mathcal {L} _ {s, y, y ^ {\prime}} = \max  \left\{c \left(\mathbf {u}, \mathbf {v} ^ {\prime}\right) - c \left(\mathbf {u}, \mathbf {v}\right) + \gamma , 0 \right\},
$$
where  $c(\cdot)$  denotes cosine similarity and  $\gamma$  is a nonnegative constant. The overall (single-task) training objective is given by:
$$
\mathcal {L} _ {t} = \frac {1}{N _ {t}} \sum_ {s \in S _ {t}} \sum_ {y \in Y _ {s}} \mathcal {L} _ {s, y, y ^ {\prime}},
$$
where  $S_{t}$  is the set of training instances for task  $t$ ,  $Y_{s}$  is the set of all positive labels of  $s$ , and  $N_{t}$  is the number of distinct pairs of training sentence and positive label. In addition to the single-task setting which optimizes an individual task-specific loss  $\mathcal{L}_t$ , we also consider a multi-task setting of UNIST where it is jointly trained on different semantic typing tasks and optimizes the following objective:
$$
\mathcal {L} = \frac {1}{N} \sum_ {t \in T} \sum_ {s \in S _ {t}} \sum_ {y \in Y _ {s}} \mathcal {L} _ {s, y, y ^ {\prime}}.
$$
where  $T$  is the set of semantic typing tasks UNIST is trained on, and  $N$  is the total number of training instances.
2.4 Inference
UNIST supports different strategies for inference depending on the task requirement. If the number of labels for each input is fixed, we simply retrieve the top- $k$  closest candidate labels to the input as the final predictions. Otherwise, all candidate labels with similarity above a certain threshold are given as predictions. Note that UNIST is not restricted to a pre-defined label set, as any textual label in natural language can be encoded by UNIST into its semantic representation and ranked accordingly during inference.
In this section, we evaluate UNIST on single-task experiments on three semantic typing tasks: entity typing (§3.1), relation classification (§3.2) and event typing (§3.3). We then assess the generalizability of UNIST by conducting zero-shot and few-shot prediction, and study the effects of task description (§3.4). Finally, we train UNIST under multi-task setting to solve all three tasks simultaneously (§3.5).
3.1 Ultra-fine Entity Typing
We first conduct experiments on the ultra-fine entity typing task, which aims at predicting fine-grained free-form words or phrases that describe the appropriate types of entities mentioned in sentences.
Dataset. We use the Ultra-Fine Entity Typing (UFET) benchmark (Choi et al., 2018), which includes 5,994 sentences split into 1,998 each for train, dev and test. Each entity mention in UFET is annotated with one or more free-form type labels, covering a set of 2,519 distinct words and phrases. Following the original evaluation protocol, we report macro precision, recall and F1 score on the UFET test set.
Model. Since the number of ground truth labels for each entity is not fixed, all candidate labels with similarity above a certain threshold is given as the final predictions. We tune the hyperparameters, including the threshold, on the UFET dev set. We use base and large versions of RoBERTa as encoders for UNIST<sub>BASE</sub> and UNIST<sub>LARGE</sub> respectively.
Baselines. UFET-biLSTM (Choi et al., 2018) learns context and mention representations by combining pre-trained word embeddings with a character-level CNN and a bi-LSTM. LabelGCN (Xiong et al., 2019) adds a graph propagation layer
Model	P	R	F1
UFET-biLSTM† (Choi et al., 2018)	48.1	23.3	31.3
LabelGCN† (Xiong et al., 2019)	50.3	29.2	36.9
LDET† (Onoe and Durrett, 2019)	51.5	33.0	40.1
Box4Types*† (Onoe et al., 2021)	52.8	38.8	44.8
LRN (Liu et al., 2021)	54.5	38.9	45.4
MLMET† (Dai et al., 2021)	53.6	45.3	49.1
UNISTBASE	49.2	49.4	49.3
UNISTLARGE*	50.2	49.6	49.9
to capture label dependencies. LDET (Onoe and Durrett, 2019) learns a denoising model that automatically filters and relabels distant supervision data for training. Box4Types (Onoe et al., 2021) introduces box embeddings to represent type hierarchies and uses BERTLARGE as context and mention encoder. LRN (Liu et al., 2021) uses an autoregressive LSTM to discover label structures, a bipartite attribute graph to capture intrinsic label dependencies, and a BERTBASE as sentence encoder. MLMET (Dai et al., 2021) generatively augments the training data with a masked language model, and fine-tunes BERTBASE on the augmented training set.
Results. As shown in Tab. 2, UNIST<sub>BASE</sub> already outperforms the SOTA baseline MLMET without training on any augmented data by  $0.2\%$  in F1 score. With a larger language model, UNIST<sub>LARGE</sub> further improves F1 score by another  $0.6\%$ . Since UFET only provides a small set of human annotated training data compared to its diverse label set, all baselines except LRN incorporate distant supervision data to alleviate data scarcity. UNIST's superior performance on UFET demonstrates the importance of capturing label semantics as an auxiliary supervision signal that is not fully exploited by previous methods. This is especially beneficial when annotated data are limited, and can alleviate the model's reliance on augmenting training data. In this way, UNIST also achieves better generalizability to unseen and rarely seen labels, for which we conduct a more detailed analysis on few-shot and zero-shot UFET labels in §3.4.
3.2 Relation Classification
The goal of relation classification is to determine the relation between a subject entity and an object entity mentioned in a sentence.
Table 2: Results of entity typing on UFET. * marks models based on large versions of PLMs. † marks models using augmented training data.
Model	P	R	F1
SpanBERT* (Joshi et al., 2020)	70.8	70.9	70.8
MTB* (Baldini Soares et al., 2019)	-	-	71.5
TANL (Paolini et al., 2021)	-	-	71.9
K-Adapter* (Wang et al., 2021a)	70.1	74.0	72.0
LUKE* (Yamada et al., 2020)	70.4	75.1	72.7
BERT-CR* (Zhou and Chen, 2021b)	-	-	73.0
IBRE* (Zhou and Chen, 2021a)	-	-	74.6
SP* (Cohen et al., 2020)	74.6	75.2	74.8
UNISTBASE	73.6	75.0	74.3
UNISTLARGE*	78.0	73.1	75.5
Table 3: Results of relation classification on TACRED. * marks models based on large versions of PLMs.
Dataset. We run the experiments on TACRED (Zhang et al., 2017), a widely used benchmark for this task that contains 106,264 sentences with entity pairs labeled as one of the 41 relation types or a no Relation type. TACRED provides 68,124 instances for training, 22,631 for dev, and 15,509 for testing. Following the original evaluation protocol, we report micro precision, recall and F1 score on the TACRED test set.
Model Configuration. UNIST retrieves the candidate label closest to the input in the embedding space as the final prediction. Since entities in TACRED are also annotated with entity types, we place the entity type labels in front of their corresponding entity mentions in the task description to provide additional information for relation classification, as shown in Tab. 1. We tune the hyperparameters on the TACRED dev set.
Baselines. SpanBERT (Joshi et al., 2020) incorporates span prediction as an additional objective for BERT pre-training. MTB (Baldini Soares et al., 2019) introduces matching-the-blank training on entity-linked text to connect relation representations among related instances. TANL (Paolini et al., 2021) proposes a unified text-to-text framework for structured prediction tasks based on T5 (Raffel et al., 2020). K-Adapter (Wang et al., 2021a) learns adapter modules to infuse structured knowledge into a RoBERTaLarge model. LUKE further trains RoBERTaLarge on entity-annotated corpus with an entity-aware self-attention mechanism. BERT-CR (Zhou and Chen, 2021b) introduces a co-regularization framework to improve learning from noisy datasets with a  $\mathrm{BERT}_{\mathrm{LARGE}}$  model. IBRE (Zhou and Chen, 2021b) incorporates entity type information into mention markers in the sentence to boost the performance of
RoBERTa $_{\text{LARGE}}$ . SP (Cohen et al., 2020) formulates relation classification as a two-way span prediction problem, and uses ALBERT (Lan et al., 2020) as encoder $^2$ .
Results. As shown in Tab. 3,  $\mathrm{UNIST}_{\mathrm{BASE}}$  already outperforms several strong baselines which are built on larger PLMs (BERTLARGE or RoBERTaLARGE), except for SP and IBRE.  $\mathrm{UNIST}_{\mathrm{LARGE}}$  further improves the performance and establishes new SOTA on TACRED, outperforming the best baseline SP by  $0.7\%$  in F1. While SP also leverages label semantics by framing relation classification as a two-way question answering problem, it requires hand-crafted question templates for each relation label and more significant computational cost for answer span prediction. In comparison, UNIST directly captures label semantics from the label text itself, while offering superior performance and inference efficiency as labels can be retrieved by simply computing embedding cosine similarity.
3.3 Event Typing
Event typing aims at assigning an event type to an event trigger that clearly indicates an event.
Dataset. We conduct the evaluation using MAVEN (Wang et al., 2020), a general-domain event extraction benchmark with 77,993/18,904/21,835 event triggers for train/dev/test annotated with 168 distinct event types. MAVEN also provides a large set of negative triggers, which includes all content words (nouns, verbs, adjectives, and adverbs) labeled by a part-of-speech tagger but not annotated as an event trigger. Since UNIST focuses on semantic typing and does not handle mention span prediction, we train a BERT-CRF model to first identify trigger candidates following Wang et al. (2020), and then predict an event type for each trigger candidate using UNIST. Following the original paper, we report micro precision, recall and F1 score on MAVEN test set.
Model Configuration. We retrieve the candidate label with the highest similarity to the input as the predicted event type. We tune the hyperparameters on the MAVEN dev set.
Model	P	R	F1
DMCNN (Chen et al., 2015)	66.3	55.9	60.6
MOGANED (Yan et al., 2019)	63.4	64.1	63.8
DMBERT (Wang et al., 2019)	62.7	72.3	67.1
BERT-CRF (Wang et al., 2020)	65.0	70.9	67.8
CLEVE* (Wang et al., 2021b)	64.9	72.6	68.5
UNISTBASE	66.7	69.9	68.3
UNISTLARGE*	66.5	69.7	68.1
Table 4: Results of event typing on MAVEN. * marks models based on large versions of PLMs. All baseline results except CLEVE are taken from Wang et al. (2020)