Venue: NAACL 2022

# Unified Semantic Typing with Meaningful Label Inference

James Y. Huang, Bangzheng Li1; Jiashu Xu* and Muhao Chen

University of Southern California

Los Angeles, California, USA {huangjam,bangzhen,jiashuxu,muhaoche}@usc.edu

Equal contributions.

###### Abstract

Semantic typing aims at classifying tokens or spans of interest in a textual context into semantic categories such as relations, entity types, and event types. The inferred labels of semantic categories meaningfully interpret how machines understand components of text. In this paper, we present UniST, a unified framework for semantic typing that captures label semantics by projecting both inputs and labels into a joint semantic embedding space. To formulate different lexical and relational semantic typing tasks as a unified task, we incorporate task descriptions to be jointly encoded with the input, allowing UniST to be adapted to different tasks without introducing task-specific model components. UniST optimizes a margin ranking loss such that the semantic relatedness of the input and labels is reflected from their embedding similarity. Our experiments demonstrate that UniST achieves strong performance across three semantic typing tasks: entity typing, relation classification and event typging. Meanwhile, UniST effectively transfers semantic knowledge of labels and substantially improves generalizability on inferring rarely seen and unseen types. In addition, multiple semantic typing tasks can be jointly trained within the unified framework, leading to a single compact multi-tasking model that performs comparably to dedicated single-task models, while offering even better transferability.1

Footnote 1: Our code and pre-trained models are available at [https://github.com/luka-group/UniST](https://github.com/luka-group/UniST).

+
Footnote †: _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 2642 - 2654

July 10-15, 2022 ©2022 Association for Computational Linguistics

## 1 Introduction

Semantic typing is a group of fundamental natural language understanding problems that aim at classifying tokens (or spans) of interest into semantic categories. This includes a wide range of long-standing NLP problems such as entity typing, relation classification, and event typing. Inferring the types of entities, relations or events mentioned is not only crucial to the structural perception of human language, but also plays an important role in many downstream tasks such as entity linking Onoe and Durrett (2020), information extraction Zhong and Chen (2021) and question answering Yavuz et al. (2016).

Most traditional methods tackle semantic typing problems by training task-specific multi-class classifiers with token or sentence representations from language models to predict a probability distribution over a pre-defined set of classes Dai et al. (2021); Yamada et al. (2020). However, this approach comes with several limitations. First, these models simply convert labels into indices, thus completely ignoring the rich semantics carried by the label text itself. For example, given "_Currently Ritek is the largest producer of OLEDs._", knowing what the entity type _company_ means would naturally simplify the inference of "_Ritek_" is a _company_ in this context. Second, models trained as classifiers do not generalize well to class labels that are rarely seen or unseen in the training data, as these models rely on the abundance of annotated examples to associate semantics to label indices. In particular, since these classifiers are limited by the pre-defined label set, they cannot infer any unseen labels unless being re-trained or incorporated with label mapping rules. As a result, these models struggle to handle more fine-grained semantic typing tasks in real-world scenarios Choi et al. (2018); Chen et al. (2020) where any free-form textual labels may be used to represent the types, many of which may also be unseen during training.

In contrast to the aforementioned traditional paradigm for semantic typing, several studies have explored alternative approaches such as prompt-based learning Schick and Schutze (2021); Ding et al. (2021) and indirect supervision from NLI models Yin et al. (2019); Sainz et al. (2021) to make more efficient use of label semantics. However, these methods usually require hand-crafted

templates or mapping between labels and language model vocabulary that do not scale well to diverse, free-form labels across various semantic typing tasks. Instead, we seek a generalizable approach that captures label semantics while requiring minimal effort to be adapted to a different task.

In this paper, we propose UniST, a unified framework for semantic typing that projects context sentences and candidate labels into a shared semantic embedding space. UniST provides a unified solution to two major categories of semantic typing tasks, namely lexical typing (e.g., entity typing, event typing) and relational typing (relation classification). By optimizing a margin ranking loss, our model captures label semantics such that positive labels are encoded closer to their respective context sentences than negative labels by at least a certain similarity margin. Depending on the task requirement, either top-\(k\) candidate labels or any candidate labels with similarity above a certain threshold are given as the final predictions. Furthermore, we add a _task description_ to the end of the context sentences to specify the task and token (spans) of interest, and use a single model for encoding both context sentences and labels. This simple technique allows us to unify different semantic typing tasks without introducing separate task-specific model components or learning objectives, while differentiating among distinct task prediction processes during inference. UniST demonstrates strong performance on three semantic typing benchmarks: UFET Choi et al. (2018) for (ultra-fine) entity typing, TACRED Zhang et al. (2017) for relation classification, and MAVEN Wang et al. (2020) for event typing, even achieving comparable performance with a single model trained to solve all three tasks simultaneously.

The main contributions of this work are threefold. First, the proposed UniST framework converts distinct semantic typing tasks into a unified formulation, where both input and label semantics can be effectively captured in the same representation space. Second, we incorporate a model-agnostic task representation scheme to allow the model to differentiate among distinct tasks in training and inference without introducing additional task-specific model components. Third, UniST demonstrates substantial improvements in both effectiveness and generalizability on entity typing, relation classification and event typing. In addition, our unified framework makes it possible to learn a single model for all three tasks, which performs comparably to dedicated models trained separately on each task.

## 2 Method

In this section, we present the technical details of UniST, our unified framework for semantic typing. We first provide a general definition of a semantic typing problem (SS2.1), followed by a detailed description of our model (SS2.2), training objective(SS2.3), and inference (SS2.4).

### Problem Definition

Given an input sentence \(s\) and a set of one or more token spans of interest \(E=\{e_{1},...e_{n}\},e_{i}\subset s\), the goal of semantic typing is to assign a set of one or more labels \(Y=\{y_{1},...y_{k}\},Y\subset\mathcal{Y}\) to \(E\) that best describes the semantic category \(E\) belongs to in the context of \(s\). \(\mathcal{Y}\) denotes the set of candidate labels, which may include a large number of free-form phrases Choi et al. (2018) or ontological labels Zhang et al. (2017). In this paper, we consider two categories of semantic typing tasks, _lexical

Figure 1: UniST projects the input sentence with task descriptions (in blue) and marked token span of interest (with enclosing special tokens), and candidate labels into a shared semantic embedding space. In training, it optimizes a margin ranking loss such that positive labels are closer to the input sentence than negative labels. During inference, UniST simply ranks candidate labels based on the similarity between input and label embeddings.

 

typing_ of a single token span (e.g., entity or event typing), and _relational typing_ between two token spans (relation classification).

### Model

**Overview.** As illustrated in Fig. 1, UniST leverages a pre-trained language model (PLM) to project both input sentences and the candidate labels into a shared semantic embedding space, where the semantic relatedness between the input and label is reflected by their embedding similarity. This is accomplished by optimizing a margin ranking objective that pushes negative labels away from the input sentence while pulling the positive labels towards the input. This simple, unified paradigm allows our model to rank candidate labels based on the affinity of semantic representations with regard to the input during inference. Meanwhile, our model is not limited to a pre-defined label set, as any textual label, whether seen or unseen during training, can be ranked accordingly as long as the model captures its semantic representation. In order to specify the task at hand along with the tokens (or spans) we aim to classify, we add a task description to the end of the input sentence. This allows our framework to use unified representations from a single encoder for both inputs and labels, as well as support the inference of distinct semantic typing tasks without introducing task-specific model components.

**Task Description.** To highlight the tokens (or spans) we aim to type, we first enclose them with special marker tokens indicating their roles (entities, subjects, objects, or triggers). Next, we leverage the existing semantic knowledge in PLMs and add a natural language task description to the end of the input sentence to specify the task at hand along with tokens (or spans) of interest. The general format for lexical semantic typing is

\begin{tabular}{c|c} \hline \hline Task & Input Format \\ \hline Entity Typing & Currently <E> Ritek /E> is the largest producer of OLEDs. \\  & Describe the type of Ritek. \\ \hline Relation Classification & <SUBJ> Herrera /<SUBJ> ’ swife’ <OBJ> Ramona /OJB> died in 1991. \\  & Describe the relationship between _person_ Herrera and _person_ Ramona. \\ \hline Event Typing & The siege <T> began /C> on 15 September. \\  & Describe the type of began. \\ \hline \hline \end{tabular}

Table 1: Input formats for different semantic typing tasks. The four pairs of special tokens marks entities, subjects, objects and triggers respectively.

\begin{tabular}{l l} \hline \hline \multicolumn{2}{|c|}{_Describe the type of <tokens>._} \\ \hline \multicolumn{2}{|c|}{and that of relational semantic typing is} \\ \hline \multicolumn{2}{|c|}{_Describe the relationship between <subject> and_} \\ \cline{2-3} \multicolumn{2}{|c|}{_<object>._} \\ \hline \hline Examples of different input formats (including special tokens and task descriptions) can be found in Tab. 1. In addition, relational typing (relation classification) tasks may incorporate entity types from NER models alongside input sentences. Entity type information has been shown to benefit relation classification Peng et al. (2020); Zhong and Chen (2021); Zhou and Chen (2021), and can be easily incorporated into our task description, as shown in the given example.

**Input Representation.** We use a RoBERTa model Liu et al. (2019) to jointly encode the input sentence and the task description. Given an input \(s\) and its task description \(d\), we concatenate \(s\) and \(d\) into a single sequence, and obtain the hidden representation of the \(<_{S}>\) token as the input sentence representation, denoted by \(\mathbf{u}\):

\[\mathbf{u}=f_{\text{encoder}}([s,d]).\]

A traditional approach to semantic typing is to train classifiers on top of the representations of specific tokens of interest Wang et al. (2021); Yamada et al. (2020). In the case of relational typing where two entities are involved, their representations are usually concatenated, leading to dimension mismatch with lexical typing tasks and requiring a different task-specific module to handle. Instead, thanks to the introduction of task description, UniST always uses the universal \(<\!\approx\) token representation for both inputs and labels, and across different semantic typing tasks.

**Label Representation.** Most semantic typing tasks provide textual labels in natural language from 

which a language model can directly capture label semantics. Some relation classification datasets such as TACRED use extra identifiers _per:_ and _org:_ to distinguish same relation type with different subject types. For example, _per:parent_ refers to the parent of a person, while _org:parent_ represents the parent of an organization such as a company. In this case, we simply replace _per:_ and _org:_ with _person_ and _organization_ respectively. The label text is encoded by the exact same model used to encode the input sentence. Given the label \(y\), we again take the <S> token representation as the label representation, denoted by \({\bf v}\):

\[{\bf v}=f_{\text{encoder}}(y).\]

### Learning Objective

Let \(\mathcal{Y}\) be the set of all candidates labels for a semantic typing task. Given an input \([s,d]\) and the positive label set \(Y\subset\mathcal{Y}\), we first randomly sample a negative label \(y^{\prime}\in\mathcal{Y}\backslash Y\) for each training instance. Then, we encode the input \([s,d]\), positive label \(y\) and negative label \(y^{\prime}\) into their respective semantic representations \({\bf u}\), \({\bf v}\), and \({\bf v}^{\prime}\). UniST optimizes a margin ranking loss such that positive labels, which are more semantically related to the input than negative labels, are also closer to the input in the embedding space. Specifically, the loss function for a single training instance is defined as:

\[\mathcal{L}_{s,y,y^{\prime}}=\max\{c({\bf u},{\bf v}^{\prime})-c({\bf u},{\bf v })+\gamma,0\},\]

where \(c(\cdot)\) denotes cosine similarity and \(\gamma\) is a non-negative constant. The overall (single-task) training objective is given by:

\[\mathcal{L}_{t}=\frac{1}{N_{t}}\sum_{s\in S_{t}}\sum_{y\in Y_{s}}\mathcal{L}_{ s,y,y^{\prime}},\]

where \(S_{t}\) is the set of training instances for task \(t\), \(Y_{s}\) is the set of all positive labels of \(s\), and \(N_{t}\) is the number of distinct pairs of training sentence and positive label. In addition to the single-task setting which optimizes an individual task-specific loss \(\mathcal{L}_{t}\), we also consider a multi-task setting of UniST where it is jointly trained on different semantic typing tasks and optimizes the following objective:

\[\mathcal{L}=\frac{1}{N}\sum_{t\in T}\sum_{s\in S_{t}}\sum_{y\in Y_{s}}\mathcal{ L}_{s,y,y^{\prime}}.\]

where \(T\) is the set of semantic typing tasks UniST is trained on, and \(N\) is the total number of training instances.

### Inference

UniST supports different strategies for inference depending on the task requirement. If the number of labels for each input is fixed, we simply retrieve the top-\(k\) closest candidate labels to the input as the final predictions. Otherwise, all candidate labels with similarity above a certain threshold are given as predictions. Note that UniST is not restricted to a pre-defined label set, as any textual label in natural language can be encoded by UniST into its semantic representation and ranked accordingly during inference.

## 3 Experiments

In this section, we evaluate UniST on single-task experiments on three semantic typing tasks: entity typing (SS3.1), relation classification (SS3.2) and event typing (SS3.3). We then assess the generalizability of UniST by conducting zero-shot and few-shot prediction, and study the effects of task description (SS3.4). Finally, we train UniST under multi-task setting to solve all three tasks simultaneously (SS3.5).

### Ultra-fine Entity Typing

We first conduct experiments on the ultra-fine entity typing task, which aims at predicting fine-grained free-form words or phrases that describe the appropriate types of entities mentioned in sentences.

**Dataset.** We use the Ultra-Fine Entity Typing (UFET) benchmark Choi et al. (2018), which includes 5,994 sentences split into 1,998 each for train, dev and test. Each entity mention in UFET is annotated with one or more free-form type labels, covering a set of 2,519 distinct words and phrases. Following the original evaluation protocol, we report macro precision, recall and F1 score on the UFET test set.

**Model.** Since the number of ground truth labels for each entity is not fixed, all candidate labels with similarity above a certain threshold is given as the final predictions. We tune the hyperparameters, including the threshold, on the UFET dev set. We use base and large versions of RoBERTa as encoders for UniSTBASE and UniSTLarge respectively.

**Baselines. UFET-biLSTM**Choi et al. (2018) learns context and mention representations by combining pre-trained word embeddings with a character-level CNN and a bi-LSTM. **LabelGCN**Xiong et al. (2019) adds a graph propagation layer 

to capture label dependencies. **LDET**Onoe and Durrett (2019) learns a denoising model that automatically filters and relabels distant supervision data for training. **Box4Types**Onoe et al. (2021) introduces box embeddings to represent type hierarchies and uses BERT\({}_{\text{LARGE}}\) as context and mention encoder. **LRN**Liu et al. (2021) uses an auto-regressive LSTM to discover label structures, a bipartite attribute graph to capture intrinsic label dependencies, and a BERT\({}_{\text{BASE}}\) as sentence encoder. **MLMET**Dai et al. (2021) generatively augments the training data with a masked language model, and fine-tunes BER\({}_{\text{BASE}}\) on the augmented training set.

**Results.** As shown in Tab. 2, UniST\({}_{\text{BASE}}\) already outperforms the SOTA baseline MLMET without training on any augmented data by 0.2% in F1 score. With a larger language model, UniST\({}_{\text{LARGE}}\) further improves F1 score by another 0.6%. Since UFET only provides a small set of human annotated training data compared to its diverse label set, all baselines except LRN incorporate distant supervision data to alleviate data scarcity. UniST's superior performance on UFET demonstrates the importance of capturing label semantics as an auxiliary supervision signal that is not fully exploited by previous methods. This is especially beneficial when annotated data are limited, and can alleviate the model's reliance on augmenting training data. In this way, UniST also achieves better generalizability to unseen and rarely seen labels, for which we conduct a more detailed analysis on few-shot and zero-shot UFET labels in SS3.4.

### Relation Classification

The goal of relation classification is to determine the relation between a subject entity and an object entity mentioned in a sentence.

Dataset.We run the experiments on TACRED Zhang et al. (2017), a widely used benchmark for this task that contains 106,264 sentences with entity pairs labeled as one of the 41 relation types or a _no_relation_ type. TACRED provides 68,124 instances for training, 22,631 for dev, and 15,509 for testing. Following the original evaluation protocol, we report micro precision, recall and F1 score on the TACRED test set.

Model Configuration.UniST retrieves the candidate label closest to the input in the embedding space as the final prediction. Since entities in TACRED are also annotated with entity types, we place the entity type labels in front of their corresponding entity mentions in the task description to provide additional information for relation classification, as shown in Tab. 1. We tune the hyperparameters on the TACRED dev set.

Baselines.SpanBERT Joshi et al. (2020) incorporates span prediction as an additional objective for BERT pre-training. **MTB**Baldini Soares et al. (2019) introduces matching-the-blank training on entity-linked text to connect relation representations among related instances. **TANL**Paolini et al. (2021) proposes a unified text-to-text framework for structured prediction tasks based on T5 Raffel et al. (2020). **K-Adapter**Wang et al. (2021) learns adapter modules to infuse structured knowledge into a RoBERTa\({}_{\text{LARGE}}\) model. **LUKE**further trains RoBERTa\({}_{\text{LARGE}}\) on entity-annotated corpus with an entity-aware self-attention mechanism. **BERT-CR**Zhou and Chen (2021) introduces a co-regularization framework to improve learning from noisy datasets with a BERT\({}_{\text{LARGE}}\) model. **IBRE**Zhou and Chen (2021) incorporates entity type information into mention markers in the sentence to boost the performance of

\begin{table}
\begin{tabular}{l|c c c} \hline Model & P & R & F1 \\ \hline UFET-biLSTM Choi et al. (2018) & 48.1 & 23.3 & 31.3 \\ LabelGCN Xiong et al. (2019) & 50.3 & 29.2 & 36.9 \\ LDET\({}^{\dagger}\)Onoe and Durrett (2019) & 51.5 & 33.0 & 40.1 \\ Box4Types* Onoe et al. (2021) & 52.8 & 38.8 & 44.8 \\ LRN Liu et al. (2021) & **54.5** & 38.9 & 45.4 \\ MLMET\({}^{\dagger}\)Dai et al. (2021) & 53.6 & 45.3 & 49.1 \\ \hline UniST\({}_{\text{BASE}}\) & 49.2 & 49.4 & 49.3 \\ UniST\({}_{\text{LARGE}}\)* & 50.2 & **49.6** & **49.9** \\ \hline \end{tabular}
\end{table}
Table 2: Results of entity typing on UFET. * marks models based on large versions of PLMs. \({}^{\dagger}\) marks models using augmented training data.

\begin{table}
\begin{tabular}{l|c c c} \hline Model & P & R & F1 \\ \hline SpanBERT* Joshi et al. (2020) & 70.8 & 70.9 & 70.8 \\ MTB* Baldini Soares et al. (2019) & - & - & 71.5 \\ TANL Paolini et al. (2021) & - & - & 71.9 \\ K-Adapter Wang et al. (2021) & 70.1 & 74.0 & 72.0 \\ LUKE+ Yamada et al. (2020) & 70.4 & 75.1 & 72.7 \\ BERT-CR* Zhou and Chen (2021) & - & - & 73.0 \\ IBRE* Zhou and Chen (2021) & - & - & 74.6 \\ SP* Cohen et al. (2020) & 74.6 & **75.2** & 74.8 \\ \hline UniST\({}_{\text{BASE}}\) & 73.6 & 75.0 & 74.3 \\ UniST\({}_{\text{LARGE}}\)* & **78.0** & 73.1 & **75.5** \\ \hline \end{tabular}
\end{table}
Table 3: Results of relation classification on TACRED. * marks models based on large versions of PLMs.

 

RoBERTaLARGE. **SP**(Cohen et al., 2020) formulates relation classification as a two-way span prediction problem, and uses ALBERT Lan et al. (2020) as encoder2.

Footnote 2: We were unable to reproduce the results of RECENT Lyu and Chen (2021) due to an error in its evaluation process that wrongly corrected all false positive predictions during testing. After correcting that error, the performance of RECENT was observed to be below the other baselines, and hence has not been included in the result discussion.

Results.As shown in Tab. 3, UniSTBASE already outperforms several strong baselines which are built on larger PLMs (BERT\({}_{\text{LARGE}}\) or RoBERTa\({}_{\text{LARGE}}\)), except for SP and IBRE. UniST\({}_{\text{LARGE}}\) further improves the performance and establishes new SOTA on TACRED, outperforming the best baseline SP by 0.7% in F1. While SP also leverages label semantics by framing relation classification as a two-way question answering problem, it requires hand-crafted question templates for each relation label and more significant computational cost for answer span prediction. In comparison, UniST directly captures label semantics from the label text itself, while offering superior performance and inference efficiency as labels can be retrieved by simply computing embedding cosine similarity.

### Event Typing

Event typing aims at assigning an event type to an event trigger that clearly indicates an event.

Dataset.We conduct the evaluation using MAVEN Wang et al. (2020), a general-domain event extraction benchmark with 77,993/18,904/21,835 event triggers for train/dev/test annotated with 168 distinct event types. MAVEN also provides a large set of negative triggers, which includes all content words (nouns, verbs, adjectives, and adverbs) labeled by a part-of-speech tagger but not annotated as an event trigger. Since UniST focuses on semantic typing and does not handle mention span prediction, we train a BERT-CRF model to first identify trigger candidates following Wang et al. (2020), and then predict an event type for each trigger candidate using UniST. Following the original paper, we report micro precision, recall and F1 score on MAVEN test set.

Model Configuration.We retrieve the candidate label with the highest similarity to the input as the predicted event type. We tune the hyperparameters on the MAVEN dev set.

Baselines.DMCNNChen et al. (2015) uses a CNN with dynamic multi-pooling to obtain trigger representations for classification. MOGANEDYan et al. (2019) proposes a multi-order GCN to capture interrelation between event trigger and argument representations based on dependency trees. DMBERTWang et al. (2019) improves DMCNN by training a BERTBASE model as sentence encoder with dynamic multi-pooling. BERT-CRF stacks a CRF layer on top of BERTBASE to model multiple event correlations in a single sentence. CLEVEWang et al. (2021b) proposes a contrastive learning framework fine-tuned on large-scale corpus with AMR structures obtained from AMR parsers, and combines AMR graph representations from a GNN and text representations from RoBERTaLARGE to classify event types.

Results.As shown in Tab. 4, UniST is able to improve event typing over BERT-CRF, and outperform all baselines except CLEVE. Note that in addition to being initialized from the same RoBERTa model as UniST, CLEVE is further fine-tuned on large-scale corpus with AMR structures obtained from a separate parsing model Xu et al. (2020) that also requires large human-annotated data to train. This indicates much more expensive supervision signals used by CLEVE. In contrast, UniST effectively captures the meaning of event types and learns to classify event triggers by only fine-tuning on MAVEN, while still achieving promising performance without the need of any additional annotated resources.

### Analysis

In this section, we provide a detailed analysis to better understand the generalizability of UniST and the effects of incorporating task description. Specifically, we examine UniST's performance on few-shot and zero-shot entity typing on UFET,

\begin{table}
\begin{tabular}{l|c c c} \hline \hline Model & P & R & F1 \\ \hline DMCNN Chen et al. (2015) & 66.3 & 55.9 & 60.6 \\ MOGANED Yan et al. (2019) & 63.4 & 64.1 & 63.8 \\ DMBERT Wang et al. (2019) & 62.7 & 72.3 & 67.1 \\ BERT-CRF Wang et al. (2020) & 65.0 & 70.9 & 67.8 \\ CLEVE* Wang et al. (2021b) & 64.9 & **72.6** & **68.5** \\ \hline UniST\({}_{\text{BASE}}\) & **66.7** & 69.9 & 68.3 \\ UniST\({}_{\text{LARGE}}\)* & 66.5 & 69.7 & 68.1 \\ \hline \hline \end{tabular}
\end{table}
Table 4: Results of event typing on MAVEN. * marks models based on large versions of PLMs. All baseline results except CLEVE are taken from Wang et al. (2020) 

zero-shot relation classification on FewRel Han et al. (2018), and how UniST performs without task descriptions.

**Few-shot & Zero-shot Entity Typing.** A large portion of UFET test set labels have very few or even no training instances. We focus on entity types with no more than 10 instances in the training set, and compare the performance of UniSTBASE with the previous SOTA model MLMET on these few-shot and zero-shot labels.

As shown in Fig. 2, the advantage of UniST over MLMET becomes more evident for rarer labels. For the most challenging zero-shot labels, UniST substantially outperforms MLMET by 7.2% in F1 score, suggesting that UniST is better generalized to infer low-resource and unseen entity types.

**Zero-shot Relation Classification.** We conduct experiments on FewRel Han et al. (2018), a widely used benchmark for low-resource relation classification. FewRel includes 64/16/20 non-overlapping relation types for train/dev/test with 700 sentences collected from Wikipedia for each relation type. We evaluate UniST under the \(N\)-way-0-shot setting, where the goal is to predict the correct relation among \(N\) candidate relations without seen training examples. Following previous studies Cetoli (2020); Dong et al. (2021), we report 5-way-0-shot and 10-way-0-shot accuracy on the FewRel dev set.

We compare UniST with following baselines: **REGRAB**Qu et al. (2020) proposes a bayesian meta-learning method to infer the posterior distribution of relation prototypes initialized with knowledge graph embeddings. **BERT-SQuAD**Cetoli (2020) formulates zero-shot relation classification as a question answering problem, and fine-tunes a BERTLARGE QA model trained on SQuAD 1.1 Rajpurkar et al. (2016) to predict relation types. **MapRE**Dong et al. (2021) proposes a contrastive pre-training framework that learns input and relation representations from large-scale relation-annotated data. All baselines, as well as UniST, are fine-tuned on the FewRel training set, and then evaluated on the FewRel dev set with a new set of relation types completely _disjoint_ from that of the training set.

As shown in Tab. 5, UniST outperforms the best baseline MapRE by 0.5% and 1.4% in accuracy on 5-way-0-shot and 10-way-0-shot tasks without first pre-training on any relation-annotated data. This demonstrates that by effectively captures label semantics, UniST allows better knowledge transfer to handle unseen relation types.

**Effects of Task Description.** We conduct an ablation experiment on task descriptions using UniST to better understand their effects on downstream tasks. As shown in Tab. 6, the performance on TACRED degrades much more significantly compared to that on UFET and MAVEN after removing task description. In lexical typing, the token span to be classified tend to share similar semantics with its type, and in many cases can be easily matched to its type label without explicitly specifying the task. In contrast, relation types are usually not semantically similar to its subject and object, and task description helps bridge this gap.

### Multi-task Learning

With a unified task formulation, UniST facilities learning a single model to jointly train on and simultaneously solve different semantic typing tasks. For more balanced training, We train UniST on

\begin{table}
\begin{tabular}{l|c c c} \hline \hline Model & UFET & TACRED & MAVEN \\ \hline UniSTBASE & **49.3** & **74.3** & **68.3** \\ - without task description & 49.2 & 72.9 & 68.2 \\ \hline \hline \end{tabular}
\end{table}
Table 6: F1 results of ablation experiments without task description on UFET, TACRED and MAVEN.

Figure 2: Comparison between MLMET and UniST on few-shot and zero-shot prediction on UFET.

 

the combined training set of UFET, TACRED and MAVEN, and report F1 performance on their respective test sets by following their respective evaluation protocol. We also include performance of single-task UniST for comparison.

As shown in Tab. 7, our multi-task model obtain generally comparable performance to dedicated UniST models trained separately on each of the three semantic typing tasks. Despite a slight decrease in performance on some of the tasks, UniST 

captures label semantics solely from label names, and unify different semantic typing tasks into a single framework by incorporating task descriptions to be jointly encoded with the input.

## 5 Conclusion

We propose UniST, a unified framework for semantic typing that exploits label semantics to learn a joint semantic embedding space for both inputs and labels. By incorporating model-agnostic task descriptions, UniST can be easily adapted to different semantic typing tasks without introducing task-specific model components. Experimental results show that UniST offers both strong performance and generalizability on entity typing, relation classification, and event typing. Our unified framework also facilitates learning a single model to solve different semantic typing tasks simultaneously, with performance on par with dedicated models trained on individual tasks.

## Acknowledgment

We appreciate the anonymous reviewers for their insightful comments and suggestions. This material is supported in part by the DARPA MCS program under Contract No. N60011924033 with the United States Office Of Naval Research, and by the National Science Foundation of United States Grant IIS 2105329.

## References

* S. Baldini Soares, N. FitzGerald, J. Ling, and T. Kwiatkowski (2019)Matching the blanks: distributional similarity for relation learning. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, Florence, Italy, pp. 2895-2905. External Links: Link Cited by: SS1, SS2.1.
* A. Cetoli (2020)Exploring the zero-shot limit of FewRel. In Proceedings of the 28th International Conference on Computational Linguistics, Barcelona, Spain, pp. 1447-1451. External Links: Link Cited by: SS1, SS2.1.
* C. Chen and C. Li (2021)ZS-BERT: towards zero-shot relation extraction with attribute representation learning. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Online, pp. 3470-3479. External Links: Link Cited by: SS1, SS2.1.
* H. Chen, Q. Ma, Z. Lin, and J. Yan (2021)Hierarchy-aware label semantics matching network for hierarchical text classification. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), Online, pp. 4370-4379. External Links: Link Cited by: SS1, SS2.1.
* M. Chen, H. Zhang, H. Wang, and D. Roth (2020)What are you trying to do? semantic typing of event processes. In Proceedings of the 24th Conference on Computational Natural Language Learning, Online, pp. 531-542. External Links: Link Cited by: SS1, SS2.1.
* Y. Chen, L. Xu, K. Liu, D. Zeng, and J. Zhao (2015)Event extraction via dynamic multi-pooling convolutional neural networks. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), Beijing, China, pp. 167-176. External Links: Link Cited by: SS1, SS2.1.
* E. Choi, O. Levy, Y. Choi, and L. Zettlemoyer (2018)Ultra-fine entity typing. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Melbourne, Australia, pp. 87-96. External Links: Link Cited by: SS1, SS2.1.
* A. D. Cohen, S. Rosenman, and Y. Goldberg (2020)Relation classification as two-way span-prediction. arXiv preprint arXiv:2010.04829. Cited by: SS1, SS2.1.
* H. Dai, D. Du, X. Li, and Y. Song (2019)Improving fine-grained entity typing with entity linking. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), Hong Kong, China, pp. 6210-6215. External Links: Link Cited by: SS1, SS2.1.
* H. Dai, Y. Song, and H. Wang (2021)Ultra-fine entity typing with weak supervision from a masked language model. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), Online, pp. 1790-1799. External Links: Link Cited by: SS1, SS2.1.
* N. Ding, Y. Chen, X. Han, G. Xu, P. Xie, H. Zheng, Z. Liu, J. Li, and H. Kim (2021)Prompt-learning for fine-grained entity typing. arXiv preprint arXiv:2108.10604. Cited by: SS1, SS2.1.
* M. Dong, C. Pan, and Z. Luo (2021)MapRE: an effective semantic mapping approach for low-resource relation extraction. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, Online, pp. 2694-2704. External Links: Link Cited by: SS1, SS2.1.

 

Xinya Du and Claire Cardie. 2020. Event extraction by answering (almost) natural questions. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 671-683, Online. Association for Computational Linguistics.
* Han et al. (2021) Xu Han, Weilin Zhao, Ning Ding, Zhiyuan Liu, and Maosong Sun. 2021. Ptr: Prompt tuning with rules for text classification. _arXiv preprint arXiv:2105.11259_.
* Han et al. (2018) Xu Han, Hao Zhu, Pengfei Yu, Ziyun Wang, Yuan Yao, Zhiyuan Liu, and Maosong Sun. 2018. FewRel: A large-scale supervised few-shot relation classification dataset with state-of-the-art evaluation. In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_, pages 4803-4809, Brussels, Belgium. Association for Computational Linguistics.
* Jin et al. (2019) Hailong Jin, Lei Hou, Juanzi Li, and Tiansi Dong. 2019. Fine-grained entity typing via hierarchical multi graph convolutional networks. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_, pages 4969-4978, Hong Kong, China. Association for Computational Linguistics.
* Joshi et al. (2020) Mandar Joshi, Danqi Chen, Yinhan Liu, Daniel S. Weld, Luke Zettlemoyer, and Omer Levy. 2020. SpanBERT: Improving pre-training by representing and predicting spans. _Transactions of the Association for Computational Linguistics_, 8:64-77.
* Lan et al. (2020) Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2020. Albert: A lite bert for self-supervised learning of language representations. In _International Conference on Learning Representations_.
* Levy et al. (2017) Omer Levy, Minjoon Seo, Eunsol Choi, and Luke Zettlemoyer. 2017. Zero-shot relation extraction via reading comprehension. In _Proceedings of the 21st Conference on Computational Natural Language Learning (CoNLL 2017)_, pages 333-342, Vancouver, Canada. Association for Computational Linguistics.
* Li et al. (2019) Xiaoya Li, Fan Yin, Zijun Sun, Xiayu Li, Arianna Yuan, Duo Chai, Mingxin Zhou, and Jiwei Li. 2019. Entity-relation extraction as multi-turn question answering. In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_, pages 1340-1350, Florence, Italy. Association for Computational Linguistics.
* Liu et al. (2021) Qing Liu, Hongyu Lin, Xinyan Xiao, Xianpei Han, Le Sun, and Hua Wu. 2021. Fine-grained entity typing via label reasoning. In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_, pages 4611-4622, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
* Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Robertra: A robustly optimized bert pretraining approach. _arXiv preprint arXiv:1907.11692_.
* Loshchilov and Hutter (2019) Ilya Loshchilov and Frank Hutter. 2019. Decoupled weight decay regularization. In _International Conference on Learning Representations_.
* Lyu and Chen (2021) Shengfei Lyu and Huanhuan Chen. 2021. Relation classification with entity type restriction. In _Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021_, pages 390-395, Online. Association for Computational Linguistics.
* Onoe et al. (2021) Yasumasa Onoe, Michael Boratko, Andrew McCallum, and Greg Durrett. 2021. Modeling fine-grained entity types with box embeddings. In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_, pages 2051-2064, Online. Association for Computational Linguistics.
* Onoe and Durrett (2019) Yasumasa Onoe and Greg Durrett. 2019. Learning to denoise distantly-labeled data for entity typing. In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_, pages 2407-2417, Minneapolis, Minnesota. Association for Computational Linguistics.
* Onoe and Durrett (2020) Yasumasa Onoe and Greg Durrett. 2020. Interpretable entity representations through large-scale typing. In _Findings of the Association for Computational Linguistics: EMNLP 2020_, pages 612-624, Online. Association for Computational Linguistics.
* Paolini et al. (2021) Giovanni Paolini, Ben Athiwaratkun, Jason Krone, Jie Ma, Alessandro Achille, RISHITA ANUBHAI, Cicero Nogueira dos Santos, Bing Xiang, and Stefano Soatto. 2021. Structured prediction as translation between augmented natural languages. In _International Conference on Learning Representations_.
* Peng et al. (2020) Hao Peng, Tianyu Gao, Xu Han, Yankai Lin, Peng Li, Zhiyuan Liu, Maosong Sun, and Jie Zhou. 2020. Learning from Context or Names? An Empirical Study on Neural Relation Extraction. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 3661-3672, Online. Association for Computational Linguistics.
* Peters et al. (2019) Matthew E. Peters, Mark Neumann, Robert Logan, Roy Schwartz, Vidur Joshi, Sameer Singh, and Noah A. Smith. 2019. Knowledge enhanced contextual word representations. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_, pages 43-54, Hong Kong, China. Association for Computational Linguistics.

 

Meng Qu, Tianyu Gao, Louis-Pascal AC Khonneux, and Jian Tang. 2020. Few-shot relation extraction via bayesian meta-learning on relation graphs. In _International Conference on Machine Learning_.
* Raffel et al. (2020) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. _Journal of Machine Learning Research_, 21(140):1-67.
* Rajpurkar et al. (2016) Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ questions for machine comprehension of text. In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing_, pages 2383-2392, Austin, Texas. Association for Computational Linguistics.
* Sainz et al. (2021) Oscar Sainz, Oier Lopez de Lacalle, Gorka Labaka, Andre Barrena, and Eneko Agirre. 2021. Label verbalization and entailment for effective zero and few-shot relation extraction. In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_, pages 1199-1212, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
* Schick and Schutze (2021) Timo Schick and Hinrich Schutze. 2021. Exploiting cloze-questions for few-shot text classification and natural language inference. In _Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume_, pages 255-269, Online. Association for Computational Linguistics.
* Shen et al. (2021) Jiaming Shen, Wenda Qiu, Yu Meng, Jingbo Shang, Xiang Ren, and Jiawei Han. 2021. TaxoClass: Hierarchical multi-label text classification using only class names. In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 4239-4249, Online. Association for Computational Linguistics.
* Wang et al. (2021a) Ruize Wang, Duyu Tang, Nan Duan, Zhongyu Wei, Xuanjing Huang, Jianshu Ji, Guihong Cao, Daxin Jiang, and Ming Zhou. 2021a. K-Adapter: Infusing Knowledge into Pre-Trained Models with Adapters. In _Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021_, pages 1405-1418, Online. Association for Computational Linguistics.
* Wang et al. (2019) Xiaozhi Wang, Xu Han, Zhiyuan Liu, Maosong Sun, and Peng Li. 2019. Adversarial training for weakly supervised event detection. In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_, pages 998-1008, Minneapolis, Minnesota. Association for Computational Linguistics.
* Wang et al. (2020) Xiaozhi Wang, Ziqi Wang, Xu Han, Wangyi Jiang, Rong Han, Zhiyuan Liu, Juanzi Li, Peng Li, Yankai Lin, and Jie Zhou. 2020. MAVEN: A Massive General Domain Event Detection Dataset. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 1652-1671, Online. Association for Computational Linguistics.
* Wang et al. (2021b) Ziqi Wang, Xiaozhi Wang, Xu Han, Yankai Lin, Lei Hou, Zhiyuan Liu, Peng Li, Juanzi Li, and Jie Zhou. 2021b. CLEVE: Contrastive Pre-training for Event Extraction. In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_, pages 6283-6297, Online. Association for Computational Linguistics.
* Xia et al. (2018) Congying Xia, Chenwei Zhang, Xiaohui Yan, Yi Chang, and Philip Yu. 2018. Zero-shot user intent detection via capsule neural networks. In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_, pages 3090-3099, Brussels, Belgium. Association for Computational Linguistics.
* Xiong et al. (2019) Wenhan Xiong, Jiawei Wu, Deren Lei, Mo Yu, Shiyu Chang, Xiaoxiao Guo, and William Yang Wang. 2019. Imposing label-relational inductive bias for extremely fine-grained entity typing. In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_, pages 773-784, Minneapolis. Association for Computational Linguistics.
* Xu et al. (2020) Dongqin Xu, Junhui Li, Muhua Zhu, Min Zhang, and Guodong Zhou. 2020. Improving AMR parsing with sequence-to-sequence pre-training. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 2501-2511, Online. Association for Computational Linguistics.
* Xu and Barbosa (2018) Peng Xu and Denilson Barbosa. 2018. Neural fine-grained entity type classification with hierarchy-aware loss. In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_, pages 16-25, New Orleans, Louisiana. Association for Computational Linguistics.
* Yamada et al. (2020) Ikuya Yamada, Akari Asai, Hiroyuki Shindo, Hideaki Takeda, and Yuji Matsumoto. 2020. LUKE: Deep contextualized entity representations with entity-aware self-attention. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 6442-6454, Online. Association for Computational Linguistics.
* Yan et al. (2019) Haoran Yan, Xiaolong Jin, Xiangbin Meng, Jiafeng Guo, and Xueqi Cheng. 2019. Event detection with multi-order graph convolution and aggregated attention. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 

9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_, pages 5766-5770, Hong Kong, China. Association for Computational Linguistics.
* Yavuz et al. (2016) Semih Yavuz, Izzeddin Gur, Yu Su, Mudhakar Srivatsa, and Xifeng Yan. 2016. Improving semantic parsing via answer type inference. In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing_, pages 149-159, Austin, Texas. Association for Computational Linguistics.
* Yin et al. (2019) Wenpeng Yin, Jamaal Hay, and Dan Roth. 2019. Benchmarking zero-shot text classification: Datasets, evaluation and entailment approach. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_, pages 3914-3923, Hong Kong, China. Association for Computational Linguistics.
* Zhang et al. (2018) Yuhao Zhang, Peng Qi, and Christopher D. Manning. 2018. Graph convolution over pruned dependency trees improves relation extraction. In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_, pages 2205-2215, Brussels, Belgium. Association for Computational Linguistics.
* Zhang et al. (2017) Yuhao Zhang, Victor Zhong, Danqi Chen, Gabor Angeli, and Christopher D. Manning. 2017. Position-aware attention and supervised data improve slot filling. In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing_, pages 35-45, Copenhagen, Denmark. Association for Computational Linguistics.
* Zhang et al. (2019) Zhengyan Zhang, Xu Han, Zhiyuan Liu, Xin Jiang, Maosong Sun, and Qun Liu. 2019. ERNIE: Enhanced language representation with informative entities. In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_, pages 1441-1451, Florence, Italy. Association for Computational Linguistics.
* Zhong and Chen (2021) Zexuan Zhong and Danqi Chen. 2021. A frustratingly easy approach for entity and relation extraction. In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 50-61, Online. Association for Computational Linguistics.
* Zhou and Chen (2021a) Wenzuan Zhou and Muhao Chen. 2021a. An improved baseline for sentence-level relation extraction. _arXiv preprint arXiv:2102.01373_.
* Zhou and Chen (2021b) Wenzuan Zhou and Muhao Chen. 2021b. Learning from noisy labels for entity-centric information extraction. In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_, pages 5381-5392, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.

## Appendix A Experiment Details

We run all single-task UniSTBASE experiments on NVIDIA RTX 2080Ti GPUs, and all UniSTLARGE and multi-task experiments on NVIDIA RTX A5000 GPUs. UniSTBASE and UniSTLARGE use base and large versions of RoBERTa as encoders with 125M and 355M parameters respectively. We conduct hyperparameter search within the following range:

* learning rate: {3e-6, 5e-6, 1e-5, 2e-5}
* Batch size: {32, 64, 128}
* Number of training epochs: {50, 100, 200, 500, 1000}
* Ranking loss margin \(\gamma\): { 0.1, 0.2, 0.3}

We optimize our models using AdamW (Loshchilov and Hutter, 2019) with linear learning rate decay. The best model checkpoints are selected based on dev set performance. Tab. 8 lists common hyperparameters used across all experiments. All datasets used in our experiments are in English. More details of individual tasks and experiments are provided below.

### Ufet

The UFET dataset is publicly available on its official website 3. Tab. 9 shows the hyperparameters and dev F1 score for UFET experiments.

Footnote 3: [https://www.cs.utexas.edu/~eunsol/html_pages/open_entity.html](https://www.cs.utexas.edu/~eunsol/html_pages/open_entity.html)

Figure 9: Hyperparameters and dev F1 score for UFET experiments.

\begin{table}
\begin{tabular}{l|c c} \hline \hline Name & UniSTBASE & UniSTLARGE \\ \hline Batch size & 64 & 64 \\ Number of training epochs & 1000 & 1000 \\ Dev F1 & 49.2 & 49.5 \\ \hline \hline \end{tabular}
\end{table}
Table 9: Hyperparameters and dev F1 score for UFET experiments.

 



### TACRED

The TACRED dataset we use is licensed by LDC 4. Tab. 10 shows the hyperparameters and dev F1 score for TACRED experiments.

Footnote 4: [https://catalog.ldc.upenn.edu/LDC2018T24](https://catalog.ldc.upenn.edu/LDC2018T24)

### Maven

The MAVEN dataset is publicly available via its official github repository 5. Tab. 11 shows the hyperparameters and dev F1 score for MAVEN experiments.

Footnote 5: [https://github.com/THU-KEG/MAVEN-dataset](https://github.com/THU-KEG/MAVEN-dataset)

### FewRel

The FewRel dataset is publicly available via its official github repository 6. We report the average accuracy of 10 runs on the dev set during evaluation. Tab. 12 shows the hyperparameters for FewRel experiments.

Footnote 6: [https://github.com/thunlp/FewRel](https://github.com/thunlp/FewRel)

### Multi-task Experiments

We conduct multi-task experiments on the combined UFET, TACRED, and MAVEN training sets. We up-sample UFET training set by a factor of 10 for more balanced training. Tab. 13 shows the hyperparameters and dev set F1 for multi-task experiments.

## Appendix B Ethics Considerations

Our experiments are all conducted on openly available and widely used datasets. We do not augment any information to those data in this research, hence this research is not expected to introduce any additional biased information to existing information in those data. However, the model may potentially capture biases reflective of the pre-trained language models and datasets we use for our experiments, in such biases have pre-existed in these pre-trained models or datasets. This is a common problem for models trained on large-scale data, and therefore we suggest conducting a thorough bias analysis before deploying our model in any real-world applications.

\begin{table}
\begin{tabular}{l|r r} \hline \hline Name & UniSTBASE & UniSTLARGE \\ \hline Batch size & 64 & 64 \\ Number of training epochs & 100 & 100 \\ Dev F1 & 73.9 & 75.3 \\ \hline \hline \end{tabular}
\end{table}
Table 10: Hyperparameters and dev F1 score for TAACRED experiments.

\begin{table}
\begin{tabular}{l|r r} \hline \hline Name & UniSTBASE & UniSTLARGE \\ \hline Batch size & 64 & 64 \\ Number of training epochs & 100 & 100 \\ Dev F1 & 68.4 & 68.5 \\ \hline \hline \end{tabular}
\end{table}
Table 11: Hyperparameters and dev F1 score for MAVEN experiments.

\begin{table}
\begin{tabular}{l|r r} \hline \hline Name & UniSTBASE & UniSTLARGE \\ \hline Batch size & 64 & 64 \\ Number of training epochs & 100 & 100 \\ Dev F1 & 73.9 & 75.3 \\ \hline \hline \end{tabular}
\end{table}
Table 10: Hyperparameters and dev F1 score for TAACRED experiments.