Venue: NAACL 2022

# _EmRel_: Joint Representation of Entities and _Emb_dedded _Rel_ations for Multi-triple Extraction

Benfeng Xu\({}^{1}\), Quan Wang\({}^{2}\), Yajuan Lyu\({}^{3}\), Yabing Shi\({}^{3}\)

**Yong Zhu\({}^{3}\), Jie Gao\({}^{1}\)**   Zhendong Mao\({}^{1}\dagger\)

\({}^{1}\)University of Science and Technology of China, Hefei, China

\({}^{2}\)Beijing University of Posts and Telecommunications, Beijing, China

\({}^{3}\)Baidu Inc., Beijing, China

benfeng@mail.ustc.edu.cn, zdmao@ustc.edu.cn

###### Abstract

Multi-triple extraction is a challenging task due to the existence of informative inter-triple correlations, and consequently rich interactions across the constituent entities and relations. While existing works only explore entity representations, we propose to explicitly introduce _relation_ representation, jointly represent it with entities, and novelly align them to identify valid triples. We perform comprehensive experiments1 on document-level relation extraction and joint entity and relation extraction along with ablations to demonstrate the advantage of the proposed method.

Footnote 1: The code will be available at [https://github.com/BenfengKu/emrel](https://github.com/BenfengKu/emrel)

+
Footnote †: This work was supported by NSF grant DMS-0854769 and DMS-0854769.

+
Footnote †: This work was supported by NSF grant DMS-0854769.

+
Footnote †: This work was supported by NSF grant DMS-0854769.

## 1 Introduction

Relation extraction aims at discovering structured knowledge in the form of <_subject-relation-object>_ triples from plain text. It is an essential task towards constructing knowledge bases. Although a lot of efforts have been made in building advanced relation extraction systems, it is still a challenging problem under certain practical scenarios where multiple entities and relations are involved, _e.g._, document-level relation extraction Yao et al. (2019) and joint entity and relation extraction Riedel et al. (2010); Gardent et al. (2017).

Existing works mostly take the _entity perspective_ that focuses on exploring cross-entity interactions Xu et al. (2021); Zeng et al. (2020). They either treat relations as atomic labels specified in a final classifier Xu et al. (2021); Zeng et al. (2020); Wang et al. (2020), or simply search subjects and objects for each individual relationWei et al. (2020); Zheng et al. (2021). However, as an essential component, relations also interact with entities and context, which jointly exhibit informative inter-triple correlations. e.g., the two relations _capital of_ and _located at_ often co-occur between the same pair of entities but with different probabilities conditioned on specific contextual clues. As a consequence, the capability to model and make full use of rich interactions across relations, entities, and context is crucial for the task.

In this paper, we advocate a novel _joint triple perspective_ for relation extraction (see Figure 1 for illustration). Different from previous works that only seek to represent entities, we propose _EmRel_ that creates, refines and leverages the _Em_bedded representations of _Rel_ations. Specifically, we first explicitly create relation representations as embedded vectors; then refine these relation (as well as entity) representations by modeling rich relationship-context interactions via an attention-based fusion module; and finally identify valid triples by aligning the representation of entities and relations in a joint space, with a novel alignment function based on Tucker Decomposition. This joint triple perspective actually considers entities along with relations as components of a small, nontext knowledge graph, and completes this graph by aligning and reasoning to extract multiple valid triples.

Figure 1: Different formulations for multi-triple extraction. 1) _entity perspective_ constructs only entity representation and feed them into a relation-specific classifier. 2) _joint triple perspective_ constructs both entity representation and relation representation to model comprehensive correlations across all components.



To demonstrate the advantage of the proposed _EmRel_ framework, we conduct experiments on two specific scenarios of multi-triple extraction: document-level relation extraction(RE) and joint entity and relation extraction, with three popular datasets including DocRED Yao et al. (2019), NYT Riedel et al. (2010) and WebNLG Gardent et al. (2017). The results verify the superiority of the joint triple perspective over the traditional entity perspective in multi-triple extraction. We also provide further ablation study to show the effectiveness of our fusion module and alignment function.

## 2 Related Works

Document-level Relation ExtractionExtracting multi-triples from document-level text has recently aroused increasing interests Yao et al. (2019). Existing methods take the entity perspective that proposes various techniques to model entity interactions. Nan et al. (2020) and Zeng et al. (2020) construct an entity graph, and perform graph-level reasoning to refine the entity node representations. Xu et al. (2021) introduces entity structure as useful prior, and models such information within the transformer attention layer. Zhang et al. (2021) utilizes a segmentation network to model the interdependency among entity pairs. Therefore, inter-triple correlations are only captured at the entity level while relation-based ones are neglected.

Joint Entity and Relation ExtractionJoint entity and relation extraction is a popular task that extracts multi-triples along with their entities. Existing works can be concluded into two frameworks: one that searches subjects and objects for each individual relation Liu et al. (2020); Wang et al. (2020); Wei et al. (2020), and the other that directly see each word as a candidate entity and assign them with relation labels Gupta et al. (2016); Zheng et al. (2021). Both formulations do not explicitly include inter-triple correlations. Very recently, Wu and Shi (2021) propose to model the interdependencies between entity labels and relation labels. However, such correlation is constrained within a specific word position, while _EmRel_ exploits the global correlations among all triples and across entities, context, and relations. Li et al. (2021) introduces a translation-based function that predicts object from subject and relation, while _EmRel_ proposes a more expressive alignment function that models the ternary interaction of subject, relation and object.

Relation EmbeddingThere is one specific previous work Chen and Badlani (2020) that also considers modeling relation representations. CRE uses the sentence representation as relation embeddings, and scores them with the entity embeddings trained along with the knowledge base. This raises significant differences with _EmRel_ in both 1) technical design, _EmRel_ constructs and models independent relation representations that are not inherited from specific context, and 2) task settings, CRE requires all entities be aligned to an existing knowledge base to train their embedding.

## 3 Methodology

### Task Formulation

We first formulate the multi-triple extraction task to suitably contain both document-level RE and joint entity and relation extraction. Given a sequence of text \(\{w_{i}\}\), a set of candidate entities \(E=\{e_{i}\}\) and the pre-defined relations \(R=\{r_{i}\}\), the candidate triples can be derived as:

\[T=\{<s,r,o>|s,o\in\{e_{i}\},r\in\{r_{i}\}\} \tag{1}\]

the target is to assign each \(t\) in \(T\) a binary label that discriminates its validity. The candidate entities can either be pre-annotated, as in document-level relation extraction, or be jointly recognized, as in joint entity and relation extraction. In the latter scenario, one prevailing solution is to directly see each word as a candidate entity, such as tagging-based methods Wang et al. (2020) or table filling methods Gupta et al. (2016). Here we follow Wang et al. (2020) as our baseline, and thus formulate both tasks under a unified framework that extracts multi-triples from a given candidate entity set.

### EmRel

_EmRel_ consists of three modules: _Representation Construction_ for both entities and relations, _Representation Fusion_ that captures multi-triple correlations by modeling the informative interactions across entities, context and relations, and _Representation Alignment_ that leverages these representations to extract triples by aligning their ternary structures (see Figure 2 for illustration).

Representation ConstructionThe entity representation is constructed similar to existing practices. We employ a text encoder, _e.g._, pretrained language models like BERT Devlin et al. (2019), and obtain 

the output from its last layer on corresponding position as the contextualized representation:

\[(h_{1},h_{2},...h_{n})=\texttt{encoder}(w_{1},w_{2},...w_{n}) \tag{2}\]

which we denote as \(\mathbf{H}\in\mathbb{R}^{|\{w_{i}\}|\times d_{h_{i}}}\). Then we construct each entity representation \(\mathbf{e}_{i}\in\mathbb{R}^{d_{o}}\) by applying a pooling operation on its corresponding mention positions, and further map it into respective subject and object representation \(\mathbf{e}_{i}^{s}\), \(\mathbf{e}_{i}^{o}\). We thus denote all extracted entity representations as \(\mathbf{E}^{s},\mathbf{E}^{o}\in\mathbb{R}^{|E|\times d_{e}}\).

We embed the target relations \(R\) into an embedding matrix \(\mathbf{R}\in\mathbb{R}^{|R|\times d_{r}}\), where each row \(\mathbf{R}_{i;}\) represents a vectorized relation \(r_{i}\). This matrix is maintained as part of the model parameter and trained accordingly.

Representation FusionIn order to jointly represent entities and relations in a shared knowledge representation space, we fuse them to be aware of each other. We adopt the attention network Bahdanau et al. (2015) to model inter-component interactions, which has proven to be very successful in modeling rich interactions across contexts Yu et al. (2018) or modalities Lu et al. (2016). Specifically, we employ the canonical multi-head attention (MHA) network Vaswani et al. (2017). Given the target representation \(\mathbf{X}_{Q}\) and the source representation \(\mathbf{X}_{S}\), each head of MHA operates them as:

\[\begin{split}\widehat{\mathbf{X}}_{Q}&=\texttt{Att} (\mathbf{X}_{Q}W^{Q},\mathbf{X}_{S}W^{K},\mathbf{X}_{S}W^{V})\\ &=\texttt{softmax}(\frac{(\mathbf{X}_{Q}W^{Q})(\mathbf{X}_{S}W^{ K})^{T}}{\sqrt{d_{k}}})\mathbf{X}_{S}W^{V}\end{split} \tag{3}\]

where \(\widehat{\mathbf{X}}_{Q}\) is the updated representation of \(\mathbf{X}_{Q}\) w.r.t. \(\mathbf{X}_{S}\), all heads operate in parallel and will be concatenated together.

In _EmRel_, to exploit the comprehensive intersections across all components, we first construct entity/context-aware relation representation:

\[\begin{split}\widehat{\mathbf{R}}^{s}=&\texttt{ Att}_{szr}(\mathbf{R}W^{Q},\mathbf{E}^{*}W^{K},\mathbf{E}^{*}W^{V})\\ \widehat{\mathbf{R}}^{o}=&\texttt{Att}_{bzr}(\mathbf{ R}W^{Q},\mathbf{E}^{*}W^{K},\mathbf{E}^{*}W^{V})\\ \widehat{\mathbf{R}}^{c}=&\texttt{Att}_{czr}( \mathbf{R}W^{Q},\mathbf{H}W^{K},\mathbf{H}W^{V})\end{split} \tag{4}\]

which are then aggregated together using layer normalization:

\[\widehat{\mathbf{R}}=\texttt{LayerNorm}(\widehat{\mathbf{R}}^{s}+\widehat{ \mathbf{R}}^{o}+\widehat{\mathbf{R}}^{c}) \tag{5}\]

we symmetrically construct relation-aware entity representation:

\[\begin{array}{l}\widehat{\mathbf{E}}^{s}=&\texttt{Attr}_{rzs}(\mathbf{ E}^{s}W^{Q},\mathbf{R}W^{K},\mathbf{R}W^{V})\\ \widehat{\mathbf{E}}^{o}=&\texttt{Att}_{rz0}(\mathbf{E}^{o}W^{Q},\mathbf{R} W^{K},\mathbf{R}W^{Y})\end{array} \tag{6}\]

\(s\), \(o\), \(c\) are abbreviations for \(subject\), \(object\) and \(context\). Each attention module is wrapped with residual connection, feedforward layer, layer normalization, and is instantiated with different parameters of \(W_{Q}\), \(W_{K}\), \(W_{V}\) to model distinguished attending patterns. The outputs of fusion module are refined representations \(\widetilde{\mathbf{R}}\), \(\widehat{\mathbf{E}}^{s}\), \(\widehat{\mathbf{E}}^{o}\) for relations, subjects and objects.

Figure 2: The overall framework of EmRel. It explicitly introduces relations embedding, and jointly represents it with entities to identify all valid triples.

 

Representation Alignment_EmRel_ extracts triples by aligning their ternary components \(\widehat{\mathbf{R}}\), \(\widehat{\mathbf{E}}^{s}\), and \(\widehat{\mathbf{E}}^{o}\). In order to fully leverage their expressiveness, we propose factorization-based alignment using Tucker decomposition Tucker et al. (1964). We introduce a core tensor \(\mathcal{Z}\in\mathbb{R}^{d_{e}*d_{v}*d_{v}}\), and the validity for each \(<s_{i},r_{k},o_{j}>\) is scored as:

\[\phi(s_{i},r_{k},o_{j})=\sigma(\mathcal{Z}\times_{1}\hat{\mathbf{e}}_{i}^{s} \times_{2}\hat{\mathbf{r}}_{k}\times_{3}\hat{\mathbf{e}}_{j}^{o}+b_{k}) \tag{7}\]

where \(\hat{\mathbf{e}}_{i}^{s}=\widehat{\mathbf{E}}_{i,}^{s}\), \(\hat{\mathbf{r}}_{k}=\widehat{\mathbf{R}}_{k,;}\), \(\hat{\mathbf{e}}_{i}^{o}=\widehat{\mathbf{E}}_{j,}^{o}\), and \(\times_{n}\) indicates tensor product along the \(n\)-th mode, \(\sigma\) denotes sigmoid function. We compute \(\phi\) for all triples in parallel using batched tensor product, and train them using cross-entropy loss:

\[L= \sum_{<s_{i},r_{k},o_{j}>}^{T}[-1^{True}(<s_{i},r_{k},o_{j}>)\log \phi(s_{i},r_{k},o_{j}) \tag{8}\] \[-1^{False}(<s_{i},r_{k},o_{j}>)\log(1-\phi(s_{i},r_{k},o_{j}))]\]

where \(\mathbbm{1}\) indicates the ground truth validity.

## 4 Experiments

### Main Results

We conduct comprehensive experiments on document-level RE dataset DocRED Yao et al. (2019) and joint entity and relation extraction dataset NYT Riedel et al. (2010) and WebNLG Gardent et al. (2017). We use BERTBase-Cased Devlin et al. (2019) as the context encoder and we also provide results with RoBERTa Liu et al. (2019) on DocRED. The dimension of embedded relation representation is set as 768 for Base models, 1024 for Large models on DocRED, and 128 on NYT / WebNLG. The number of attention heads in the fusion module is simply set as 4. We provide our reproduced results of TPLinker Wang et al. (2020) and the baseline system of Xu et al. (2021). Both are competitive baselines based on the entity perspective, and are directly comparable with _EmRel_. Further specifics about these datasets and implementation details can be referred to Appendix.

The results (see Table 1 and Table 2) show that _EmRel_ universally outperforms its baselines on all datasets. Respectively, **+0.3 F1** for NYT\({}^{*}\), **+0.8 F1** for WebNLG\({}^{*}\), **+1.0 F1** for NYT and **+1.1 F1** for WebNLG. On DocRED, _EmRel_ improves the baseline by **+0.95 Dev F1**, **+1.47 Test F1**, and also outperforms several previous studies including BERT-TS Wang et al. (2019), CorefBERT Ye et al. (2020), LSR Nan et al. (2020), and SSAN Xu et al. (2021). On stronger backbone encoders like RoBERTa, similar improvements over baselines can also be observed.

\begin{table}
\begin{tabular}{l c c c c c c c c c c c c} \hline \hline \multirow{2}{*}{**Method**} & \multicolumn{3}{c}{**NVT\({}^{*}\)**} & \multicolumn{3}{c}{**WebNLG\({}^{*}\)**} & \multicolumn{3}{c}{**NYT**} & \multicolumn{3}{c}{**WebNLG**} \\  & **Prec.** & **Rec.** & **F1** & **Prec.** & **Rec.** & **F1** & **Prec.** & **Rec.** & **F1** & **Prec.** & **Rec.** & **F1** \\ \hline CasRel Wei et al. (2020) & 89.7 & 89.5 & 89.6 & **93.4** & 90.1 & 91.8 & - & - & - & - & - & - \\ TPLinker Wang et al. (2020) & 91.3 & **92.5** & 91.9 & 91.8 & 92.0 & 91.9 & 91.4 & 92.6 & 92.0 & 88.9 & 84.5 & 86.7 \\ \hline Baseline\({}^{\dagger}\) & 91.1 & **92.5** & 91.8 & 91.4 & 92.7 & 92.1 & 91.2 & 92.1 & 91.6 & 88.7 & 86.5 & 87.6 \\ EmRel & **91.7** & **92.5** & **92.1** & 92.7 & **93.0** & **92.9** & **92.6** & **92.7** & **92.6** & **90.2** & **87.4** & **88.7** \\ \hline \hline \end{tabular}
\end{table}
Table 1: Results on NYT and WebNLG. \({}^{*}\) denotes task settings that only annotate the last word. \({}^{\dagger}\) denotes our reproduced results of Wang et al. (2020) as the baseline. Best results in **bold**.

\begin{table}
\begin{tabular}{l c c c} \hline \hline \multirow{2}{*}{**Method**} & \multicolumn{2}{c}{**Dev**} & \multicolumn{2}{c}{**Test**} \\  & **IgnF1** & **F1** & **IgnF1** & **F1** \\ \hline BenRel & 57.23\({}_{\pm 0.15}\) & 59.30\({}_{\pm 0.10}\) & 57.27 & 59.66 \\ – Fusion & 57.02\({}_{\pm 0.20}\) & 59.12\({}_{\pm 0.19}\) & 56.66 & 58.92 \\ – Alignment & 56.45\({}_{\pm 0.47}\) & 58.56\({}_{\pm 0.44}\) & 55.84 & 58.15 \\ \hline \hline \end{tabular}
\end{table}
Table 3: Ablation results on _EmRel_ modules.

 



### Ablation Studies

On EmRel ModulesWe first varify the design of _EmRel_ modules. Table 3 shows that both fusion and alignment module contribute to the improvements. We also observe that _EmRel_ has more robust performance across multiple runs. This can be attributed to our alignment function, which, once removed, would result in an increased standard deviation from \(\pm 0.20\) to \(\pm 0.47\).

On the Dimensionality of Relation RepresentationsWe investigate the effects of choices for \(d_{r}\) in Fig 3. First of all, the advantage of _EmRel_ is general across variant choices comparing to the baseline. As we gradually set a higher \(d_{r}\) from 64 to 1024, we get improved performance for its stronger expressive capability. While we further increase \(d_{r}\) to 2048, the performance starting to degrades, which might attribute to overfitting. Overall, the optimal dimension lies within [512; 2048], which is quite robust and also computationally acceptable.

## 5 Conclusion

In this paper, we propose _EmRel_ for multi-triple extraction. Distinguished from existing works, _EmRel_ explicitly creates, refines, and leverages the embedded representation of relations. Notably, we design a novel alignment function that discriminates triple validity by aligning its components in a joint representation space. We conduct experiments on both document-level relation extraction and joint entity and relation extraction, to demonstrate the advantage of _EmRel_ over its baselines.

_EmRel_ also provides a new joint triple perspective, where multi-triple extraction is formulated as completion of a small, context-dependent knowledge graph, with candidate entities and relations as its components. In the future, we think more intricate techniques _e.g_., graph-based reasoning, can be explored following such formulation.

## Acknowledgements

We thank all anonymous reviewers for their valuable comments. This work is supported by the National Key Research and Development Program of China No.2020AAA0109400, the National Natural Science Foundation of China under grants No.61876223, No.U19A2057.

## References

* Bahdanau et al. (2015) Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In _ICLR 2015 : International Conference on Learning Representations 2015_.
* Chen and Badlani (2020) Xiaoyu Chen and Rohan Badlani. 2020. Relation extraction with contextualized relation embedding (CRE). In _Proceedings of Deep Learning Inside Out (DeoLIO): The First Workshop on Knowledge Extraction and Integration for Deep Learning Architectures_, pages 11-19, Online. Association for Computational Linguistics.
* Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_, pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
* Gardent et al. (2017) Claire Gardent, Anastasia Shimorina, Shashi Narayan, and Laura Perez-Beltrachini. 2017. Creating training corpora for nlg micro-planning. In _55th annual meeting of the Association for Computational Linguistics (ACL)_.
* Gupta et al. (2016) Pankaj Gupta, Hinrich Schutze, and Bernt Andrassy. 2016. Table filling multi-task recurrent neural network for joint entity and relation extraction. In _Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers_, pages 2537-2547, Osaka, Japan. The COLING 2016 Organizing Committee.
* Li et al. (2021) Xianming Li, Xiaotian Luo, Chenghao Dong, Daichuan Yang, Beidi Luan, and Zhen He. 2021. TDEER: An efficient translating decoding schema for joint extraction of entities and relations. In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_, pages 8055-8064, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
* Liu et al. (2020) Jie Liu, Shaowei Chen, Bingquan Wang, Jiaxin Zhang, Na Li, and Tong Xu. 2020. Attention as relation: Learning supervised multi-head self-attention for relation extraction. In _Proceedings of the Twenty-Ninth

Figure 3: Ablation on dimensions of relations.



International Joint Conference on Artificial Intelligence, IJCAI-20_, pages 3787-3793. International Joint Conferences on Artificial Intelligence Organization. Main track.
* Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. _arXiv preprint arXiv:1907.11692_.
* Lu et al. (2016) Jiasen Lu, Jianwei Yang, Dhruv Batra, and Devi Parikh. 2016. Hierarchical question-image co-attention for visual question answering. _Advances in neural information processing systems_, 29:289-297.
* Nan et al. (2020) Guoshun Nan, Zhijiang Guo, Ivan Sekulic, and Wei Lu. 2020. Reasoning with latent structure refinement for document-level relation extraction. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 1546-1557, Online. Association for Computational Linguistics.
* Riedel et al. (2010) Sebastian Riedel, Limin Yao, and Andrew McCallum. 2010. Modeling relations and their mentions without labeled text. In _Joint European Conference on Machine Learning and Knowledge Discovery in Databases_, pages 148-163. Springer.
* Tucker et al. (1964) Ledyard R Tucker et al. 1964. The extension of factor analysis to three-dimensional matrices. _Contributions to mathematical psychology_, 110119.
* Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In _Advances in Neural Information Processing Systems_, volume 30. Curran Associates, Inc.
* Wang et al. (2019) Hong Wang, Christfried Focke, Rob Sylvester, Nilesh Mishra, and William Wang. 2019. Fine-tune bert for docred with two-step process. _arXiv preprint arXiv:1909.11898_.
* Wang et al. (2020) Yucheng Wang, Bowen Yu, Yueyang Zhang, Tingwen Liu, Hongsong Zhu, and Limin Sun. 2020. TPLinker: Single-stage joint extraction of entities and relations through token pair linking. In _Proceedings of the 28th International Conference on Computational Linguistics_, pages 1572-1582, Barcelona, Spain (Online). International Committee on Computational Linguistics.
* Wei et al. (2020) Zhepei Wei, Jianlin Su, Yue Wang, Yuan Tian, and Yi Chang. 2020. A novel cascade binary tagging framework for relational triple extraction. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 1476-1488, Online. Association for Computational Linguistics.
* Wu and Shi (2021) Hui Wu and Xiaodong Shi. 2021. Synchronous dual network with cross-type attention for joint entity and relation extraction. In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_, pages 2769-2779, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
* Xu et al. (2021) Benfeng Xu, Quan Wang, Yajuan Lyu, Yong Zhu, and Zhendong Mao. 2021. Entity structure within and throughout: Modeling mention dependencies for document-level relation extraction. _Proceedings of the AAAI Conference on Artificial Intelligence_, 35(16):14149-14157.
* Yao et al. (2019) Yuan Yao, Deming Ye, Peng Li, Xu Han, Yankai Lin, Zhenghao Liu, Zhiyuan Liu, Lixin Huang, Jie Zhou, and Maosong Sun. 2019. DocRED: A large-scale document-level relation extraction dataset. In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_, pages 764-777, Florence, Italy. Association for Computational Linguistics.
* Ye et al. (2020) Deming Ye, Yankai Lin, Jiaju Du, Zhenghao Liu, Peng Li, Maosong Sun, and Zhiyuan Liu. 2020. Coreferential Reasoning Learning for Language Representation. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 7170-7186, Online. Association for Computational Linguistics.
* Yu et al. (2018) Adams Wei Yu, David Dohan, Quoc Le, Thang Luong, Rui Zhao, and Kai Chen. 2018. Fast and accurate reading comprehension by combining self-attention and convolution. In _International Conference on Learning Representations_.
* Zeng et al. (2020) Shuang Zeng, Runxin Xu, Baobao Chang, and Lei Li. 2020. Double graph based reasoning for document-level relation extraction. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 1630-1640, Online. Association for Computational Linguistics.
* Zhang et al. (2021) Ningyu Zhang, Xiang Chen, Xin Xie, Shumin Deng, Chuanqi Tan, Mosha Chen, Fei Huang, Luo Si, and Huajun Chen. 2021. Document-level relation extraction as semantic segmentation. In _Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI-21_, pages 3999-4006. International Joint Conferences on Artificial Intelligence Organization. Main Track.
* Zheng et al. (2019) Heliang Zheng, Jianlong Fu, Zheng-Jun Zha, and Jiebo Luo. 2019. Learning deep bilinear transformation for fine-grained image representation. In _Advances in Neural Information Processing Systems_, volume 32. Curran Associates, Inc.
* Zheng et al. (2021) Hengyi Zheng, Rui Wen, Xi Chen, Yifan Yang, Yunyan Zhang, Ziheng Zhang, Ningyu Zhang, Bin Qin, Xu Ming, and Yefeng Zheng. 2021. PRGC: Potential relation and global correspondence based joint relational triple extraction. In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_, pages 6225-6235, Online. Association for Computational Linguistics 



## Appendix A Benchmarks

We introduce the benchmarks used in this work. Table 4 gives their detailed statistics. **DocRED** is constructed from Wikipedia document. It provides comprehensive human annotations for entity mentions, entity types, relational triples, along with their supporting evidences. Each document is a semantically integrate unit that centers in one concept (the title of the wiki page), resulting multiple triples with rich correlations. **NYT** is constructed from New York Times news articles and annotated through distant supervision. **WebNLG** is originally created for natural language generation task, and the sentences are written by humans to cover given triples. Both datasets have the other version denoted as NYT\({}^{*}\) and WebNLG\({}^{*}\). The texts in NYT and WebNLG are much shorter than DocRED documents. These two datasets also feature in multiple triples. In this paper, we solve all three datasets under a unified multi-triple extraction formulation with _EmRel_.

## Appendix B Implementation Details

To provide comparable results, we set hyper-parameters following previous works Wang et al. (2020); Xu et al. (2021). On NYT / WebNLG, we set learning rate as 5e-5, batch size as 24 / 6, and epoch as 100, as each word is seen as a candidate entity, we directly take the word representation as entity representation. On DocRED, we set learning rate as 3e-5, batch size as 4, and search epochs in {40, 60, 80, 100}. Each document is truncated by 512 sequence length. Entity representation is constructed by pooling from all its mention positions. To produce more robust results, we further perform multiple searches using 5 different seeds, resulting a grid search on both epochs and random seeds. The mean and standard deviation results across different seed are reported on development set. All experiments are conducted on a single NIVDIA V100 or A100 GPU machine.

## Appendix C Grouped Alignment

The WebNLG dataset has up to 216 relations, which requires increased computational cost. Inspired by Zheng et al. (2019), we split the alignment tensors into \(N\) groups across its dimensions to reduce the computational overhead, and re-write Eq. 7 as:

\[\phi(s_{i},p_{k},o_{j})=\sum_{n=1}^{N}\mathcal{Z}^{n}\times_{1}\hat{\mathbf{ e}}^{s,n}_{i}\times_{2}\hat{\mathbf{r}}^{n}_{k}\times_{3}\hat{\mathbf{e}}^{ o,n}_{j}+b_{k} \tag{9}\]

\[\begin{split}\hat{\mathbf{e}}^{s,n}_{i}=&\widehat{ \mathbf{E}}^{s}_{i,[(n-1)\frac{d_{c}}{N}\cdot n\frac{d_{c}}{N}]}\\ \hat{\mathbf{r}}^{n}_{k}=&\widehat{\mathbf{R}}_{k,[( n-1)\frac{d_{v}}{N}\cdot n\frac{d_{v}}{N}]}\\ \hat{\mathbf{e}}^{o,n}_{i}=&\widehat{\mathbf{E}}^{ o}_{j,[(n-1)\frac{d_{v}}{N}\cdot n\frac{d_{v}}{N}]}\end{split} \tag{10}\]

We set group \(N\) to 4 for WebNLG, and 1 for other datasets (that is, without further splitting).

\begin{table}
\begin{tabular}{l c c c c c c c c} \hline \hline Dataset & No. of Instances w.r.t. Split & Entities (Avg.) & Relations & \multicolumn{4}{c}{No. of Instances w.r.t. Multi-triples} \\ \cline{2-10}  & Train & Dev & Test & & \(N=1\) & \(1<N<=5\) & \(5<N<=25\) & \(N>25\) \\ \hline DocRED & 3053 & 1000 & 1000 & 19.5 & 96 & 48 & 561 & 3171 & 234 \\ NYT\({}^{*}\) & 56195 & 4999 & 5000 & 2.15 & 24 & 43397 & 22207 & 590 & NA \\ WebNLG\({}^{*}\) & 5019 & 500 & 703 & 3.15 & 171 & 2189 & 3969 & 64 & NA \\ NYT & 56196 & 5000 & 5000 & 2.16 & 24 & 43358 & 22237 & 601 & NA \\ WebNLG & 5019 & 500 & 703 & 3.26 & 216 & 2277 & 3862 & 83 & NA \\ \hline \hline \end{tabular}
\end{table}
Table 4: Statistics of used datasets. \({}^{*}\) denotes task settings that only annotate the last word. \(N\) denotes the number of valid triples within an instance. We can see that these selected benchmarks all involve multiple triples, thus pose significant challenge for relation extraction systems.