SlowGuess/NAACL_2022_OCR huggingface dataset:
As for HTNN, we can see that it achieves the best performance overall. The fact that HTNN outperforms GAT and GCN while GAT-sib and GCN-sib underperform GAT and GCN suggests that the superiority of HTNN does not simply come from adding sibling information into computation. It is the ability to group and integrate bottom-up and top-down information that leads to the superior performance of HTNN.
It is worth noting that for the NEL task, HTNN outperforms Pooling on F1-const but underperforms it on F1-all because HTNN predicts no label for distituent spans. A simple remedy is to combine HTNN and Pooling, using HTNN for constituent
spans and reverting to Pooling for distituent spans. This yields 96.27 F1-all for NEL, surpassing the strongest Pooling baseline.
3.5 SRL Experiments
We extend SRL in probing tasks to the general setting, in which the predicate is given and we conduct argument identification and role classification. Our SRL model is most closely related to the work of He et al. (2018). The difference is that instead of using a separate argument pruning module, we simply prune all the distituent spans and keep all the constituent spans. Given a predicate and a candidate span, we use their biaffine scorer to predict a label, where the label set is the set of semantic roles plus a null label indicating the span not being an argument.
The results of SRL experiments are shown in Table 2. Once again, we show F1 scores for both all the spans (F1-all) and constituent spans only (F1-const). We only show the results of GNN-based models since RvNN-based models have shown their disadvantages in the probing experiments. SentiBERT does not perform well in this task, falling behind even the simple baseline Pooling. GCN and GAT show significant gain against baseline, and GCN is still better than GAT on all the three datasets, especially on the out-of-domain BROWN test dataset. We also find that taking sibling nodes into computation causes opposite effects on GAT and GCN. It does great harm to GAT, similar to the case in the probing experiments, but GCN-sib shows small gains over GCN, which is different from the case in the probing experiments.
Our model HTNN achieves the best performance on all the three datasets. Since SRL is more challenging than the probing tasks, we can see that the
CONLL12	CONLL05 WSJ	CONLL05 BROWN
F1-const	F1-all	F1-const	F1-all	F1-const	F1-all
Pooling	82.36	82.23	82.82	81.90	71.51	70.43
SentiBERT	75.31	75.18	74.52	73.69	66.52	65.51
GAT	85.29	85.16	84.69	83.74	73.32	72.24
GCN	87.91	87.77	88.06	87.08	79.22	78.09
GAT-sib	76.41	76.29	78.34	77.46	62.52	61.58
GCN-sib	88.40	88.27	88.45	87.46	80.03	79.87
HTNN	89.94	89.81	90.77	89.76	82.88	81.68
Wang et al. (2019)†	-	84.21	-	85.23	-	75.36
Fei et al. (2021)*	-	87.35	-	88.81	-	81.27
Table 2: Results of SRL experiments. “†”: reimplemented and reported by Fei et al. (2021). “*”: results reported in the original paper.
SRC	NEL	COREF	CONLL12	WSJ	BROWN
GAT -w/o tag	-0.27	-0.14	-0.46	-7.74	-8.49	-7.36
GCN -w/o tag	-0.37	0.02	-0.09	-1.74	-1.38	-0.76
GAT-sib -w/o tag	-0.82	-0.34	-1.85	-7.09	-9.30	-7.05
GCN-sib -w/o tag	-0.07	0.06	0.02	-2.06	-1.90	-1.11
HTNN -w/o tag	-0.04	-0.07	0.13	-1.73	-2.29	-2.28
HTNN -tag-dim=32	0.07	-0.14	0.22	-0.03	0.00	0.15
Table 3: Influence of nonterminal tags in constituency parse tree. The value in this table is the variation against the results in Table 1 and Table 2.
gap between HTNN and the other models is more prominent. The improvements over the best baseline model GCN-sib are 1.54 on the CONLL-2012 test dataset, 2.32 on the CONLL-2005 WSJ test dataset, 2.85 on the CONLL-2005 BROWN test dataset.
We also show results of two other SRL models that use gold constituency parse trees in Table 2. HTNN outperforms both of them.
Incorporation of Nonterminal Tags We incorporate nonterminal tags into HTNN as described in Section 2.2. Here, we want to explore how nonterminal tags influence HTNN, as well as other models. When we use different embedding dimensions for nonterminal tags, the dimension of the attention pooling module changes accordingly to keep the dimension of span representations unchanged.
The results are shown in Table 3. As we can see, GAT, GCN and HTNN are negatively affected on nearly all the tasks when we remove the incorporation of nonterminal tags. The influence on both GCN and HTNN is quite small on the three probing tasks, compared with the SRL task. There are even improvements for HTNN and GCN-sib when
Figure 3: Performance of different tasks as #layers of HTNN changes.
removing nonterminal tags on NEL and COREF. However, on the SRL task, the performance decreases notably for all the models.
We also conduct experiments of HTNN when we set the dimension of nonterminal tags to 32 and we observe little change in performance.
Number of Layers In each layer of an HTNN, a node exchanges information with its neighbor. Therefore, the more layers the HTNN has, the fur
Figure 4: Performance of different aggregation methods.
ther the information of a node would spread. In theory, for a binary constituency tree with  $n$  leaf nodes, to make each leaf node receive information from arbitrary leaf nodes, the number of HTNN layers should be set to  $n - 1$ . In practice, however, this may be unnecessary and local information exchange may be sufficient. We conduct experiments to analyze how the number of HTNN layers affect the performance on different tasks.
The results are shown in Figure 3. We can see that the tendency is not all the same on the four tasks. As we increase the number of layers, the increase of the F1 score is getting smaller, and in some cases it even declines. Therefore, we regard the number of layers as a hyperparameter which should be tuned on specific tasks.
Aggregation Method We compare three different methods for the aggregation module (Section 2.4):
AVG, which simply takes the average of the representations.
- ATTN, which is the default method we introduce in Section 2.4.
- ATTN-2WAY, similar to ATTN, but does not include the representation from previous layer.
The results are shown in Figure 4. It can be seen that ATTN indeed performs the best.
To get distributed span representations, existing methods are mostly based on simple derivations from word or sub-word representations. Lee et al. (2017) propose the attention-weighted pooling strategy for coreference resolution. Stern et al. (2017) concatenate the sum and the difference of
the span endpoints for parsing. Seo et al. (2019) use the "coherent" endpoint-based representation for question answering. All of these methods are originally designed for specific tasks, but because of their simplicity, they can be easily employed for other tasks. Toshniwal et al. (2020) make a comprehensive survey on these methods and conduct experiments to compare their performance. Some other studies also focus on the representations of sentences or longer texts (Wieting et al., 2015; Conneau et al., 2017; Shen et al., 2018).
Instead of representing an arbitrary, some studies consider composing span representations via syntactic parse trees, which are more related to our work. Le and Zuidema (2014) proposed a recursive model IORNN to produce constituent representations for supervised task. Drozdov et al. (2019) proposed an unsupervised method for constituency parse tree induction, during which span representations are produced as a byproduct. More recently, graph neural networks have been applied with constituency parse trees. Marcheggiani and Titov (2020) use GCN to encode constituent spans in an SRL system. However, they decompose the constituent span representations back to start and end tokens instead of directly using them. Li et al. (2020) use GAT to generate constituent span representations for the sentiment analysis task.
In this paper, we propose hypertree neural networks (HTNN) to generate better representations of constituent spans following constituency parse tree structures. Each node in the HTNN represents a constituent span and each hyperedge represents a local composition structure. Each HTNN layer first computes a representation of each node in a hyperedge using the other nodes from the same hyperedge, and then aggregates the representations of a node from all the connected hyperedges. We empirically compare HTNNs with previous RvNN-based models and GNN-based models. The outstanding performance of HTNNs on both probing and SRL tasks shows the effectiveness of HTNN.
For future work, we plan to tackle two related issues of our approach, namely its reliance on high-quality constituency parses and its inability to represent distituent spans (some of which may nonetheless be important). We will also extend HTNNs to more tasks such as sentiment analysis and document classification, and incorporate HTNNs into
popular pretrained language models as a method to inject syntactic information.
This work was supported by the National Natural Science Foundation of China (61976139) and the Joint Funds of the National Natural Science Foundation of China (U21B2020).
Steven Bird, Ewan Klein, and Edward Loper. 2009. Natural language processing with Python: analyzing text with the natural language toolkit. "O'Reilly Media, Inc."
Xavier Carreras and Lluis Marquez. 2005. Introduction to the CoNLL-2005 shared task: Semantic role labeling. In Proceedings of the Ninth Conference on Computational Natural Language Learning (CoNLL-2005), pages 152–164, Ann Arbor, Michigan. Association for Computational Linguistics.
Noam Chomsky. 1957. Syntactic structures.
Alexis Conneau, Douwe Kiela, Holger Schwenk, Loic Barrault, and Antoine Bordes. 2017. Supervised learning of universal sentence representations from natural language inference data. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 670-680, Copenhagen, Denmark. Association for Computational Linguistics.
Andrew Drozdov, Patrick Verga, Mohit Yadav, Mohit Iyyer, and Andrew McCallum. 2019. Unsupervised latent tree induction with deep inside-outside recursive auto-encoders. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 1129-1141, Minneapolis, Minnesota. Association for Computational Linguistics.
Hao Fei, Shengqiong Wu, Yafeng Ren, Fei Li, and Donghong Ji. 2021. Better combine them together! integrating syntactic constituency and dependency representations for semantic role labeling. In *Findings of the Association for Computational Linguistics: ACL-IJCNLP* 2021, pages 549-559, Online. Association for Computational Linguistics.
Christoph Goller and Andreas Kuchler. 1996. Learning task-dependent distributed representations by backpropagation through structure. In Proceedings of International Conference on Neural Networks (ICNN'96), volume 1, pages 347-352. IEEE.
Luheng He, Kenton Lee, Omer Levy, and Luke Zettlemoyer. 2018. Jointly predicting predicates and arguments in neural semantic role labeling. In Proceedings of the 56th Annual Meeting of the Association for
Computational Linguistics (Volume 2: Short Papers), pages 364-369, Melbourne, Australia. Association for Computational Linguistics.
Liang Huang and David Chiang. 2005. Better k-best parsing. In Proceedings of the Ninth International Workshop on Parsing Technology, pages 53-64, Vancouver, British Columbia. Association for Computational Linguistics.
Mandar Joshi, Danqi Chen, Yinhan Liu, Daniel S. Weld, Luke Zettlemoyer, and Omer Levy. 2020. SpanBERT: Improving pre-training by representing and predicting spans. Transactions of the Association for Computational Linguistics, 8:64-77.
Thomas N Kipf and Max Welling. 2016. Semi-supervised classification with graph convolutional networks. arXiv preprint arXiv:1609.02907.
Dan Klein and Christopher D. Manning. 2001. Parsing and hypergraphs. In Proceedings of the Seventh International Workshop on Parsing Technologies, pages 123-134, Beijing, China.
Yuxuan Lai, Yijia Liu, Yansong Feng, Songfang Huang, and Dongyan Zhao. 2021. Lattice-BERT: Leveraging multi-granularity representations in Chinese pretrained language models. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1716-1731, Online. Association for Computational Linguistics.
Phong Le and Willem Zuidema. 2014. Inside-outside semantics: A framework for neural models of semantic composition. In NIPS 2014 Workshop on Deep Learning and Representation Learning.
Kenton Lee, Luheng He, Mike Lewis, and Luke Zettle-moyer. 2017. End-to-end neural coreference resolution. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 188-197, Copenhagen, Denmark. Association for Computational Linguistics.
Yuncong Li, Cunxiang Yin, and Sheng-hua Zhong. 2020. Sentence constituent-aware aspect-category sentiment analysis with graph attention networks. In CCF International Conference on Natural Language Processing and Chinese Computing, pages 815-827. Springer.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692.
Diego Marcheggiani and Ivan Titov. 2020. Graph convolutions over constituent trees for syntax-aware semantic role labeling. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 3915-3928, Online. Association for Computational Linguistics.
Sameer Pradhan, Alessandro Moschitti, Nianwen Xue, Olga Uryupina, and Yuchen Zhang. 2012. CoNLL-2012 shared task: Modeling multilingual unrestricted coreference in OntoNotes. In Joint Conference on EMNLP and CoNLL - Shared Task, pages 1-40, Jeju Island, Korea. Association for Computational Linguistics.
Minjoon Seo, Jinhyuk Lee, Tom Kwiatkowski, Ankur Parikh, Ali Farhadi, and Hannaneh Hajishirzi. 2019. Real-time open-domain question answering with dense-sparse phrase index. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4430-4441, Florence, Italy. Association for Computational Linguistics.
Dinghan Shen, Guoyin Wang, Wenlin Wang, Martin Renqiang Min, Qinliang Su, Yizhe Zhang, Chunyuan Li, Ricardo Henao, and Lawrence Carin. 2018. Baseline needs more love: On simple word-embedding-based models and associated pooling mechanisms. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 440-450, Melbourne, Australia. Association for Computational Linguistics.
Richard Socher, Cliff Chiung-Yu Lin, Andrew Y Ng, and Christopher D Manning. 2011. Parsing natural scenes and natural language with recursive neural networks. In ICML.
Mitchell Stern, Jacob Andreas, and Dan Klein. 2017. A minimal span-based neural constituency parser. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 818-827, Vancouver, Canada. Association for Computational Linguistics.
Kai Sheng Tai, Richard Socher, and Christopher D. Manning. 2015. Improved semantic representations from tree-structured long short-term memory networks. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 1556–1566, Beijing, China. Association for Computational Linguistics.
Shubham Toshniwal, Haoyue Shi, Bowen Shi, Lingyu Gao, Karen Livescu, and Kevin Gimpel. 2020. A cross-task analysis of text span representations. In Proceedings of the 5th Workshop on Representation Learning for NLP, pages 166-176, Online. Association for Computational Linguistics.
Yufei Wang, Mark Johnson, Stephen Wan, Yifang Sun, and Wei Wang. 2019. How to best use syntax in semantic role labelling. In Proceedings of the 57th Annual Meeting of the Association for Computational
Linguistics, pages 5338-5343, Florence, Italy. Association for Computational Linguistics.
John Wieting, Mohit Bansal, Kevin Gimpel, and Karen Livescu. 2015. Towards universal paraphrastic sentence embeddings. arXiv preprint arXiv:1511.08198.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. 2019. Huggingface's transformers: State-of-the-art natural language processing. arXiv preprint arXiv:1910.03771.
Da Yin, Tao Meng, and Kai-Wei Chang. 2020. SentiBERT: A transferable transformer-based architecture for compositional sentiment semantics. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 3695-3706, Online. Association for Computational Linguistics.
Lang Yu and Allyson Ettinger. 2020. Assessing phrasal representation and composition in transformers. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4896-4907, Online. Association for Computational Linguistics.
Xinsong Zhang, Pengshuai Li, and Hang Li. 2021. AMBERT: A pre-trained language model with multi-grained tokenization. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 421-435, Online. Association for Computational Linguistics.
A.1 Details of Composition
The following is the detailed computation process of equation (2), in which we compose the right child node and the parent node to get the representation of the left child node:
$$
\mathbf {i} = \sigma \left(\mathbf {W} ^ {(i)} \mathbf {h} _ {r} + \mathbf {U} _ {l} ^ {(i)} \mathbf {h} _ {l} + \mathbf {U} _ {p} ^ {(i)} \mathbf {h} _ {p} + \mathbf {b} ^ {(i)}\right)
$$
$$
\mathbf {f} ^ {l} = \sigma \left(\mathbf {W} ^ {(f l)} \mathbf {h} _ {r} + \mathbf {U} _ {l} ^ {(f l)} \mathbf {h} _ {l} + \mathbf {U} _ {p} ^ {(f l)} \mathbf {h} _ {p} + \mathbf {b} ^ {(f)}\right)
$$
$$
\mathbf {f} ^ {p} = \sigma \left(\mathbf {W} ^ {(f p)} \mathbf {h} _ {r} + \mathbf {U} _ {l} ^ {(f p)} \mathbf {h} _ {l} + \mathbf {U} _ {p} ^ {(f p)} \mathbf {h} _ {p} + \mathbf {b} ^ {(f)}\right)
$$
$$
\mathbf {o} = \sigma \left(\mathbf {W} ^ {(o)} \mathbf {h} _ {r} + \mathbf {U} _ {l} ^ {(o)} \mathbf {h} _ {l} + \mathbf {U} _ {p} ^ {(o)} \mathbf {h} _ {p} + \mathbf {b} ^ {(o)}\right)
$$
$$
\mathbf {u} = \mathbf {t a n h} \left(\mathbf {W} ^ {(u)} \mathbf {h} _ {r} + \mathbf {U} _ {l} ^ {(u)} \mathbf {h} _ {l} + \mathbf {U} _ {p} ^ {(u)} \mathbf {h} _ {p} + \mathbf {b} ^ {(u)}\right)
$$
The following is the detailed computation process of equation (3), in which we compose the left child node and the parent node to get the represent
Task	|L|	#Instances (Train / Val. / Test)	#Missing (Train / Val. / Test)
NEL	18	128738 / 20354 / 12586	1860 / 297 / 215
SRC	66	598983 / 83362 / 61716	1508 / 281 / 185
COREF	2	207830 / 26333 / 27800	1156 / 286 / 213
CoNLL-2005	53	235353 / 8549 / (14463 / 2228)	5897 / 225 / (315 / 63)
Table 4: Statistics of the CoNLL-2012 and CoNLL-2005 datasets. The NEL, SRC and COREF are from the CoNLL-2012 dataset. “ $|\mathcal{L}|$ ” is the number of labels. “#Instances” means the total instances in the original datasets. “#Missing” means the instances that contain distituent spans. Note that for the CoNLL-2005 dataset, the test set is composed of WSJ and BROWN, so we show their numbers respectively.
tation of the right child node:
$$
\mathbf {i} = \sigma \left(\mathbf {W} ^ {(i)} \mathbf {h} _ {l} + \mathbf {U} _ {r} ^ {(i)} \mathbf {h} _ {r} + \mathbf {U} _ {p} ^ {(i)} \mathbf {h} _ {p} + \mathbf {b} ^ {(i)}\right)
$$
$$
\mathbf {f} ^ {r} = \sigma \left(\mathbf {W} ^ {(f r)} \mathbf {h} _ {l} + \mathbf {U} _ {r} ^ {(f r)} \mathbf {h} _ {r} + \mathbf {U} _ {p} ^ {(f r)} \mathbf {h} _ {p} + \mathbf {b} ^ {(f)}\right)
$$
$$
\mathbf {f} ^ {p} = \sigma \left(\mathbf {W} ^ {(f p)} \mathbf {h} _ {l} + \mathbf {U} _ {r} ^ {(f p)} \mathbf {h} _ {r} + \mathbf {U} _ {p} ^ {(f p)} \mathbf {h} _ {p} + \mathbf {b} ^ {(f)}\right)
$$
$$
\mathbf {o} = \sigma \left(\mathbf {W} ^ {(o)} \mathbf {h} _ {l} + \mathbf {U} _ {r} ^ {(o)} \mathbf {h} _ {r} + \mathbf {U} _ {p} ^ {(o)} \mathbf {h} _ {p} + \mathbf {b} ^ {(o)}\right)
$$
$$
\mathbf {u} = \operatorname {t a n h} \left(\mathbf {W} ^ {(u)} \mathbf {h} _ {l} + \mathbf {U} _ {r} ^ {(u)} \mathbf {h} _ {r} + \mathbf {U} _ {p} ^ {(u)} \mathbf {h} _ {p} + \mathbf {b} ^ {(u)}\right)
$$
We show the statistics of the datasets in Table 4.