SlowGuess/NAACL_2022_OCR huggingface dataset:
Hao Zhou $^{\diamond}$ , Gongshen Liu $^{\diamond}$ , Kewei  $\mathbf{T}\mathbf{u}^{\dagger *}$
$\diamond$  School of Electronic Information and Electrical Engineering, Shanghai Jiao Tong University  
† School of Information Science and Technology, ShanghaiTech University  
Shanghai Engineering Research Center of Intelligent Vision and Imaging
{zhou1998,lgshen}@sjtu.edu.cn tukw@shanghaitech.edu.cn
Many natural language processing tasks involve text spans and thus high-quality span representations are needed to enhance neural approaches to these tasks. Most existing methods of span representation are based on simple derivations (such as max-pooling) from word representations and do not utilize compositional structures of natural language. In this paper, we aim to improve representations of constituent spans using a novel hypertree neural networks (HTNN) that is structured with constituency parse trees. Each node in the HTNN represents a constituent of the input sentence and each hyperedge represents a composition of smaller child constituents into a larger parent constituent. In each update iteration of the HTNN, the representation of each constituent is computed based on all the hyperedges connected to it, thus incorporating both bottom-up and top-down compositional information. We conduct comprehensive experiments to evaluate HTNNs against other span representation models and the results show the effectiveness of HTNN.
Distributed span representations are useful in various natural language processing tasks such as question answering (Seo et al., 2019), coreference resolution (Lee et al., 2017), sentiment classification (Yin et al., 2020) and semantic role labeling (He et al., 2018). Since spans can have arbitrary length, to get fix-dimensional span representations, existing methods are mostly based on simple derivations from word or sub-word representations. These methods either apply some form of pooling over all the words within the target span or utilize only the boundary words of the target span to compute the span representation. There have also been efforts
(a)
Figure 1: An example binarized constituency parse tree (a) and its corresponding hypertree (b). For each hyperedge, P/L/R denote the parent, left-child, right-child constituent spans respectively.
to improve word representation learning using information from text spans, which may at the same time improve span representations built on top of word representations (Joshi et al., 2020). Further, some previous methods introduce short text spans (phrases or n-grams) as additional inputs to BERT and thus learn contextual representations for such spans along with word representations (Lai et al., 2021; Zhang et al., 2021).
However, it is known that natural language text has underlying compositional structure, which can often be represented with constituency parse trees (Chomsky, 1957). What is missing from the aforementioned span representation methods is the utilization of compositional structures both within and outside text spans. Specifically, the recursive compositional structure inside a text span suggests more structured bottom-up computation of the span representation from word representations; the compositional structure outside a text span specifies how the span joins its sibling spans to form a larger span, implying top-down computation of the span representation from its parent and siblings<sup>1</sup>. By
* Kewei Tu is the corresponding author. This work was conducted when Hao Zhou was a visiting student at Shanghai Tech University.
This is only possible for constituent spans and one has to revert to aforementioned methods for distituent spans. However, we note that most NLP tasks concern constituents more
Human Language Technologies, pages 1682 - 1692
taking into account such compositional structures, we could obtain better span representations and avoid problems with existing methods such as insufficient modeling of meaning composition (Yu and Ettinger, 2020).
There are several existing methods of incorporating constituency tree structures in neural networks that can be used to produce constituent span representations. These methods fall into two main categories: recursive neural networks (RvNNs) (Goller and Kuchler, 1996) and graph neural networks (GNNs). RvNN-based models recursively compose span representations from their sub-spans following a constituency tree structure (Socher et al., 2011; Tai et al., 2015). They hence only model bottom-up composition and miss information from parent and sibling spans in their span representations. Some extensions of RvNNs (Le and Zuidema, 2014; Drozdov et al., 2019) try to tackle this problem by additionally modeling top-down (de-)composition in a similar recursive fashion. However, they separate the representations computed from the directions and disallow them to directly interact with (and potentially disambiguate) each other. GNN-based models, such as graph convolutional networks (GCN) (Kipf and Welling, 2016) and graph attention networks (Velicković et al., 2017), have been applied based on constituency tree structures to compute word and span representations (Marcheggiani and Titov, 2020; Li et al., 2020). One important flaw of GNN-based methods is that a GNN represents each composition with multiple edges that become mixed up with edges from other compositions during GNN updates, thus losing critical information of the compositional structure. More specifically, a GNN does not adequately formulate and differentiate bottom-up computation from all the child spans versus top-down computation from the parent and sibling spans. In addition, most GNN-based methods do not model relations between sibling spans.
To overcome the above drawbacks, we propose hypertree neural networks (HTNN) to improve constituent span representations. Specifically, we view a constituency parse tree as a hypertree (Figure 1(b)), in which each node represents a constituent and each hyperedge is a tuple of multiple nodes representing a composition of smaller child constituents into a larger parent constituent. We then build an HTNN with the hypertree as its skele
ton. During iterative update in the HTNN, we first visit each hyperedge, computing a representation of each node in the hyperedge from the other nodes using a direction-specific composition function; then we visit each node, aggregating its representations computed from all the hyperedges connected to it. In this way, not only can we keep the hierarchical information during encoding process, but also update the representation of constituents simultaneously. HTNNs combine the strengths of RvNNs and GNNs in span representation while avoid their drawbacks. Like GNNs, HTNNs compute a single unified representation for each span that integrates information from all the directions in the constituency tree. On the other hand, HTNNs follow RvNNs and utilize direction-specific composition functions to group and integrate information from different directions.
To evaluate the effectiveness of our method, we firstly conduct detailed experiments on three probing tasks: semantic role classification, named entity labeling, coreference arc prediction. Then we apply our method to the task of semantic role labeling. The results show that our method is superior to word-based, RvNN-based models and GNN-based models.
2 Hypertree Neural Networks
A hypergraph is a generalization of a graph in which an edge can join any number of nodes. A hypertree is a hypergraph  $\mathcal{H}$  such that there exists a tree  $\mathcal{T}$  and every hyperedge of  $\mathcal{H}$  is the set of nodes of a connected subtree of  $\mathcal{T}$ . We consider a directed version of hypertree that defines a hyperedge as a tuple of nodes instead of a set.
There is a long tradition of representing constituency parse trees with hypergraphs or hypertrees (Klein and Manning, 2001; Huang and Chiang, 2005). We define the hypertree representation of a constituency parse tree as follows. Each node in the hypertree represents a constituent in the parse tree and is labeled with the nonterminal symbol of the constituent. Each hyperedge represents a composition of smaller child constituents into a larger parent constituent. In this paper, we assume that the constituency parse tree is binarized, so each hyperedge contains exactly three nodes, marked with P (parent), L (left child) and R (right child) respectively. Figure 1 shows the hypertree representation of an example constituency parse tree.
than distituents. See statistics in Section 3.4.
(c)
Figure 2: Illustration of an HTNN. We show the update process of node (2,4). (a) The hypertree structure. (b) Composition within hyperedges. (c) Aggregation of multiple representations.
We generalize the methodology of GNNs on hypertrees and propose hypertree neural networks (HTNN, Figure 2). Similar to GNNs, several HTNN layers are stacked to form the model. Each layer is composed of two modules: composition within hyperedges (Section 2.3), which computes a representation of each node in a hyperedge using the other nodes from the same hyperedge; and aggregation of multiple representations (Section 2.4), which aggregates the representations of a node from all the hyperedges connected to it and forms a single unified representation. Each layer is fed with the node representation from the previous layer. The initial node representation in the first layer is computed from word representation using the attention pooling method (Section 2.2).
We treat the node representation from the last layer as the final constituent span representations and feed them into downstream task-specific decoders. For example, we may run an MLP classifier on top of a span representation for named entity recognition, or run a biaffine classifier on top of the representations of two spans for semantic role labeling. Training of HTNN can be done by optimizing any task-specific objective in an end-to-end manner along with the initial span encoder and the task-specific decoder.
2.2 Initialization of Node Representation
We follow Toshniwal et al. (2020) to compute initial span representations from word representations produced by a pretrained language model. We choose to use their attention pooling method because it performs the best in our pilot exper
iments. Specifically, the input sentence  $\mathbf{x} = \{w_1, w_2, \dots, w_l\}$  of  $l$  words, is firstly tokenized and passed through a pretrained language model to get contextual token embeddings  $\{\mathbf{e}_1, \mathbf{e}_2, \dots, \mathbf{e}_T\}$ . These embeddings are then fed to the attention pooling module to get fix-dimensional span embeddings. For a span  $s = [i,j]$  whose corresponding contextualized embeddings are  $\{\mathbf{e}_i, \dots, \mathbf{e}_{j-1}\}$ , the span representation  $\mathbf{s}_{ij}$  is calculated as:
$$
\mathbf {s} _ {i j} = \sum_ {k = i} ^ {j - 1} a _ {k} \cdot \mathbf {e} _ {k}
$$
$$
a _ {k} = \mathbf {S o f t m a x} (\mathbf {v} _ {1} ^ {T} \cdot \mathbf {e} _ {k})
$$
where  $\mathbf{v}_1$  is a learned parameter vector. Note that in a constituency parse tree, each constituent is attached with a nonterminal tag. We use an embedding layer to convert the tag to distributed vector space, and then concatenate it to  $\mathbf{s}_{ij}$ .
$$
\mathbf {s} _ {i j} ^ {\prime} = \mathbf {C o n c a t} ([ \mathbf {s} _ {i j}; \mathbf {E m b e d d i n g} (t a g) ])
$$
We project  $\mathbf{s}_{ij}^{\prime}$  using two different learned matrices to obtain  $\mathbf{h}_{ij}$  and  $\mathbf{c}_{ij}$ , the initial hidden state and memory cell of the node representing span  $[i,j]$ .
2.3 Composition within Hyperedge
Within each hyperedge, there are three nodes: parent  $(p)$ , left child  $(l)$ , right child  $(r)$ . Traditional RvNN-based methods model the relationships among them mainly in a bottom-up manner: a composition function combines the representations of the two children to obtain the representation of the parent. Le and Zuidema (2014) extend such
methods by adding a top-down process, where a node's representation can also be computed from its parent and sibling. We follow this idea that any node can be computed from the others within each hyperedge. The computation is direction-specific, meaning that the composition function is different for each of the three nodes.
Our composition function is inspired by TreeLSTM (Tai et al., 2015). The computing process can be denoted as:
$$
\left[ \mathbf {h} _ {p} ^ {\prime}; \mathbf {c} _ {p} ^ {\prime} \right] = \operatorname {C o m p o s e} \left(\mathbf {h} _ {p}, \mathbf {h} _ {l}, \mathbf {h} _ {r}, \mathbf {c} _ {l}, \mathbf {c} _ {r}\right) \tag {1}
$$
$$
\left[ \mathbf {h} _ {l} ^ {\prime}; \mathbf {c} _ {l} ^ {\prime} \right] = \operatorname {C o m p o s e} \left(\mathbf {h} _ {l}, \mathbf {h} _ {p}, \mathbf {h} _ {r}, \mathbf {c} _ {p}, \mathbf {c} _ {r}\right) \tag {2}
$$
$$
\left[ \mathbf {h} _ {r} ^ {\prime}; \mathbf {c} _ {r} ^ {\prime} \right] = \operatorname {C o m p o s e} \left(\mathbf {h} _ {r}, \mathbf {h} _ {p}, \mathbf {h} _ {l}, \mathbf {c} _ {p}, \mathbf {c} _ {l}\right) \tag {3}
$$
where subscript  $p, l, r$  denote the parent, left child and right child respectively,  $\mathbf{h}$  and  $\mathbf{c}$  denote hidden state and memory cell vectors from the previous layer, and  $\mathbf{h}'$  and  $\mathbf{c}'$  denote newly composed vectors.
We illustrate the detailed computation process of equation (1), in which we compose the left child node and the right child node to get the representation of the parent node.
$$
\mathbf {i} = \sigma \left(\mathbf {W} ^ {(i)} \mathbf {h} _ {p} + \mathbf {U} _ {l} ^ {(i)} \mathbf {h} _ {l} + \mathbf {U} _ {r} ^ {(i)} \mathbf {h} _ {r} + \mathbf {b} ^ {(i)}\right)
$$
$$
\mathbf {f} ^ {l} = \sigma \left(\mathbf {W} ^ {(f l)} \mathbf {h} _ {p} + \mathbf {U} _ {l} ^ {(f l)} \mathbf {h} _ {l} + \mathbf {U} _ {r} ^ {(f l)} \mathbf {h} _ {r} + \mathbf {b} ^ {(f)}\right)
$$
$$
\mathbf {f} ^ {r} = \sigma \left(\mathbf {W} ^ {(f r)} \mathbf {h} _ {p} + \mathbf {U} _ {l} ^ {(f r)} \mathbf {h} _ {l} + \mathbf {U} _ {r} ^ {(f r)} \mathbf {h} _ {r} + \mathbf {b} ^ {(f)}\right)
$$
$$
\mathbf {o} = \sigma \left(\mathbf {W} ^ {(o)} \mathbf {h} _ {p} + \mathbf {U} _ {l} ^ {(o)} \mathbf {h} _ {l} + \mathbf {U} _ {r} ^ {(o)} \mathbf {h} _ {r} + \mathbf {b} ^ {(o)}\right)
$$
$$
\mathbf {u} = \tanh  \left(\mathbf {W} ^ {(u)} \mathbf {h} _ {p} + \mathbf {U} _ {l} ^ {(u)} \mathbf {h} _ {l} + \mathbf {U} _ {r} ^ {(u)} \mathbf {h} _ {r} + \mathbf {b} ^ {(u)}\right)
$$
$$
\mathbf {c} _ {p} ^ {\prime} = \mathbf {i} \odot \mathbf {u} + \mathbf {f} ^ {l} \odot \mathbf {c} _ {l} + \mathbf {f} ^ {r} \odot \mathbf {c} _ {r}
$$
$$
\mathbf {h} _ {p} ^ {\prime} = \mathbf {o} \odot \operatorname {t a n h} \left(\mathbf {c} _ {p} ^ {\prime}\right)
$$
$\mathbf{i}, \mathbf{f}^l, \mathbf{f}^r, \mathbf{o} \in \mathbb{R}^d$  represent the input gate, two forget gates and output gate respectively.  $\mathbf{u} \in \mathbb{R}^d$  is the newly composed input for the memory cell.  $\mathbf{U}$  and  $\mathbf{W} \in \mathbb{R}^{d \times d}$  denote trainable weight matrices and  $\mathbf{b} \in \mathbb{R}^d$  denotes trainable bias vectors. The superscript of a matrix denotes which gate it is used for, and the subscript of a matrix denotes which node representation it is multiplied to. The computation processes of equation (2) and (3) are similarly defined, and they share some of the trainable matrices. (See more details in Appendix A.1)
2.4 Aggregation of Multiple Representations
With the exception of the root and the leaf nodes, a node in the HTNN is connected with two hyperedges, one connecting to its parent and sibling and
the other connecting to its two children. The computation process in Section 2.3 is independently carried out for each of the two hyperedges, resulting in two different representations for the node. Besides, we assume that the representation of the nodes from the previous layer also conveys important information. Therefore, we aggregate these three representations using the attention mechanism. Specifically, we use  $\mathbf{h}_0^{\prime}$  to denote the output hidden state from the previous layer, and  $\mathbf{h}_1^{\prime},\mathbf{h}_2^{\prime}$  to denote the representations computed from the two hyperedges. We use  $\mathbf{h}_0^{\prime}$  as the query vector, to attend to  $\mathbf{h}_0^{\prime},\mathbf{h}_1^{\prime},\mathbf{h}_2^{\prime}$  and compute a single aggregated representation  $\mathbf{h}^{\prime}$ .
$$
a _ {i} = \operatorname {S o f t m a x} \left(\mathbf {v} _ {2} ^ {T} \tanh  \left(\mathbf {W} \left[ \mathbf {h} _ {0} ^ {\prime}; \mathbf {h} _ {i} ^ {\prime} \right]\right)\right)
$$
$$
\mathbf {h} ^ {\prime} = \sum_ {i} a _ {i} \cdot \mathbf {h} _ {i} ^ {\prime}, \quad i \in \{0, 1, 2 \}
$$
where  $\mathbf{W} \in \mathbb{R}^{2d \times d}$  and  $\mathbf{v}_2 \in \mathbb{R}^d$  are trainable parameters. Note that the memory cell  $\mathbf{c}'$  is aggregated in the same process with shared parameters.
We use gold constituency parse trees from the datasets, and binarize them using NLTK Toolkit (Bird et al., 2009). We use pretrained RoBERTa-large $^{2}$  (Liu et al., 2019) from HuggingFace (Wolf et al., 2019) as the word encoder. We also tried SpanBERT (Joshi et al., 2020) for word encoding but found it inferior to RoBERTa-large in our pilot experiments. We freeze the parameters in the pretrained language model for all experiments. The dimension of attention pooling is 192 and the dimension of tag embedding is 64. The dimensions of hidden state and memory cell in HTNN are all set to 256. We set the default number of HTNN layers to 3. For each layer, the number of training parameters is 1,707,520. During training, the model is evaluated on the validation set every 1000 training steps. We adopt the Adam optimizer with an initial learning rate of 2e-3, which is halved if the validation score is stuck for 3 evaluations. The batch size is 64 and the dropout probability is 0.2. We train our model for 40 epochs on 4 NVIDIA Tesla P40 GPUs. $^{3}$
$^{2}$ https://huggingface.co/roberta-large
3The source code and trained model are available on https://github.com/GreyChou98/HTNN
CoNLL-2012 We conduct both probing and SRL experiments on the CoNLL-2012 dataset (Pradhan et al., 2012). This dataset is extracted from the OntoNotes v5.0 corpus, which provides gold constituency parses.
CoNLL-2005 We also conduct SRL experiments on the CoNLL-2005 dataset (Carreras and Marquez, 2005), which takes section 2-21 of the Wall Street Journal (WSJ) corpus as the training set, and section 24 as the development set. The test set consists of section 23 of WSJ for in-domain evaluation together with 3 sections from the Brown corpus for out-of-domain evaluation. CoNLL-2005 is also annotated with gold constituency parses.
3.3 Models for Comparison
Pooling A baseline model that computes span representations using attention pooling (Toshniwal et al., 2020). Our method and all the methods listed below use this representation for initialization.
SentiBERT A model for composing sentiment semantics on top of BERT (Yin et al., 2020). We re-implement their composition module based on the attention mechanism for span representation, instead of using their full model designed for sentiment analysis. We consider SentiBERT as a GNN-based model because it updates representations of all nodes simultaneously, though it only applies the attention mechanism from each parent node to its child nodes.
TreeLSTM An RvNN-based model, proposed by Tai et al. (2015), that generalizes the standard LSTM architecture to tree-structures.
Bi-TreeLSTM Le and Zuidema (2014) proposed IORNN to capture top-down decomposition besides bottom-up composition. We extend their model by using the composition function of TreeLSTMs. We refer to this model as bidirectional TreeLSTM (Bi-TreeLSTM).
GCN/GCN-sib Marcheggiani and Titov (2020) apply GCN structured with constituency trees in an SRL system. We re-implement their constituent GCN module to produce span representations, omitting their decomposition stage that is used for word representations. Furthermore, we extend GCN by adding sibling nodes in constituency trees for better comparison with our method. We refer to this modification as GCN-sib.
GAT/GAT-sib Li et al. (2020) use GAT to generate representations of the nodes in constituency parse trees. Similarly, we extend GAT by connecting sibling nodes, and we refer to this modification as GAT-sib.
3.4 Probing Experiments
We firstly conduct a probing evaluation of span representations, following Toshniwal et al. (2020). We conduct experiments on three of their probing tasks, omitting the other two tasks that probe syntax because constituency syntactic parses are input to most of the methods compared here.
- Named entity labeling (NEL): a task of predicting the entity type of a given span corresponding to an entity.
- Semantic role classification (SRC): a task of predicting the semantic roles of phrases in a sentence. Unlike the standard SRL task, in this probing task, predicates and their argument spans are given, so the goal is to classify the argument spans into specific semantic roles.
- Coreference arc prediction (COREF): a task of predicting whether a pair of spans refer to the same entity.
The task-specific decoder for these probing tasks is a two-layer MLP followed by sigmoid layers to predict the labels. For SRC and COREF, which involve two spans, the MLP takes the concatenation of the representations of the two spans as input. Following Toshiwal et al. (2020), we make predictions for each label independently, which allows using the micro-averaged F1 score as the evaluation metric. Note that in all the methods except Pooling, we only compute representations for constituent spans (those appearing in constituency parse trees). Therefore, given a distituent span, we simply predict no label. Fortunately, almost all the spans of interest in three tasks are constituent spans (98.5% for NEL, 99.7% for SRC, and 99.4% for COREF)<sup>4</sup>. We report F1 scores for all the spans (F1-all) as well as for constituent spans only (F1-const).
The results of three probing tasks are shown in Table 1. As a baseline model, Pooling's architecture is the simplest. Surprisingly, it outperforms all the other methods (except for HTNN on F1-const) on the NEL task. We believe this is because NEL is so simple that contextual word representations
See more details of the statistics in Appendix A.2
NEL	SRC	COREF	AVG
F1-const	F1-all	F1-const	F1-all	F1-const	F1-all	F1-const	F1-all
Pooling	96.18	96.07	93.08	93.05	92.99	93.01	94.08	94.04
TreeLSTM	95.05	94.22	90.02	89.88	90.07	89.70	91.71	91.27
Bi-TreeLSTM	95.25	94.42	90.49	90.35	90.33	89.96	92.02	91.58
SentiBERT	92.98	92.84	95.92	95.09	96.01	95.64	94.97	94.52
GAT	96.01	95.18	93.41	93.26	96.13	95.76	95.18	94.73
GCN	96.03	95.20	93.51	93.37	96.22	95.85	95.25	94.81
GAT-sib	95.79	94.96	92.85	92.71	95.66	95.29	94.77	94.32
GCN-sib	95.87	95.04	93.27	93.13	95.68	95.31	94.94	94.50
HTNN	96.28	95.45	93.88	93.74	96.33	95.96	95.50	95.05
Table 1: Results of probing experiments.
already provide sufficient information. More complicated models show poor performance possibly because they overfit this simple task. On the other hand, Pooling underperforms many other methods on the SRC and COREF tasks, showing the utility of syntactic information. For COREF in particular, many input spans are pronouns and their meanings may be more difficult to capture without syntactic information.
The RvNN-based models TreeLSTM and BiTreeLSTM perform worst on all these three tasks. Previous recursive models are mostly used for sentence-level tasks, and our results show their deficiencies in composing semantics for fine-grained spans rather than sentences.
For GNN-based models, SentiBERT, GAT and GCN all show great improvement over recursive models, and they are also better than Pooling on average on these three tasks. Both GAT and GCN show slight superiority over SentiBERT, probably because they take parent node into computation, rather than only children nodes. Moreover, GCN performs slightly better than GAT.