SlowGuess/NAACL_2021_OCR huggingface dataset:
Yichen Jiang\*1, Asli Celikyilmaz2, Paul Smolensky2,3, Paul Soulos\*3, Sudha Rao2, Hamid Palangi2, Roland Fernandez2, Caitlin Smith\*3, Mohit Bansal1, Jianfeng Gao2
$^{1}$ UNC Chapel Hill  $^{2}$ Microsoft Research, Redmond  $^{3}$ Johns Hopkins University
{yichenj,mbansal}@cs.unc.edu
{aslicel,psmo,sudha.rao,hpalangi,rfernand,jfgao}@microsoft.com
{psoulos1, csmit372}@jhu.edu
Abstractive summarization, the task of generating a concise summary of input documents, requires: (1) reasoning over the source document to determine the salient pieces of information scattered across the long document, and (2) composing a cohesive text by reconstructing these salient facts into a shorter summary that faithfully reflects the complex relations connecting these facts. In this paper, we adapt TP-TRANSFORMER (Schlag et al., 2019), an architecture that enriches the original Transformer (Vaswani et al., 2017) with the explicitly compositional Tensor Product Representation (TPR), for the task of abstractive summarization. The key feature of our model is a structural bias that we introduce by encoding two separate representations for each token to represent the syntactic structure (with role vectors) and semantic content (with filler vectors) separately. The model then binds the role and filler vectors into the TPR as the layer output. We argue that the structured intermediate representations enable the model to take better control of the contents (salient facts) and structures (the syntax that connects the facts) when generating the summary. Empirically, we show that our TP-TRANSFORMER outperforms the Transformer and the original TP-TRANSFORMER significantly on several abstractive summarization datasets based on both automatic and human evaluations. On several syntactic and semantic probing tasks, we demonstrate the emergent structural information in the role vectors and improved syntactic interpretability in the TPR layer outputs.<sup>1</sup>
Abstractive summarization is the task of generating a shorter version of a source text without necessarily reusing the sentences from the original source,
Original Text (Truncated): Authorities said the incident took place on Sao Joao beach in Caparica, south-west of Lisbon. The National Maritime Authority said a middle-aged man and a young girl died after they were unable to avoid the plane. [...] Other reports said the victims had been sunbathing when the plane made its emergency landing. [...] Video footage from the scene carried by local broadcasters showed a small recreational plane parked on the sand, apparently intact and surrounded by beachgoers and emergency workers. [...]
Reference Summary: A man and a child have been killed after a light aircraft made an emergency landing on a beach in Portugal.
Figure 1: An example document and its one line summary from XSum dataset. Document content that is composed into an abstractive summary is color-coded.
while preserving the meaning of its salient contents. It is a complex task that requires: semantic understanding of the source text and reasoning over its lexical units, making inferences about their relation to extract salient facts which are scattered across the long document, as well as generating a concise and coherent sequence of new sentences that covers the salient facts. While humans are remarkably good at this type of reasoning and abstraction, developing models that are capable of extraction, comprehension, abstraction, and reformulation of salient contents has been an open research question.
One prominent aspect of abstractive summarization is that models struggle with combining multiple salient aspects in the source text into a coherent and grammatical set of sentences that preserve the original information in the source document. As shown in Fig. 1, these pieces of salient information ("death", "emergency landing", "beach") are often connected by complex syntactic, causal, and temporal relations and are loosely grouped under the main topic of the source document. The transformer models (Vaswani et al., 2017) encode syntactic and semantic information of the input text into a single representation space with the self-attention, and decode the salient aspects into a short summary with the cross-attention. However, despite the large number of training examples, current state-of-the-art transformer based approaches still struggle with systematic generalization of the composition of multiple salient pieces of information.
*Work partially done while at Microsoft Research.
$^{1}$ Code and models are available at
https://github.com/jiangycTarheel/
TPT-Summ
Association for Computational Linguistics: Human Language Technologies, pages 4780-4793
In this paper, we investigate new types of computational primitives for transformers based on Tensor Product Representations (TPRs) (Smolensky, 1990) which are explicitly-compositional vector embeddings of symbolic structures. A Tensor Product Representation encodes a constituent in a symbolic structure as a composite of a role, which encodes the structural information (e.g., the dependency relation with another word), and a filler, which encodes the content of the constituent (e.g., the meaning of a word). Analogously, the TP-TRANSFORMER constructs a pair of representations for every token at every layer: a filler vector returned by attention and a novel role vector. As visualized in Fig. 2, the model then binds the role and filler vectors to produce the output of every token as a TPR. We adapt the TP-TRANSFORMER (Schlag et al., 2019), which was proposed for solving mathematics problems, for the task of abstractive summarization. Unlike the original TP-TRANSFORMER, which directly projects the input representation into a continuous role vector space, our model generates the role vectors by attending to a learned dictionary of role embeddings (Palangi et al., 2018). We observe that most learned role attention distributions are approximately one-hot, thus restricting the role vectors to a highly discrete space. This structural inductive bias encourages the TP-TRANSFORMER to encode the syntactic information in the discrete roles while isolating the semantics in the continuous fillers.
To test the ability of our TP-TRANSFORMER with discrete roles against the standard Transformer and the TP-TRANSFORMER with continuous roles, we build several models from scratch on a number of summarization datasets spanning different degrees of abstractiveness, output summary lengths, and domains. Our TP-TRANSFORMER significantly outperforms the standard Transformer and the TP-TRANSFORMER with continuous roles on the XSum (Narayan et al., 2018), Wikihow (Koupaee and Wang, 2018), and Arxiv (Cohan et al., 2018) datasets and achieves competitive performance on the CNN/Daily Mail (Hermann et al., 2015; Nallapati et al., 2016) dataset, measured by automatic metrics including ROUGE (Lin, 2004) and METEOR (Denkowski and Lavie, 2014). Our human evaluations on XSum and Wikihow datasets also correlate with the automatic metrics, demonstrating that summaries generated by our TP-TRANSFORMER are indeed better than the Trans
former's generations.
Furthermore, to investigate the structural representation that naturally emerges during training and the advantage of having compositional TPR hidden states, we design a suite of decoder probing tasks to explore the information encoded in the role, filler, and TPR space. We adopt the encoder probing task design presented in Tenney et al. (2019b) and create four decoder probing tasks: Part-of-speech tagging (POS), Dependency Labeling (DEP), Semantic Role Labeling (SRL), and Named Entity Labeling (NEL). Our findings collectively show that the decoder's role vectors encode a wealth of syntactic structures, aiding the decoder in deducing the syntactic features (e.g., being a proper noun, being the object of the root predicate) of the next token to be generated. The decoder's filler vectors on the other hand encode more semantic information (e.g., being a person's name). Furthermore, we observe that having the compositional TPR results in a more interpretable final representation than the original Transformer has at every layer, regarding the syntactic features of the next word to be generated. Our results support our hypothesis that by disentangling semantics and syntax, such structured intermediate representations enable the model to better control both the content to be conveyed and the syntactic structure needed to express it, ultimately improving the factuality and grammaticality of the generated summaries.
Our overall contributions are as follows: (1) we present a novel adaptation of the original Transformer architecture that incorporates a dictionary of role embeddings at every layer and generates Tensor Product Representation by binding the role vectors with attention outputs (filler vectors); (2) show that our TP-TRANSFORMER outperforms the Transformer as well as the original TP-TRANSFORMER (Schlag et al., 2019) on several abstractive summarization datasets; and (3) demonstrate the emergent structures in representations by revealing the disentangled syntactic and semantic information encoded in the role and filler spaces.
2 The TP-TRANSFORMER
We build our TP-TRANSFORMER based on the Transformer architecture used in Raffel et al. (2020). A TP-TRANSFORMER encoder applied to a sequence of tokens  $i = 1, \dots, I$  can be seen as a 2-dimensional lattice of cells  $(i, l)$  where  $i$  is the
Figure 2: The Filler and Role Binding operation of the TPTRANSFORMER Model architecture.
position of the input token and  $l = 1, \dots, L$  are the layer indices. All cells in the encoder have the same architecture and the cells at the same layer share the same weights. We introduce the basic components of a TP-TRANSFORMER cell in Sec. 2.2 and its encoder and decoder cells in Sec. 2.3.
2.1 Tensor-Product Representation Basics
Tensor-Product Representations (TPR; (Smolensky, 1990)) are explicitly-compositional vector embeddings of symbolic structures, where each constituent of the structure is represented as the product of a role vector, which encodes its structural information, and a filler vector, which contains the content. The TPR of a whole structure is the sum of the representation of its constituents. To represent any 3-digit number using TPRs, we need three role vectors:  $\{r(p1)$ : Ones place,  $r(p2)$ : Tens place,  $r(p3)$ : Hundreds place\} and ten filler vectors  $f$  for ten digits. For example, the TPR of the number 985 is  $r(p1) \otimes f(5) + r(p2) \otimes f(8) + r(p3) \otimes f(9)$ , where  $\otimes$  is the tensor product. When representing a number, the role vectors operate similarly as the positional embeddings in a Transformer (Vaswani et al., 2017). However, when representing natural languages, the role vectors need to encode a variety of structural information (e.g., predicate-argument, tense, etc) and thus it is infeasible to hand-design an entire suite of role vectors as we did for numbers. To overcome this challenge, for every token, we dynamically compute its role vector from a dictionary of a finite number of role embeddings learned with the entire model and treat the self-attention outputs as the fillers. We introduce the full computation procedure in Sec. 2.2.2.
2.2 The TP-TRANSFORMER Cell
Similar to the basic Transformer cell, at every layer, a TP-TRANSFORMER Encoder cell starts with a layer normalization and the multi-head self-attention followed by a residual layer. Then, the cell treats the output vectors as fillers and binds them to role vectors to construct a Tensor Product Representation, which is then passed through the feed-forward network to yield the final states.
2.2.1 Multi-Head Attention
The TP-TRANSFORMER cell adopts multi-head attention (Vaswani et al., 2017) to enable information passing between tokens. At any layer, denote the input vectors as  $X \in \mathbb{R}^{k_x \times d_m}$  and the attention target vectors as  $Y \in \mathbb{R}^{k_y \times d_m}$ , where  $k_x, k_y$  are the length of the sequences and  $d_m$  is the dimension of the input vectors. In the case of self attention, we have  $Y = X$ ; while for the encoder-decoder cross attention,  $Y$  is the encoder's output vectors. We first apply layer normalization (Ba et al., 2016) to get  $\hat{X}$  and then linearly project it to the query, key, and value vectors for each attention head  $h = 1, \dots, H$ .
$$
Q ^ {h} = \hat {X} \mathbf {W} _ {q} ^ {h} + \mathbf {b} _ {q} ^ {h}
$$
$$
K ^ {h} = Y \mathbf {W} _ {k} ^ {h} + \mathbf {b} _ {k} ^ {h} \tag {1}
$$
$$
V ^ {h} = Y \mathbf {W} _ {v} ^ {h} + \mathbf {b} _ {v} ^ {h}
$$
where  $\mathbf{W}_q, \mathbf{W}_k, \mathbf{W}_v \in \mathbb{R}^{d_m \times d_k}$ . The attention output matrix  $\bar{V}$  for each head  $h$  is computed as:
$$
\bar {V} = \operatorname {s o f t m a x} \left(\frac {Q K ^ {T}}{\sqrt {d _ {k}}}\right) V \tag {2}
$$
where  $d_{k}$  is the dimension of the key vectors  $K$ . The multi-head attention output  $O$  is the concatenation of the attention outputs from all heads followed by another linear projection  $\mathbf{W}_o\in \mathbb{R}^{d_m\times d_m}$ . We end the Multi-head Attention with a residual connection with the layer input vectors  $\hat{X}$ :
$$
\operatorname {M H A t t n} (X, Y) = \hat {X} + \left[ \bar {V} _ {1}, \dots , \bar {V} _ {H} \right] \mathbf {W} _ {o} \tag {3}
$$
where  $\bar{V}_h$  is the attention output for the  $h$ -th head.
2.2.2 Computing TPRs
Role Embeddings. Following Palangi et al. (2018), but departing from Schlag et al. (2019), every layer of our TP-TRANSFORMER is equipped with a dictionary  $\mathbf{r} \in \mathbb{R}^{N_r \times d_r}$  of  $N_r$  distinct role embeddings with a dimension of  $d_r$ . Each role embedding  $\mathbf{r}_n, n = 1, \dots, N_r$ , is randomly initialized
in the entire network. The role embeddings are normalized before computing role vectors:
$$
\hat {\mathbf {r}} _ {n} = \frac {\mathbf {r} _ {n}}{\| \mathbf {r} _ {n} \| _ {2}} \text {f o r} n = 1, \dots , N _ {r} \tag {4}
$$
At each layer, the model computes a weighted combination of these role embeddings  $\hat{\mathbf{r}}$  to form a unique role vector for every token.
Multi-Head TPR Binding. Our filler vectors correspond to the multi-head attention output  $F = \mathrm{MHA} \operatorname{attn}(X)$  (Eqn. 3). The filler  $F$  of each token has a corresponding role vector  $R$ . We first compute the  $R^h \in \mathbb{R}^{d_r}$  at every head  $h = 1, \dots, H$  as a weighted average of the normalized role embeddings  $\hat{\mathbf{r}}$ . We then concatenate the  $R^h \in \mathbb{R}^{k_x \times d_r}$  of  $H$  heads to get the multi-head role vectors  $R \in \mathbb{R}^{k_x \times (d_r \cdot H)}$  for all  $k_x$  tokens. We define this process formally as:
$$
R ^ {h} = \operatorname {s o f t m a x} \left(F \mathbf {W} _ {r} ^ {h}\right) \hat {\mathbf {r}} \tag {5}
$$
$$
R = [ R ^ {1}, \dots , R ^ {H} ]
$$
where  $\mathbf{W}_r\in \mathbb{R}^{d_m\times N_r}$  is the linear projection that computes the attention scores over the role embeddings for every token.2
We use a Hadamard product to approximate the full Tensor product in binding the role vectors  $R$  with filler vectors  $F$ , as it was shown in Schlag et al. (2019) that using the Hadamard products allows learning an optimal lower-rank approximation of the full TPRs. The binding operation is followed by an addition with the unbound fillers  $(F)$  to return the residual TPR vectors.
$$
\operatorname {T P R} (F) = R \odot F + F \tag {6}
$$
2.2.3 Residual Feed-forward Layer
The feed-forward layer of a cell consists of a linear projection followed by a ReLU activation and a second linear projection. The feed-forward output is then added to the input vectors:
$$
\mathrm {F F} (X) = X + \operatorname {R e L U} \left(X \mathbf {W} _ {g} + \mathbf {b} _ {g}\right) \mathbf {W} _ {f} + \mathbf {b} _ {f} \tag {7}
$$
Here,  $\mathbf{W}_g\in \mathbb{R}^{d_m\times d_f}$ ,  $\mathbf{b}_g\in \mathbb{R}^{d_f}$ ,  $\mathbf{W}_f\in \mathbb{R}^{d_f\times d_m}$ ,  $\mathbf{b}_f\in \mathbb{R}^{d_m}$ , and  $x$  is the function argument.
2.3 TP-TRANSFORMER Encoder & Decoder
Given the components of our basic TPTRANSFORMER cell in the previous section, we now describe how we construct the TPTRANSFORMER encoder and decoder.
First, the self-attention and the encoder-decoder cross-attention for every token can be computed as:
$$
\operatorname {S e l f} (X) = \operatorname {T P R} (\operatorname {M H A t t n} (X, X))
$$
$$
\operatorname {C r o s s} (Y, H) = \operatorname {T P R} (\operatorname {M H A t t n} (Y, H)) \tag {8}
$$
where  $H$  is the output of the encoder's final layer.  $Y$  represent the previous layer's output vectors of either the partially (so-far) decoded sequence at test time or the masked reference summary at training time. The encoder and decoder's operations at every layer can be summarized as:
$$
\operatorname {E n c o d e} (X) = \operatorname {F F} (\operatorname {S e l f} (X))
$$
$$
\operatorname {D e c o d e} (H, Y) = \operatorname {F F} (\operatorname {C r o s s} (\operatorname {S e l f} (Y), H)) \tag {9}
$$
After  $L$  layers of encoding and decoding, the final distribution of the  $i$ -th output token is given by:
$$
\hat {z} _ {i} = \operatorname {s o f t m a x} \left(\mathbf {E} ^ {T} y _ {i, L}\right) \tag {10}
$$
where  $Y_{L} = \mathrm{Decode}(H, Y_{L-1})$  are the decoder's output states at the last layer and  $\mathbf{E}$  is the tied input/output word embeddings.
3 Summarization Experiments
3.1 Abstractive Summarization Datasets
We train our models on four English abstractive summarization datasets varying the level of abstractiveness (explained below) and the length of summaries, as well as input domain.
XSum (Narayan et al., 2018) consists of 227k BBC articles from 2010 to 2017 concerning various subjects along with professionally written single-sentence summaries. Its summaries cover a wide variety of syntactic structures (relative clause, etc) and relations (causal, temporal, etc).
Wikihow (Koupaee and Wang, 2018) is a dataset consisting of instructions from the WikiHow.com website. Each of 200k examples has multiple instruction-step paragraphs, each paired with a summarizing sentence. The task is to generate the concatenated summaries of all paragraphs.
2We set  $d_r\cdot H = d_m$  so that the multi-head role vectors  $R$  have the same dimension as  $F$
3The Hadamard (or elementwise) product is the diagonal of the full tensor product.
Datasets	Summary
XSum	Luxury fashion designer Burberry has returned to profit after opening new stores and spending more on online marketing.
Wikihow	Build a trustworthy bond with your piggy. Research different training methods. Choose the training method that works best for you and your guinea pig. Gather the materials that you will need for training.
Arxiv (Abbreviated)	We study the phase behavior of a nematic liquid crystal confined between a flat substrate with strong anchoring and a patterned substrate whose structure and local anchoring strength we vary. [...] In addition the effective energy method allows one to determine the energy barriers between two states in a bistable nematic device.
CNN/DM	Mentally ill inmates in Miami are housed on the "forgotten floor". Judge Steven Leifman says most are there as a result of "avoidable felonies". While CNN tours facility, patient shouts: "I am the son of the president".
Table 1: Example summaries from XSum, Arxiv, Wikihow, and CNN/Daily Mail datasets. Text segments directly extracted from the source document are underlined.
Datasets	Split	# beam	Transformer	TPT-c (Schlag et al., 2019)	TPT-d (Ours)
XSum	Dev	1	33.34/12.07/26.47/22.28	30.73/10.38/24.39/21.14	34.61/13.13/27.59/23.43
4	34.48/13.08/27.29/24.59	31.83/11.28/25.11/22.39	35.70/14.11/28.38/25.80
Test	1	33.22/11.90/26.32/23.02	30.74/10.23/24.32/21.11	34.62/12.98/27.49/24.38
4	34.46/12.97/27.21/24.42	32.01/11.26/25.19/22.45	35.84/14.06/28.40/25.79
Wikihow	Dev	1	33.11/11.90/25.46/19.00	28.44/7.65/20.07/16.38	34.12/12.36/26.02/20.16
4	35.85/13.32/26.83/21.57	29.98/8.34/20.70/17.95	36.54/13.69/27.21/22.53
Test	1	33.40/12.18/25.66/19.31	28.63/7.82/20.23/16.49	34.19/12.47/25.99/20.23
4	35.91/13.49/27.01/21.57	30.13/8.50/20.78/18.11	36.70/13.75/27.36/22.53
Arxiv	Dev	1	35.08/10.13/31.86/19.91	32.27/7.50/29.34/17.72	35.91/10.32/32.55/20.82
4	37.95/11.48/34.03/23.31	34.45/8.40/30.91/20.17	38.35/11.56/34.32/23.74
Test	1	35.00/9.98/31.79/19.72	32.46/7.53/29.47/17.75	35.82/10.12/32.46/20.65
4	38.01/11.33/34.02/23.19	34.68/8.50/31.15/20.17	38.36/11.43/34.29/23.61
CNN/DM	Dev	1	40.56/18.18/37.73/31.91	39.66/17.45/36.99/31.15	40.61/18.17/31.77/31.35
4	41.97/19.23/38.84/34.55	41.49/18.83/38.45/34.14	41.81/19.11/38.73/34.49
Test	1	39.83/17.63/37.02/31.75	39.10/16.96/36.41/31.15	39.63/17.35/36.80/31.57
4	41.22/18.70/38.09/34.50	40.68/18.19/37.70/33.99	41.01/18.38/37.91/34.34
Table 2: Automatic evaluation results on the dev/test set of XSum, Arxiv, Wikihow, and CNN/Daily Mail dataset. The results in every cell represent F1 variant of ROUGE-1/ROUGE-2/ROUGE-L/Meteor scores. The best ROUGE scores with a statistically significant advantage, and the best meteor scores with at least 0.3 advantage are bolded.
Arxiv (Cohan et al., 2018) is a long document summarization dataset of scientific publications from arXiv.org (113k). The task is to generate the abstract from the paper body.
CNN/Daily Mail (Hermann et al., 2015; Nallapati et al., 2016) dataset contains 93k articles from CNN and 220k articles from the Daily Mail. Every article is accompanied by a few human-written bullet points about its content. We use the non-anonymized version used in See et al. (2017).
Dataset Abstractiveness. We show a summary from each of these four datasets in Table 1. According to the comparison made by Zhang et al. (2020) using the coverage and density measures (Grusky et al., 2018), the XSum and Wikihow datasets are more abstractive than the others since their summaries rarely contain large chunks of words overlapping with the source documents. CNN/Daily Mail is the least abstractive of the four. Furthermore, in most cases, a sentence in a CNN/Daily Mail summary only refers to a single sentence from the source document as suggested in Lebanoff et al.
(2019), while a sentence in an XSum or Wikihow summary usually aggregates information from multiple source sentences.
The Transformer and the two TP-TRANSFORMERS all have 6 layers, 8 heads per layer, dimension per head  $d_{k} = 64$ , model dimension  $d_{m} = 512$ , and feedforward dimension  $d_{f} = 2048$  for the encoder and decoder. Our TP-TRANSFORMER with discrete roles has  $N_{r} = 50$  role embeddings of dimension  $d_{r} = 64$  at every layer. For each dataset above, we train the all three models from scratch using an Adafactor Optimizer (Shazeer and Stern, 2018) with square root learning rate decay and dropout rate of 0.1. We evaluate the models using automatic metrics including ROUGE F1 score and METEOR.
We report automatic metric scores from our evaluated models in Table 2. We refer to the TPTRANSFORMER, with freely-generated continuous role vectors (no role dictionary) (Schlag et al.,
Datasets	Models	Grammar	Coherency	Faithfulness	Saliency	Repetition	Overall
XSum	Transformer wins	39	48	43	50	38	48
TP-TRANSFORMER wins	47	48	46	47	42	52
Tie / No agreement	34	24	31	23	40	20
Wikihow	Transformer wins	45	45	43	54	48	43
TP-TRANSFORMER wins	48	45	46	47	48	59
Tie / No agreement	27	30	31	19	24	18
Table 3: Human Evaluation results on 120 random samples from the XSum (Narayan et al., 2018) and Wikihow (Koupae and Wang, 2018) test sets. The best numbers with an advantage of at least 5 points are underlined.
2019) as TPT-c, and our own TP-TRANSFORMER with a discrete set of role embeddings as TPT-d. On the XSum, Arxiv, and Wikihow datasets, our TP-TRANSFORMER (TPT-d) outperforms the original Transformer on all metrics. On the CNN/Daily Mail dataset, both models obtain similar performance across all metrics. On every dataset, the TPT-c model which excels on the mathematics dataset, is the worst among the three models being compared. This suggests that continuous role vectors are not suited to the summarization tasks.