Venue: NAACL 2018

# A Mixed Hierarchical Attention based Encoder-Decoder Approach for

Standard Table Summarization

Parag Jain\({}^{\dagger}\) Anirban Laha\({}^{\dagger,*}\) Karthik Sankaranarayanan\({}^{\dagger}\)

**Preksha Nema\({}^{*}\) Mitesh M. Khapra\({}^{\ddagger}\) Shreyas Shetty\({}^{*}\) \({}^{\dagger}\)IBM Research \({}^{\star}\)IIT Madras, India**

\({}^{\ddagger}\) Robert Bosch Center for Data Science and Artificial Intelligence, IIT Madras

{pajain34,anirlaha,kartsank}@in.ibm.com

{preksha,miteshk,shshett}@cse.iitm.ac.in

###### Abstract

Structured data summarization involves generation of natural language summaries from structured input data. In this work, we consider summarizing structured data occurring in the form of tables as they are prevalent across a wide variety of domains. We formulate the _standard table summarization_ problem, which deals with tables conforming to a single predefined schema. To this end, we propose a _mixed hierarchical attention_ based encoder-decoder model which is able to leverage the structure in addition to the content of the tables. Our experiments on the publicly available weathergov dataset show around 18 BLEU (\(\sim 30\%\)) improvement over the current state-of-the-art.

## 1 Introduction

Abstractive summarization techniques from structured data seek to exploit both structure and content of the input data. The type of structure on the input side can be highly varied ranging from key-value pairs (e.g. WikiBioLebret et al. (2016)), source code Iyer et al. (2016), ontologies Androutsopoulos et al. (2014); Colin et al. (2016), or tables Wiseman et al. (2017), each of which require significantly varying approaches. In this paper, we focus on generating summaries from tabular data. Now, in most practical applications such as finance, healthcare or weather, data in a table are arranged in rows and columns where the schema is known beforehand. However, change in the actual data values can necessitate drastically different output summaries. Examples shown in the figure 1 have a predefined schema obtained from the weathergov dataset Liang et al. (2009) and its corresponding weather report summary. Therefore, the problem that we seek to address in this paper is to generate abstractive summaries of tables conforming to a pre-defined fixed schema (as opposed to cases where the schema is unknown). We refer to this setting as **standard table summarization** problem. Another problem that could be formulated is one in which the output summary is generated from multiple tables as proposed in a recent challenge Wiseman et al. (2017) (this setting is out of the scope of this paper). Now, as the schema is fixed, simple rule based techniques Konstas and Lapata (2013) or template based solutions could be employed. However, due to the vast space of selection (which attributes to use in the summary based on the current value it takes) and generation (how to express these selected attributes in natural language) choices possible, such approaches are not scalable in terms of the number of templates as they demand hand-crafted rules for both selection and generation.

We attempt to solve the problem of standard table summarization by leveraging the hierarchical nature of fixed-schema tables. In other words, rows consist of a fixed set of attributes and a table is defined by a set of rows. We cast this problem into a **mixed hierarchical attention model** following the encode-attend-decode Cho et al. (2015) paradigm. In this approach, there is static attention on the attributes to compute the row representation followed by dynamic attention on the rows, which is subsequently fed to the decoder. This formulation is theoretically more efficient than the fully dynamic hierarchical attention framework followed by Nallapati et al. (2016). Also, our model does not need sophisticated sampling or sparsifying techniques like Ling and Rush (2017); Deng et al. (2017), thus, retaining differentiability. To demonstrate the efficacy of our approach, we transform the publicly available Weathergov dataset Liang et al. (2009) into fixed-schema tables, which is then used for our experiments. Our proposed _mixed hierarch

cal attention model_ provides an **improvement of around 18 BLEU (around 30%) over the current state-of-the-art result** by Mei et al. (2016).

## 2 Tabular Data Summarization

A standard table consist of set of records (or rows) \(R=(r_{1},r_{2},...r_{T})\) and each record \(r\) has a fixed set of attributes (or columns) \(A^{r}=(a_{r1},a_{r2},...a_{rM})\). Tables in figure 1 have 7 columns (apart from 'TYPE') which correspond to different attributes. Also \(U=(u_{1},u_{2},....u_{T})\) represents the type of each record where \(u_{k}\) is one-hot encoding for the record type for record \(r_{k}\). Training data consists of instance pairs \((X_{i},Y_{i})\) for \(i=1,2,..n\), where \(X_{i}=(R_{i},U_{i})\) represents the input table and \(Y_{i}=(y_{1},...,y_{T^{\prime}})\) represents the corresponding natural language summary. In this paper, we propose an end-to-end model which takes in a table instance \(X\) to produce the output summary \(Y\). This can be derived by solving in \(Y\) the following conditional probability objective:

\[Y^{*}=\arg\max_{Y}\prod_{t=1}^{T^{\prime}}p(y_{t}|y_{1},...,y_{t-1},X) \tag{1}\]

### Mixed Hierarchical Attention Model (MHAM)

Our model is based on the _encode-attend-decode_ paradigm as defined by Cho et al. (2015). It consists of an _encoder_ RNN which encodes a variable length input sequence \(\textbf{x}=(x_{1},...,x_{T})\) into a representation sequence \(\textbf{c}=(c_{1},...,c_{T})\). Another decoder RNN generates sequence of output symbols \(\textbf{y}=(y_{1},...,y_{T^{\prime}})\), attending to different combinations of \(c_{i}\) while generating different \(y_{t}\).

As illustrated in figure 2, our encoder is not a single RNN. The encoder has a hierarchical structure to leverage the structural aspect of a _standard table_: a table consists of a set of records (or rows) and each record consists of values corresponding to a fixed set of attributes. We call it a _mixed hierarchical attention_ based encoder, as it incorporates static attention and dynamic attention at two different levels of the encoder. At the record level, the attention over record representations is _dynamic_ as it changes with each decoder time step. Whereas at the attribute level, since the schema is fixed, a record representation can be computed without the need of varying attention over attributes - thus _static_ attention is used. For

Figure 1: Standard Table Summarization with fixed schema tables as input.

Figure 2: Proposed Architecture



example, with respect to weatheringOV dataset, a _temperature_ record will always be defined by the attributes like min, max and mean irrespective of the decoder time step. So, attention over attributes can be static. On the other hand, while generating a word by the decoder, there can be a preference of focusing on a record type say, _temperature_, over some other type say, _windSpeed_. Thus, dynamic attention is used across records.

[!ht]

\[\bar{A}^{r}_{j}=W_{j}A^{r}_{j},\;\forall j\in[1,M] \tag{2}\] \[I^{r}_{j}=\bar{A}^{r}_{j}\odot\bar{u}_{r}\quad,\quad\bar{u}_{r}= W_{0}u_{r}\] (3) \[\alpha^{r}_{j}=softmax_{j}(I^{r})\] (4) \[B^{r}=\sum_{j}\alpha^{r}_{j}\bar{A}^{r}_{j} \tag{5}\]

\[c_{r}=[h_{r};B^{r}]\quad,\quad h_{r}=GRU(B^{r}) \tag{6}\] \[g_{r}=\sigma(q^{T}tanh(Pc_{r}))\] (7) \[\beta^{r}_{t}=v^{T}tanh(W_{s}s_{t-1}+W_{c}c_{r})\] (8) \[w^{r}_{t}=softmax_{r}(\beta_{t})\] (9) \[z_{t}=\sum_{r}\gamma^{r}_{t}c_{r}\quad,\quad\gamma^{r}_{t}= \frac{g_{r}w^{r}_{t}}{\sum\limits_{r}g_{r}w^{r}_{t}}\] (10) \[s_{t}=GRU(z_{t},s_{t-1})\] (11) \[l_{t}=W_{1}s_{t}+W_{2}z_{t}+b_{l}\] (12) \[p_{t}= softmax(l_{t}) \tag{13}\]

Capturing attribute semantics:We learn record type embeddings and use them to calculate attentions over attributes. _For the trivial case of all records being same type, it boils down to having a single record type embedding._ Given attributes \(A^{r}\) for a record \(r\), where each attribute \(a^{r}_{i}\) is encoded into a vector \(A^{r}_{i}\) based on the attribute type (discussed further in section 3), using equation 2 we embed each attribute where \(W_{j}\) is the embedding matrix for \(j^{th}\) attribute. We embed record type one-hot vector \(u_{r}\) through \(W_{0}\), which is used to compute the importance score \(I^{r}_{j}\) for attribute \(j\) in record \(r\) according to equation 3.

Static Attribute attention:Not all attribute values contribute equally to the record. Hence, we introduce attention weights for attributes of each record. These attention weights are static and does not change with decoder time step. We calculate the attention probability vector \(\alpha^{r}\) over attributes using the attribute importance vector \(I^{r}\). The attention weights can then be used to calculate the record representation \(B^{r}\) for record \(r\) by using equations 4 and 5.

Record Encoder:A GRU based RNN encoder takes as input a sequence of attribute attended records \(B^{1:N}\) and returns a sequence of hidden states \(h_{1:N}\), where \(h_{r}\) is the encoded vector for record \(B^{r}\). We obtain the final record encoding \(c_{r}\) (equation 6) by concatenating the GRU hidden states with the embedded record encodings \(B^{r}\).

Static Record attention:In a table, a subset of record types can always be more salient compared to other record types. This is captured by learning a static set of weights over all the records. These weights regulate the dynamic attention weights computed during decoding at each time step. Equation 7 performs this step where \(g_{r}\) is the static record attention weight for \(r^{th}\) record and \(q\) and \(P\) are weights to be learnt. We do not have any constraints on static attention vector.

Dynamic Record attention for Decoder:Our decoder is a GRU based decoder with dynamic attention mechanism similar to Mei et al. (2016) with modifications to modulate attention weights at each time step using static record attentions. At each time step \(t\) attention weights are calculated using 8, 9, 10, where \(\gamma^{r}_{t}\) is the aggregated attention weight of record \(r\) at time step \(t\). We use the _soft attention_ over input encoder sequences \(c_{r}\) to calculate the weighted average, which is passed to the GRU. GRU hidden state \(s_{t}\) is used to calculate output probabilities \(p_{t}\) by using a softmax as described by equation 11, 12, 13, which is then used to get output word \(y_{t}\).

Due to the static attention at attribute level, the time complexity of a single pass is \(O(TM+TT^{\prime})\), where \(T\) is the number of records, \(M\) is the number of attributes and \(T^{\prime}\) is the number of decoder steps. In case of dynamic attention at both levels (as in Nallapati et al. (2016)), the time complexity is much higher \(O(TMT^{\prime})\). Thus, _mixed hierarchical attention_ model is faster than fully dynamic hierarchical attention. For better understanding of the contribution of hierarchical attention(MHAM), we propose a simpler non-hierarchical (NHM) architecture with attention only at record level. In NHM, \(B^{r}\) is calculated by concatenating all the record attributes along with corresponding record type 



## 3 Experiments

**Dataset and methodology**: To evaluate our model we have used weathergov dataset Liang et al. (2009) which is the standard benchmark dataset to evaluate tabular data summarization techniques. We compared the performance of our model against the state-of-the-art work of MBW Mei et al. (2016), as well as two other baseline models KL Konstas and Lapata (2013) and ALK Angeli et al. (2010). Dataset consists of a total of 29,528 tables (25000:1000:3528 ratio for train:validation:test splits) corresponding to scenarios created by collecting weather forecasts for 3,753 cities in the U.S.A over three days. There are 12 record types consisting of both numeric and categorical values. Each table contains 36 weather records (e.g., temperature, wind direction etc.) along with a corresponding natural language summary.

Input Encodings:Attributes were encoded based on the attribute type. Numbers are encoded in binary representation. Record type is encoded as a one-hot vector. Mode attribute is encoded using specific ordinal encodings for example 'Lkly', 'SChc', 'Chc' are encoded as '00100000000000', '00010000000000000' and '000001000000000' respectively. Similar works for directions, for example 'NW', 'NNE' and 'NE' are encoded as '00000100000000', '00000011000000' and '00000001000000' resp. Time interval were also encoded as ordinal encodings, for example '6-21' is encoded as '111100' and '6-13' is encoded as '110000', the six bits corresponding to six atomic time intervals available in the dataset. Other attributes and words were encoded as one-hot vectors.

### Training and hyperparameter tuning

We used TensorFlow Abadi et al. (2015) for our experiments. Encoder embeddings were initialized by generating the values from a uniform distribution in the range [-1, 1]. Other variables were initialized using Glorot uniform initialization Glorot and Bengio (2010). We tune each hyperparameter by choosing parameter from a ranges of values, and selected the model with best sBLEU score in validation set over 500 epochs. We did not use any regularization while training the model. For both the models, the hyperparameter tuning

\begin{table}
\begin{tabular}{|p{42.7pt}|p{42.7pt}|p{42.7pt}|p{42.7pt}|p{42.7pt}|p{42.7pt}|p{42.7pt}|} \hline  & \multicolumn{6}{c|}{**Input Table**} & \multicolumn{1}{p{42.7pt}|}{**Generated Output**} \\ \hline Type & TIME & MN & MAX & MODP & MBB+144 & MBS+2.5 \\ \hline transparent window & 17.38 & 28 & 8 & - & - & - \\ window window & 17.38 & 28 & 8 & - & - & - \\ window window & 17.38 & 28 & 8 & - & - & - \\ window window & 17.38 & 28 & 8 & - & - & - \\ window window & 17.38 & 28 & 8 & - & - & - \\ window window & 17.38 & 28 & 8 & - & - & - \\ window window & 17.38 & 28 & 8 & - & - & - \\ window window & 17.38 & 28 & 8 & - & - & - \\ window window & 17.38 & 28 & 8 & - & - & - \\ window window & 17.38 & 28 & 8 & 28 & - & - \\ window window & 17.



was separately performed to give both models a fair chance of performance. For both the models, Adam optimizer [14] was used with learning rate set to 0.0001. We found embedding size of 100, GRU size of 400, static record attention size\(P\) of 150 to work best for MHAM model. We also experimented using bi-directional GRU in the encoder but there was no significant boost observed in the BLEU scores.

**Evaluation metrics**: To evaluate our models we employed BLEU and Rouge-L scores. In addition to the standard BLEU (sBleu) [23], a customized BLEU (cBleu) [28] has also been reported. cBleu does not penalize numbers which differ by at most five; hence 20 and 18 will be considered same.

## 4 Results and Analyses

Table 2 describes the results of our proposed models (MHAM and NHM) along with the aforementioned baseline models. We observe a significant performance improvement of 16.6 cBleu score (24%) and 18.3 sBleu score (30%) compared to the current state-of-the-art model of MBW. MHAM also shows an improvement over NHM in all metrics demonstrating the importance of hierarchical attention.

**Attention analysis:** Analysis of figure 3 reveals that the learnt attention weights are reasonable. For example, as shown in figure 3(a), for the phrase 'with a high near 52', the model had a high attention on _temperature_ before and while generating the number '52'. Similarly while generating 'mostly cloudy', the model had a high attention on _precipitation potential_. Attribute attentions are also learned as expected (in figure 3(b)). The _temperature_, _wind speed_ and _gust_ records have high weights on min/max/mean values which describe these records.

**Qualitative analysis:** Table 1 contains example table-summary pairs, with summary generated by the proposed hierarchical and non-hierarchical versions. We observe that our model is able to generate numbers more accurately by enabling hierarchical attention. Our model is also able to capture weak signals like _snow accumulation_. Further, our proposed model MHAM is able to avoid repetition as compared to NHM.

## 5 Conclusion and Future Work

In this work, we have formulated the problem of _standard table summarization_ where all the tables come from a predefined schema. Towards this, we proposed a novel mixed hierarchical attention based encoder-decoder approach. Our experiments on the publicly available weathergov benchmark dataset have shown significant improvements over the current state-of-the-art work. Moreover, this proposed method is theoretically more efficient compared to the current fully dynamic hierarchical attention model. As future work, we propose to tackle general tabular summarization where the schema can vary across tables in the whole dataset.

## 6 Acknowledgements

We would like to acknowledge Abhijit Mishra for his valuable feedback and comments.

## References

* [Abadi et al.2017] Martin Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S. Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin,

Figure 3: Heatmaps: (a) record level attention (top) and (b) attribute level attention (bottom).

\begin{table}
\begin{tabular}{l l l l} \hline Model & sBleu & cBleu & Rouge-L \\ \hline KL & 36.54 & - & - \\ ALK & 38.40 & 51.50 & - \\ MBW & 61.0 & 70.4 & - \\ \hline NHM & 76.2 & 85.0 & 86.4 \\
**MHAM** & **79.3** & **87.0** & **88.5** \\ \hline \end{tabular}
\end{table}
Table 2: Overall results 

Sanjay Ghemawat, Ian Goodfellow, Andrew Harp, Geoffrey Irving, Michael Isard, Yangqing Jia, Rafal Jozefowicz, Lukasz Kaiser, Manjunath Kudlur, Josh Levenberg, Dan Mane, Rajat Monga, Shery Moore, Derek Murray, Chris Olah, Mike Schuster, Jonathon Shlens, Benoit Steiner, Ilya Sutskever, Kunal Talwar, Paul Tucker, Vincent Vanhoucke, Vijay Vasudevan, Fernanda Viegas, Oriol Vinyals, Pete Warden, Martin Wattenberg, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng. 2015. TensorFlow: Large-scale machine learning on heterogeneous systems. Software available from tensorflow.org.
* Androutsopoulos et al. (2014) Ion Androutsopoulos, Gerasimos Lampouras, and Dimitrios Galanis. 2014. Generating natural language descriptions from OWL ontologies: the naturalowl system. _CoRR_, abs/1405.6164.
* Angeli et al. (2010) Gabor Angeli, Percy Liang, and Dan Klein. 2010. A simple domain-independent probabilistic approach to generation. In _Proceedings of the 2010 Conference on Empirical Methods in Natural Language Processing_, pages 502-512. Association for Computational Linguistics.
* Cho et al. (2015) KyungHyun Cho, Aaron C. Courville, and Yoshua Bengio. 2015. Describing multimedia content using attention-based encoder-decoder networks. _CoRR_, abs/1507.01053.
* Proceedings of the Ninth International Natural Language Generation Conference, September 5-8, 2016, Edinburgh, UK_, pages 163-167.
* Deng et al. (2017) Yuntian Deng, Anssi Kanervisto, Jeffrey Ling, and Alexander M. Rush. 2017. Image-to-markup generation with coarse-to-fine attention. In _Proceedings of the 34th International Conference on Machine Learning, ICML 2017, Sydney, NSW, Australia, 6-11 August 2017_, pages 980-989.
* Glorot and Bengio (2010) Xavier Glorot and Yoshua Bengio. 2010. Understanding the difficulty of training deep feedforward neural networks. In _In Proceedings of the International Conference on Artificial Intelligence and Statistics (AISTATS'10). Society for Artificial Intelligence and Statistics_.
* Iyer et al. (2016) Srinivasan Iyer, Ioannis Konstas, Alvin Cheung, and Luke Zettlemoyer. 2016. Summarizing source code using a neural attention model. In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 2073-2083, Berlin, Germany. Association for Computational Linguistics.
* Kingma and Ba (2014) Diederik Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. _arXiv preprint arXiv:1412.6980_.
* Konstas and Lapata (2013) Ioannis Konstas and Mirella Lapata. 2013. Inducing document plans for concept-to-text generation. In _EMNLP_.
* Lebret et al. (2016) Remi Lebret, David Grangier, and Michael Auli. 2016. Neural text generation from structured data with application to the biography domain. In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing_, pages 1203-1213, Austin, Texas. Association for Computational Linguistics.
* Volume 1_, ACL '09, pages 91-99, Stroudsburg, PA, USA. Association for Computational Linguistics.
* Ling and Rush (2017) Jeffrey Ling and Alexander M. Rush. 2017. Coarse-to-fine attention models for document summarization. In _Proceedings of the Workshop on New Frontiers in Summarization, NFiS@EMNLP 2017, Copenhagen, Denmark, September 7, 2017_, pages 33-42.
* Mei et al. (2016) Hongyuan Mei, Mohit Bansal, and Matthew R. Walter. 2016. What to talk about and how? selective generation using lstms with coarse-to-fine alignment. In _Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 720-730, San Diego, California. Association for Computational Linguistics.
* Nallapati et al. (2016) Ramesh Nallapati, Bing Xiang, and Bowen Zhou. 2016. Sequence-to-sequence rnns for text summarization. _CoRR_, abs/1602.06023.
* Papineni et al. (2002) Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In _Proceedings of the 40th annual meeting on association for computational linguistics_, pages 311-318. Association for Computational Linguistics.
* Wiseman et al. (2017) Sam Wiseman, Stuart M. Shieber, and Alexander M. Rush. 2017. Challenges in data-to-document generation. In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, EMNLP 2017, Copenhagen, Denmark, September 9-11, 2017_, pages 2253-2263.