IM-TQA: A Chinese Table Question Answering Dataset with Implicit and Multi-type Table Structures
Year: 2023

Abstract:
Various datasets have been proposed to promote the development of Table Question Answering (TQA) technique. However, the problem setting of existing TQA benchmarks suffers from two limitations. First, they directly provide models with explicit table structures where row headers and column headers of the table are explicitly annotated and treated as model input during inference. Second, they only consider tables of limited types and ignore other tables especially complex tables with flexible header locations. Such simplified problem setting cannot cover practical scenarios where models need to process tables without header annotations in the inference phase or tables of different types. To address above issues, we construct a new TQA dataset with implicit and multi-type table structures, named IM-TQA, which not only requires the model to understand tables without directly available header annotations but also to handle multi-type tables including previously neglected complex tables. We investigate the performance of recent methods on our dataset and find that existing methods struggle in processing implicit and multi-type table structures. Correspondingly, we propose an RGCN-RCI framework outperforming recent baselines. We will release our dataset to facilitate future research.1

heading: 1 Introduction
text: To automatically gain valuable information from numerous tables, Table Question Answering (TQA) technique was developed and can answer natural language questions over tables (Zheng et al., 2022; Hui et al., 2022; Herzig et al., 2020). Correspondingly, researchers proposed various TQA datasets aiming at different application scenarios (Zhong
1The code and data will be released at https://github. com/SpursGoZmy/IM-TQA
∗ Indicates equal contribution. †This work was done during internship at Baidu Inc. ‡ Corresponding author: Zheng Lin.
et al., 2017; Iyyer et al., 2017; Chen et al., 2020). As the model performance continues to improve, TQA technique has been widely used in intelligent data analysis tools, e.g., Power BI2, and Tableau3
Though previous datasets have promoted the development of TQA technique, the problem setting of existing benchmarks suffers from two limitations. First, existing benchmarks only evaluate the performance of model on tables with explicit table structures, which means locations and directions of headers are explicitly annotated and treated as model input during inference. For example, Text2SQL benchmarks offer annotated column headers (Zhong et al., 2017; Yu et al., 2018) and recent hierarchical table datasets also contain hierarchical header annotations (Katsis et al., 2022; Cheng et al., 2022), which are available at inference time. This setting artificially lowers the difficulty of the task. Nevertheless, in practical scenarios, model may encounter plenty of tables without labeled headers. Manually annotating headers for these tables is prohibitively expensive and timeconsuming. As a result, a benchmark is needed to evaluate the performance of TQA models on tables
2https://powerbi.microsoft.com/en-us/ 3https://www.tableau.com/
5074
with implicit table structures. Here, “implicit table structures” represents that header annotations of the table are not directly available during inference.
Second, existing benchmarks only consider limited table types and ignore complex tables with flexible header locations. Previous datasets mainly focus on vertical or hierarchical tables whose header cells only locate on the top or left side of the table (Pasupat and Liang, 2015; Wang et al., 2020b; Guo et al., 2021; Cheng et al., 2022), as shown in Figure 1 (a) and (c), but neglect the fact that model may require handling multi-type tables, especially complex tables whose header cells also appear at other positions in the table such as the bottom-right region, as illustrated in Figure 1 (d). Complex tables are prevalent in professional equipment specifications and record sheets, which are beyond the ability of current TQA systems.
Above analyses show that the problem setting of previous benchmarks restricts the application of TQA models. In this paper, we define a new problem, table question answering over implicit and multi-type table structures. In this problem setting, annotations of table headers are not available during inference and model needs to comprehend implicit and multi-type table structures to answer questions. To facilitate the study on this problem, we build the first Chinese Table Question Answering dataset with Implicit and Multi-type table structures, named IM-TQA, which consists of 1,200 tables and 5,000 questions (Section 3). Our dataset includes tables of four types from different domains, especially complex tables neglected by published studies. We annotate tables not only with various Lookup questions and their answer cells, but also with functional roles of table cells.
IM-TQA presents a strong challenge to existing models. Because of the inability to understand implicit and diverse table structures, the state-of-theart baseline RCI (Row and Column Intersection) model (Glass et al., 2021) only achieves 49.6% overall accuracy and 23.8% accuracy on complex tables. To improve the performance of RCI, we propose an RGCN-RCI framework (Section 4). Specifically, this framework solves the new problem in two stages: (1) Table is modeled as graph and an RGCN (Schlichtkrull et al., 2017) is used to comprehend table structure and predict table header cells. (2) Based on predictions of RGCN, header information is merged into the text sequence representation of each row or column, which helps
RCI model to predict whether a row or column contains the answer or not. Though our framework shows high effectiveness and improves the accuracy on all tables and complex tables by 3.8% (49.6% → 53.4%) and 8.2% (23.8% → 32.0%) respectively, it still lags far behind human performance, i.e., 95.1% and 94.1% on all and complex tables. We hope this work could raise attention to the implicit and multi-type table structures, facilitating the development of TQA models to address more complex and realistic table data.
To summarize, we conclude our contributions as follows:
• Considering the limitations of current TQA setting, we propose a new problem, table question answering over implicit and multi-type table structures, which is complementary to traditional TQA problem, i.e., TQA over explicit and limited (usually single-type) table structures.
• We construct and publicly release a new dataset, IM-TQA, to promote the research on this problem. Our dataset includes tables of four different types with implicit structures, especially complex tables ignored by former benchmarks.
• We investigate the performance of existing methods on our dataset and propose an RGCNRCI framework which outperforms state-ofthe-art baselines on all table types.

heading: 2 Problem Statement
text: In this section, we define the problem of table question answering over implicit and multi-type table structures. Consisting of multiple cells, a table t can be defined as t .= {P,R,V}, where P, R and V represents the set of position information, functional roles and values of cells in table t respectively. Table cells can be categorized into different types according to their functional roles, e.g., some cells are column headers and others are data cells.
Under the setting of traditional TQA problem, table structures are explicitly annotated (Zhong et al., 2017; Yu et al., 2018) or can be easily obtained by heuristic methods designed for specific table type (Cheng et al., 2022; Katsis et al., 2022). In this setting, functional roles R are directly provided for the model f(·), which outputs answer y given a natural language question q, i.e.,
y = f(q, t) = f(q,P,R,V). However, in the setting of real applications where the model may need to process implicit and multi-type table structures, functional roles R are not available in advance and the model requires distinguishing useful headers from data cells. Thus, TQA over implicit and multi-type table structures can be formalized as: y = f(q, t ′ ) = f(q,P,V). This problem setting poses a great challenge to existing methods. Unfortunately, none of previous benchmarks served as testbed for this problem. To fill the gap, we build IM-TQA, which includes multi-type tables and requires understanding implicit table structures. Compared with previous datasets, the improvements of IM-TQA are demonstrated in Figure 2.

heading: 3 Dataset Construction
text: In this section, we first introduce considered table types, and then elaborate the dataset construction procedure. We also compare IM-TQA with other related datasets. We recruit 10 professional annotators and provide them with sufficient instructions. The dataset construction totally costs 1,200 working hours. The ethical considerations will be discussed in the Section 9.

heading: 3.1 Considered Table Types
text: As shown in Figure 1, we divide tables into 4 types according to their structure characteristics, which is in line with previous work (Wang et al., 2021; Ghasemi-Gol and Szekely, 2018) with complex table as an important complement. More table and question examples are shown in App. F. Vertical Table Table data is arranged in the vertical direction, with the first row as column headers and other rows as data tuples. Horizontal Table Table data is arranged in the horizontal direction, with the first column as row headers and other columns as data tuples. Hierarchical Table Table data is arranged in both
vertical and horizontal directions, with headers exhibiting a multi-level hierarchical structure. Complex Table In tables of above 3 types, header cells only locate on the top or left side of the table. But in complex tables, headers also appear at other positions such as bottom-right region in the table and can be mixed with data cells, as depicted in Figure 1 (d). Such tabular structures with flexible header locations often appear in professional equipment specifications and record sheets, presenting a great challenge to existing methods.

heading: 3.2 Table Collection and Storage
text: To ensure the diversity of table data, we collect tables from open websites of different domains. Data sources include company annual reports from different industries4 such as manufacturing, medicine and education; web pages of Baidu Encyclopedia5 on science, professional equipment, etc. Tables were extracted to Excel files by annotators. We correct typos in the collected tables and filter tables without meaningful data. Finally, we preserve 300 tables for every table type. Figure 4 shows the distribution of domains.
In order to store various tables, we design a storage method which separately stores cell positions P and cell contents V. To store cell positions, a cell ID is assigned to each table cell in the row-first order. For a table including m rows and n columns, its cell IDs constitute an m×n matrix representing cell locations. This matrix contains table layout information such as neighbouring relations between different cells. As for cell contents, every cell value is put into a list in the same row-first order. Figure
4http://eid.csrc.gov.cn/101111/index.html 5https://baike.baidu.com/
3 (a) and (b) demonstrate the storage format of the complex table in Figure 1 (d). The original table can be recovered based on cell ID matrix and cell value list. The storage method does not waste extra space for merged cells and is also convenient to annotate header cells and answer cells.

heading: 3.3 Cell Type Annotation
text: In our problem setting, models are required to recognize functional roles R of table cells, i.e., conducting cell type classification (CTC) task. There are different taxonomies of cell types which focus on hierarchical spreadsheet tables (Ghasemi Gol et al., 2019; Dong et al., 2019; Zhang et al., 2021). To model different table structures especially complex tables, we adopt a new taxonomy of cell types with the concentration on header cells that are useful for TQA models to locate correct answer cells. Specifically, we categorize table cells into 5 types based on their functional roles. Row Attribute and Column Attribute Row attribute and column attribute are traditional table headers which describes other cells in the same row and in the same column respectively, e.g., yellow cells and red cells in Figure 1. Attribute cells only serve the purpose of describing other cells and they are not meaningful data. Row Index and Column Index Row index and column index are individual cells that are used to index data records in the row or column orientation, e.g., blue cells and green cells in Figure 1. Index cells are also meaningful data. For instance, in vertical tables, data cells in the primary key column are unique identifiers of each row. Pure Data Pure data cells are the core body of a table. They do not have the function of describing or indexing other cells and their meanings should be understood with the help of above header cells.
We instructed annotators in distinguishing 5 cell types and asked them to annotate cell ID lists of attribute and index cells, as shown in Figure 3 (c). Other table cells are deemed pure data cells.

heading: 3.4 QA Pairs Construction
text: After identifying header cells, we asked annotators to raise questions about data cells and label answer cell IDs, as illustrated in Figure 3 (d). To avoid annotation artifacts from the homogeneous patterns of questions, e.g., always using the same question expression, annotators were asked to use diverse language expressions to raise questions and
answers’ locations should change frequently. Annotators were also encouraged to paraphrase questions to increase difficulty.
Questions about table are broadly classified into two types: Lookup and Aggregation (Glass et al., 2021). Lookup questions require selecting table cells as answers whereas Aggregation questions are answered by performing arithmetic operations over a subset of cells, such as Sum(). In this work, our primary focus is on Lookup questions and we leave the annotation of more complex aggregation and numerical reasoning questions in the future. Besides single-cell answer, annotators were allowed to select one row or one column or arbitrary table cells as answer. Figure 4 shows the distribution of question types. Four questions were raised for each vertical and horizontal table, and five questions for every hierarchical and complex table.

heading: 3.5 Data Review
text: In order to guarantee annotation quality, before each annotation task began, all annotators conducted a trial annotation with 50 samples and the results were checked. Feedback and sufficient Q&As were given to corresponding annotators until they fully comprehended the annotation requirements. After all annotation tasks finished, we and two most experienced annotators performed the final review to fix labeling errors. We inspected annotations for the correctness of cell type annotations and answer cell annotations. We also inspected the grammar and wording and filtered questions with obvious mistakes. We checked for offensive content and identifiers, and replaced identifying information with mono-directional hashes. Finally, we preserve 1,200 tables and 5,000 questions. Inner-annotator agreement and annotation instruction screenshots are shown in App. A and E respectively.

heading: 3.6 Dataset Analysis and Comparison
text: Table 1 shows a comprehensive comparison of IMTQA to related TQA datasets. The advantages of the proposed dataset are as follows: (1) It is the first TQA dataset that contains implicit and multitype table structures, especially complex tables ignored by former datasets. Though AIT-QA (Katsis et al., 2022), HiTab (Cheng et al., 2022) and MultiHiertt (Zhao et al., 2022) include vertical and hierarchical tables, they ignore the other two types. (2) It is annotated with cell functional roles and QA pairs, which supports both CTC and TQA task. Diverse table structures in our dataset challenge ex-
isting CTC and TQA models. (3) Compared with single-domain datasets like AIT-QA and WTQ, IMTQA includes tables from various domains.
We split tables of each type based on table structures. If the location distributions of header cells in two tables are exactly same, i.e., same cell type at the same table position, we consider they share the same table structure. Table structures are randomly split into train (80%), valid (7%) and test set (13%). Tables of the same structure and questions of the same table are assigned to the same split. Table 2 shows the basic statistics of each split. More statistics are shown in App. A.

heading: 4 Method
text: We propose the RGCN-RCI framework for IMTQA, shown in Figure 5. RGCN-RCI consists of an RGCN-based cell type classification module and an RCI-based table question answering module. CTC module uses an RGCN to aggregate neighbour cell information and predicts cell’s functional
role. Based on the predictions of CTC module, TQA module adopts an improved RCI model to predict whether a row or column contains the answer or not. Final answer cells are selected from intersections of positive rows and positive columns.

heading: 4.1 Cell Type Classification Module
text: We convert the table into a graph, where nodes are table cells, and neighbouring relations between cells are edges. The resulted graph is processed by an RGCN to predict cell types.
Cell Features The initial node representation vector consists of two parts: hand-crafted features and semantic features extracted from the pretrained language model. We select 24 available manual features from Koci et al. (2016) such as the cell text length (listed in App. B.3). The 24-dim integer vectors are transformed into 32-dim continuous numerical vectors by a trained auto encoder. We input cell text to the pre-trained BERT (Devlin et al., 2019) and take the 768-dim output vector of the special [CLS] token as semantic features of the whole cell. In the end, hand-crafted features and semantic features are concatenated to produce 800-dim initial node representation.
Edges We design four directed edges which point from each cell to its surrounding neighbour cells: top to down, left to right, down to top, right to left. We argue that neighbour cell information is important for predicting cell functional roles. For example, most data cells are surrounded by other data cells. As the table is converted to a heterogeneous graph, a relational graph convolutional network (RGCN) (Schlichtkrull et al., 2017) is used to aggregate neighbour cell information in different orientations and updates node representations. The updated node representations are input to the final linear layer to predict cell types.

heading: 4.2 Table Question Answering Module
text: RCI (Glass et al., 2021) is a state-of-the-art TQA model for Lookup questions. It concatenates a text sequence representation of each row (or column) to the question text, and uses a pre-trained language model like ALBERT (Lan et al., 2020) to predict whether the row/column contains the answer or not. Cells on the intersection of positive rows and positive columns are final answers. When constructing the textual representation of each row (or column), RCI incorporates the structure of vertical table. The row textual representation is the concatenation of “column header : cell text”, and the column textual representation is the concatenation of column header and all cell texts in this column.
However, this representation method does not fit other types of tables. Due to the inability to understand implicit and diverse table structures, this method may include irrelevant headers or miss useful header information. For example, when constructing the representation of the third row in Figure 1 (b), RCI treats all cells in the first row as column headers and gives wrong textual representation:
Name : Listed height | Tim Duncan : 6 ft 11 in (2.11 m) |
where “:” is a delimiter token between column header and cell text, and “|” is a delimiter token between different cells. When building the representation of the fourth column in Figure 1 (d), it will lose relevant headers in the third column and output representation without necessary information:
Major Parameter | 37 | 120±23 V | 700A | 624kV |
To overcome the defect of RCI’s original textual representation method, we propose a new representation method with the help of predicted cell types from CTC module and cell ID matrix. Specifically, when constructing row textual representation, we locate the nearest column attribute and column index for every data cell in this row. These column headers are regarded as relevant headers and will be concatenated with the corresponding data cell. Similarly, when constructing column textual representation, the nearest row attribute and row index will be concatenated with the corresponding data cell in this column. Again, let’s take the third row in Figure 1 (b) as an example. Based on CTC module’s predicted result and cell ID matrix, “Tim Duncan” is the nearest column index to the data cell “6 ft 11 in (2.11 m)”, and “Name” is a row attribute irrelevant to cells in the third row. The new row textual representation is:
Listed height | Tim Duncan : 6 ft 11 in (2.11 m) |
For the fourth column in Figure 1 (d), its new column representation is:
Major Parameter | On-Load Tap Changer of #1 Transformer - Tap number : 37 | On-Load Tap Changer of #1 Transformer - Voltage regulation range : 120±23 V | . . . |
where “-” is a delimiter token between index cell and attribute cell. Compared with original textual representation method, our proposed method helps RCI exclude irrelevant headers and include useful headers, which contributes to final predictions.

heading: 5 Experiments

heading: 5.1 Cell Type Classification

heading: 5.1.1 Experimental Setup
text: Considered baselines are as follows. Random Forest (Koci et al., 2016): A random forest classifier is used to conduct CTC task based on manual features. CNN-BERT (Dong et al., 2019): A method using BERT to extract semantic features and a CNN to learn spatial correlations between cells. Bi-LSTM (Ghasemi Gol et al., 2019): Two bidirectional LSTM are used to capture dependencies between different cells, one observing the sequence of cells in each row, and the other in each column. MLP: Directly applying a two-layer neural network to cell features to predict cell types without neighbour information. RAT (Shaw et al., 2018): The Relation-Aware-Transformer whose self-attention mechanism is extended to consider the edge label between each pair of nodes. In this setting, the table can be seen as a directed complete graph where each cell can communicate with all the other cells to update its node representations. Implementation details can be found in App. B.1.

heading: 5.1.2 Experimental Results
text: Table 3 shows CTC results on all tables and complex tables, averaged over 5 repeated experiments. Models are evaluated with F1 score. Macro F1header represents the macro F1 on four header types. Gap represents the Macro F1 difference between all tables and complex tables.
From the results shown in Table 3, we can observe that: (1) RGCN achieves the best macro F1 on all tables and complex tables, beating the best-performing baseline Bi-LSTM by 3.5% and 5.1% respectively, which demonstrates the effectiveness of our proposed graph-based CTC model over various table types, especially complex table. (2) CNN-BERT, Bi-LSTM, RAT and RGCN aggregates information from different neighbour
cells. The best performance of RGCN indicates that neighbour information in a local area is most important for CTC task. This is consistent with the intuition that human can determine cell types based on a small part of the table and do not need to read the whole table. (3) Performance on complex tables is worse than overall performance, showing that CTC models struggle in comprehending complex table structures with flexible header locations. More analyses are shown in App. C.1.

heading: 5.2 Table Question Answering

heading: 5.2.1 Experimental Setup
text: We include following TQA baselines. RCI (Glass et al., 2021): A state-of-the-art model with its original textual representation method. RCI-AIT (Katsis et al., 2022): A variant of RCI for AIT-QA dataset with the textual representation method designed for hierarchical tables. TAPAS (Herzig et al., 2020): A classic table pre-training model which takes the linearization of question and table cells as input and outputs token representations for answer cell selection. Tapex (Liu et al., 2022): A recent table pre-training model based on BART (Lewis et al., 2020) which can directly generate answer text given the flattened table and
the question. Ernie-Layout (Peng et al., 2022): A visually-rich document understanding pre-training model which conducts TQA via prompting. RAT: Table cells and question are converted to a graph and cell representations are updated by an RAT model. We follow Mueller et al. (2019) to use cell-level relations such as cell to column header. Human: We evaluate the average performance of five annotators with bachelor degree. Implementation details are shown in App. B.2.

heading: 5.2.2 Experimental Results
text: Table 4 shows exact match accuracy on the IMTQA test set, averaged over 5 repeated experiments. “RGCN-RCI” denotes our proposed framework. “RGCN-RCI + Oracle headers” represents using new representation with annotated headers.
From the results shown in Table 4, we can find that: (1) Existing TQA models struggle on our dataset. The best baseline RCI-AIT achieves an overall accuracy of 49.6%, and its accuracy on complex tables is only 23.8%. This shows that models designed for specific table types cannot achieve a great generalization on multi-type tables, especially previously ignored complex tables. (2) Compared with baselines which lack the ability to comprehend implicit and diverse table structures, RGCNRCI framework achieves better performance and improves the accuracy on all tables and complex tables by 3.8% and 8.2% respectively, which demonstrates the effectiveness of our proposed textual representation method. RGCN-RCI + Oracle headers can achieve more performance boost, which further proves that correctly understanding table structure is beneficial to question answering over implicit and multi-type tables. A case study is shown in App. C.3. (3) Table pre-training models like TAPAS and Tapex still cannot generalize well on various table types. This is probably because they were originally designed for and only pre-trained on vertical tables with explicit structures available. Considering more diverse table types in pre-training stage may improve their performance on multi-type tables. (4) Interestingly, Ernie-Layout performs better on complex tables than other tables. An explanation might be that it is biased by layout information from document where answer text is often next to the keywords in reference text, which makes it more difficult for the model to locate answers that are far from their header cells. More discussions are given in App. D. (5) The performance of RAT is poor because
it treats input tables as vertical tables to build the graph. When handling other tables, the resulted graph will contain wrong relations between different cells which hinder the model from locating correct answers. (6) Compared with human baseline, even with an oracle providing correct headers, RGCN-RCI only achieves 33.1% accuracy on complex tables. Therefore there is a long way to go for TQA models to accurately find correct answers in complicated tables. More analyses are shown in App. C.2.

heading: 5.2.3 Error Analysis
text: We randomly selected 100 error cases of RGCNRCI framework to conduct error analysis. Error cases fall into four categories: (1) Row mistakes(40%): model fails to select the correct row which contains answer. (2) Column mistakes(28%): model fails to select the correct column. This shows that predicting the correct column is usually easier than predicting the correct row. (3) Row and column mistakes(14%): model selects wrong row and wrong column at the same time, which is common in complex tables(57%). (4) Missing rows or columns(18%): model mistakenly predicts that none of rows or columns contains answer, which often results from the paraphrased question expression such as synonyms. Model may need external knowledge to perform better on these questions.
We also analyse the performance of RGCN-RCI on different question types. The exact match score (51.2%) on the questions whose answers are arbitrary several cells is lower than that on the other questions whose answers are single cell (54.9%), cells in one column (55.6%) and cells in one row (55.0%). This shows that it is more difficult for the model to find answer cells with flexible and discontinuous locations.

heading: 5.2.4 Ablation Study of Header Cells
text: To analyse the contribution of different header cells to the final TQA prediction, we conduct ablation study by removing different headers when constructing new textual representation for each row and column. We use different textual representation to train and evaluate RCI models. As we can see from the results shown in Table 5, header cells have different contributions to the final TQA results. The attribute header cells are more important than index cells as they are most significant for understanding simple data cells. And the row attribute cells seem to be more important than column ones.
If we remove all header cells and simply concatenate every cell to build row/column representation, the model performance would suffer a dramatic decrease to 46.3% which is worse than original RCI method designed for vertical tables. This further validates that header cells are beneficial for RCI model to locate the correct answer cells.

heading: 6 Related Work
text: Table QA methods can be categorized into two types: semantic-parsing-based methods which transform the question into an executable logical form such as SQL (Wang et al., 2020a; Hui et al., 2022; Guo et al., 2019), and non-semantic-parsing methods which directly output final answers without generating logical forms (Yang et al., 2022; Liu et al., 2022). Researchers also proposed various TQA datasets (Iyyer et al., 2017; Chen et al., 2020). Most of previous datasets consist of vertical tables with regular structure except HiTab (Cheng et al., 2022), AIT-QA (Katsis et al., 2022) and MultiHiertt (Zhao et al., 2022), which also consider hierarchical tables but ignore other table types especially complex tables. By contrast, IM-TQA is the first dataset that include tables with implicit and multi-type structures.
Cell type classification is crucial for table structure understanding, which aims at recognizing table cells’ functional roles. Previous work proposed different taxonomies of cell types, which mainly focused on spreadsheet tables (Dong et al., 2019; Zhang et al., 2021; Koci et al., 2016). Coarsegrained taxonomies classify table cells according to their roles in basic table structure (Sun et al., 2021), e.g., header, metadata, data. Fine-grained taxonomies will subdivide header cells or data cells according to more specific functions (Zhang et al., 2021). The proposed taxonomy belongs to coarsegrained taxonomies and we focus on headers that are helpful to locating answers.
Document Understanding is another interest-
ing and related research direction, which requires model to answer questions over a document image that may also contain tables (Borchmann et al., 2021; Zhu et al., 2022; Huang et al., 2022). Compared with TQA models which usually consider table as semi-structured text data, document understanding models consider table as a part of document picture and try to understand table structures based on visual information. These actually represents two different perspectives to tackle table data. We give more discussion about document understanding in App. D.

heading: 7 Conclusion
text: We propose a new problem, table question answering over implicit and multi-type table structures, and construct a corresponding dataset named IMTQA by collecting tables from various domains. Tables in our dataset range from the traditional simple table with fixed headers to the complex table with flexible headers, which poses a new challenge to previous methods. Besides QA pairs, we also annotate functional roles of table cells to promote the understanding of implicit table structures. In experiments, we benchmark recent methods on CTC and TQA tasks on IM-TQA, and propose a two-stage RGCN-RCI framework that outperforms existing methods. Experimental results show that IM-TQA can provide a challenging and valuable testbed for future research.

heading: 8 Limitations
text: Our proposed dataset is in Chinese and focuses on single tables. Though we translate the dataset from Chinese into English, we think it is better to directly construct a corresponding large-scale English TQA dataset in consideration of data quality. To build such a dataset with limited resource, one can fully utilize abundant tables in existing English TQA datasets. As an exploration of table question answering over implicit and multi-type structures, this work focuses on Lookup questions and we leave annotations of Aggregation and other numerical reasoning questions in the future work.
The proposed RGCN-RCI framework is not an end-to-end model. We look forward to seeing more end-to-end models which can simultaneously learn table structures and question answering with the help of our dataset. In addition, it takes more time to convert very large tables into graph structures which are used to recognize cell functional roles.

heading: 9 Ethical Considerations
text: Our proposed benchmark is a free and open resource for the community to study table question answering over implicit and multi-type tables. We follow existing works AITQA (Katsis et al., 2022) and DuSQL (Wang et al., 2020b) to select data sources when building our dataset. Tables are collected from the public organization China Securities Regulatory Commission and Baidu Encyclopedia of Baidu Company. We got the reply and permission from these two organizations which allow us to share and redistribute data for noncommercial use. Tables in these annual reports and web pages are open to public so there is no privacy risk.
In the annotation process, we asked annotators to check if there exist any offensive content such as insulting or discriminatory speech. They did not find any such content in our benchmark. We also checked for identifiers and replaced identifying information with mono-directional hashes. We recruit 10 professional annotators with bachelor degree (6 males and 4 females) and pay them at a price of 6.5 dollars per hour (above the average local payment of similar jobs). The total time cost for annotation is 1,200 working hours. Annotators were informed that these labeled data would be used as a table question answering dataset. The data collection protocol was approved by an ethics review board of an IT company. Main experiments in this paper can be run on a single NVIDIA GeForce RTX 3090 GPU. Our dataset follows the Computational Use of Data Agreement v1.06.

heading: Acknowledgements
text: This work was supported by National Natural Science Foundation of China (No. 61976207).

heading: A More Information about Dataset Construction and Statistics
text: Table 6 shows IM-TQA’s basic statistical information such as average question number, row number, column number per table. We also count the number of distinct table structures. We think that table structure is represented by the distribution of locations of header cells. If header cells of two tables are distributed in the same way, i.e., same cell type at the same table position, we consider they share the same table structure. As expected, due to much more flexible header locations, structure number of complex tables (241) and hierarchical tables (153) are larger than that of vertical tables (103) and horizontal tables (89). Table 7 shows number of different table headers.
In our formal annotation process, every annotator handled annotation of different samples to meet the annotation schedule. To further quantitatively study the inner-annotator agreement, we randomly selected three annotators and asked them to annotate the same 100 samples in two annotation tasks. The resulted Fleiss’ Kappa of cell type annotation task and answer cell annotation task are 0.86 and 0.88 respectively, which can be interpreted as ‘Almost perfect agreement’ (Landis and Koch, 1977).
B Implementation Details
B.1 Implementation Details for CTC Experiments
In this paper, we use PaddlePaddle7 and PGL8 to implement our model and build graphs. We use valid set for model selection and hyper-parameter
7https://www.paddlepaddle.org.cn/ 8https://github.com/PaddlePaddle/PGL
tuning, and then evaluate the best model on test set. We use bert-base-chinese to extract cell semantic features. For the RGCN model, the graph neural network (GNN) layer number is 4, the dimension of hidden layers is 800, and the ReLu activation is used between adjacent GNN layers. We train RGCN for 20 epochs. The Adam optimizer is adopted with learning rate of 1e-4. To mitigate the class-imbalance in the CTC task, we follow Ghasemi Gol et al. (2019) and use a weighted Cross Entropy Loss as our loss function. We set the weight wk of cell type k to be inversely proportional to the number of cells with class type k in train set ntraink , i.e., wk = 1−
ntraink∑5 k′=1 n train k′ .
For RAT baseline, we follow the implementation in Wang et al. (2020a). The RAT layer number and the attention head number is 8. The probability for dropout layers is set to 0.1. The AdamW optimizer is adopted with learning rate of 2e-5, warmup fraction of 0.1 and weight decay of 0.01. We train RAT for 50 epochs with batch size of 16. For MLP baseline, we adopt a multilayer perceptron of one hidden layer with hidden size 800 and ReLu activation. For CNN-BERT baseline, we adopt a text CNN (Kim, 2014) on cell feature matrix along the row direction. For other baselines, we follow the original experimental setup in their papers. PSLRF (Sun et al., 2021) is a very recent CTC model, but the code has not been publicly available. Thus, we do not compare with this method.
B.2 Implementation Details for TQA Experiments
We follow the original paper to train RCI model with our proposed textual representation method. We use bert-base-chinese as the sequence-pair classifier, which is trained for 3 epochs with batch size of 64. The AdamW optimizer is adopted with learning rate of 2e-5, warmup fraction of 0.1 and weight decay of 0.01. The probability of dropout layers is set to 0.1. The max grad norm is set to 1.0. For RAT baseline, to achieve better performance in multiturn TQA task, Mueller et al. (2019) also introduce a node in the graph representing the answer of last question. We do not use these unnecessary designs. For TAPAS baseline, We pre-train a TAPAS-base model on about 6 million Chinese tables. The pretraining was run on four Tesla V100 GPU for about one week. The table corpus used for pre-training was collected from Baidu Encyclopedia. In fine-
tuning stage, we concatenate output representations of the first and the last character in the cell text as cell representation, which is used to predict the probability of selecting the cell as answer. Note that out of resource limitation, we only pretrained TAPAS for once. For Tapex baseline, we are unable to pre-train a new Tapex model for Chinese due to the lack of synthetic Chinese SQL corpus. As an alternative, we adopt a commercial machine translation model9 to translate tables and questions in IM-TQA from Chinese into English and finetune a pre-trained Tapex-base model with the official fine-tuning script. But it should noted that the performance of the model maybe influenced by the translation quality. For Ernie-Layout baseline, we transform tables into document images and adopt the official released model to evaluate its performance on our dataset. For RCI-AIT baseline, we use the same transformations from the original paper to transform all tables into vertical tables. Transformations include taking row headers as table cells in a new column and flatting hierarchical headers by concatenating parent header text with children text. Main model parameter number and training time are listed in Table 8. Main experiments in this paper can be run on a single NVIDIA GeForce RTX 3090 GPU.
B.3 Complete Manual Features
We list all manual features in Table 9. We consider 15 content features and 9 spatial features. We follow Ghasemi Gol et al. (2019) and train an auto encoder to transform 24-dim integer vectors into 32-dim continuous numerical vectors. We train the auto encoder by a vector reconstruction task. The auto encoder tries to reconstruct the input integer vector at its output, and generates continuous vector representations at the output of the encoder
9https://cloud.baidu.com/product/mt/text_trans
layer. Mean square error is used as loss function. At test time, we feed the 24-dim integer manual vectors into the trained auto encoder and gain the 32-dim continuous vector from the encoder output.

heading: C More Experimental Analyses
text: C.1 More CTC Experimental Analyses
(4) The performance of MLP baseline is the worst among methods based on neural networks, which proves that neighbour information is essential for distinguishing divergent types of cells. (5) F1 scores on the column index (CI) and row index (RI) are lower than that on the column attribute (CA) and row attribute (RA). This indicates that, compared with attribute cells that are not data, it is more difficult for CTC models to distinguish index cells from pure data cells. (6) CTC results on vertical, horizontal and hierarchical tables are shown in Table 10, Table 11 and Table 12, which demonstrates that our graph-based method achieves a better generalization on tables of different types.
C.2 More TQA Experimental Analyses
(7) As a non-table-pre-training method, RCI beats the TAPAS on all model types, which is consistent with the results in Glass et al. (2021). This shows that selecting answer cells based on column and row information maybe better than directly predicting whether a cell is the answer or not. This is more close to the way that human locates answer cells. (8) RCI-AIT achieves better performance than RCI especially on hierarchical tables, which shows the effectiveness of its special textual representation method. But it cannot perform well on all table types as its special representation method aims at hierarchical table and may include irrelevant information when applied to other tables. (9) Compared with TAPAS, the recent Tapex model achieves worse performance. We suppose the reason is that TAPAS contains extra position embeddings such as column/row ID embedding, which may benefit the model to understand diverse table structures to some extent. By contrast, Tapex is totally based on BART (Lewis et al., 2020) architecture and does not include such special designs, which increases difficulty for the model to understand various table structures except vertical tables. In addition, Tapex autoregressively outputs answer(s) separated by commas. We think it is also difficult for this fashion to generate answer text in multiple answer cells.
C.3 TQA Case Study
Figure 6 shows a QA sample on a complex table containing parameters of an air filter. In this case, original RCI cannot correctly determine which column contains the answer as its representation of the fourth column lacks necessary row header infor-
mation, such as “Operating Temperature Range”. By contrast, RGCN-RCI with our new textual representation method could incorporate useful row header information into its column representation and finally locates the correct answer.

heading: D Discussion about Document Understanding and LayoutLM
text: Document understanding aims at converting a document into meaningful information, and it involves several subtasks including answering questions over tables in a document. Document understanding benchmarks (Borchmann et al., 2021; Zhu et al., 2022; Mathew et al., 2020) emphasize that no structured representation of the underlying document text is provided, such as a table structure given in advance, and it has to be learned by models from the input document file. From this perspective, document understanding requires a more general “implicit” structure understanding ability which needs to handle more data structure types, such as tables, graphs and lists and so on.
By contrast, IM-TQA mainly focuses on implicit structure understanding over different tables. However, IM-TQA considers more diverse table types and table structures than document understanding benchmarks like DUE (Borchmann et al., 2021) and TAT-DQA (Zhu et al., 2022). For instance, DUE mainly considers vertical tables from the WTQ (Pasupat and Liang, 2015) dataset and TAT-DQA considers vertical or hierarchical tables from the TAT-QA (Zhu et al., 2021) dataset. We believe that it is meaningful for document understanding benchmarks to include more table types and we hope IM-TQA could also serve as a useful resource to achieve this target.
LayoutLMs (Xu et al., 2020b,a; Huang et al.,
2022; Peng et al., 2022) are important and recent methods for the Document Understanding task, which shares some similarity with our motivation of understanding implicit table structures. LayoutLM jointly models interactions between text and layout information across scanned document images and it supports several downstream document understanding tasks including answering questions over tables in a document. We converted tables into document images and investigated the performance of a state-of-the-art Chinese LayoutLM Ernie-Layout (Peng et al., 2022).
From the results shown in Table 4, we find that the Ernie-Layout model struggles in dealing with diverse table structures. Interestingly, its performance on complex table is better than that on other table types. We suppose the reason is that the LayoutLM is biased towards excessively utilizing layout information from the scanned document images. For instance, in a registration form of personal information, given a key (e.g., “ID number”), its corresponding value is much more likely on its right or below rather than on the left or above. The LayoutLM tends to use such layout information to answer questions about document but such biased information may distract the model from locating the correct answer cells in the table. As a result, the model performs better on complex tables where lots of answer cells are next to their header cells, but it struggles on other tables where answer cells are relatively farther from their header cells.
E Instructions for Annotators
Screenshots of original instructions for annotators have been saved and shown in this section. The instruction for CTC annotation task is shown in Figure 12. And the instruction for TQA annotation task is shown in Figure 13. A real CTC annotation case of complex table is shown in Figure 14. Besides instructions, we also provided annotators with sufficient QAs to ensure that they fully comprehended the annotation requirements.

heading: F More Table and Question Examples
text: Figure 7, Figure 8, Figure 9, Figure 10 and Figure 11 depict more table examples and question examples. We translated them into English for reading convenience. Different header cells are represented with the same color as Figure 1. As demonstrated in these examples, header cells of complex table are more flexible than those of traditional table. They
may appear at arbitrary positions and can be mixed with other data cells. Such table structures have not been thoroughly investigated and challenge existing TQA methods.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:

heading: 3 A1. Did you describe the limitations of your work?
text: Section 8 Limitations.

heading: 3 A2. Did you discuss any potential risks of your work?
text: Section 9 Ethical Considerations.

heading: 3 A3. Do the abstract and introduction summarize the paper’s main claims?
text: Abstract and Section 1 Introcution.
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? Section 3 Dataset Construction and Section 4 Method.

heading: 3 B1. Did you cite the creators of artifacts you used?
text: Section 3 Dataset Construction and Section 4 Method.

heading: 3 B2. Did you discuss the license or terms for use and / or distribution of any artifacts?
text: Section 9 Ethical Considerations.

heading: 3 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided
text: that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Section 3 Dataset Construcution and Section 9 Ethical Considerations.

heading: 3 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any
text: information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Section 3 Dataset Construcution and Section 9 Ethical Considerations.

heading: 3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and
text: linguistic phenomena, demographic groups represented, etc.? Section 3 Dataset Construcution and Appendix A.
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Section 3 Dataset Construcution.
C 3 Did you run computational experiments? Section 5 Experiments

heading: 3 C1. Did you report the number of parameters in the models used, the total computational budget
text: (e.g., GPU hours), and computing infrastructure used? Section 5 Experiments and Appendix B.
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.

heading: 3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found
text: hyperparameter values? Section 5.1.1, 5.2.1 and Appendix B.

heading: 3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary
text: statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Section 5.1.2, 5.2.2 and Appendix C.

heading: 3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did
text: you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Appendix B.
D 3 Did you use human annotators (e.g., crowdworkers) or research with human participants? Section 3 Dataset Construcution, Section 9 Ethical Considerations and Appendix E.

heading: 3 D1. Did you report the full text of instructions given to participants, including e.g., screenshots,
text: disclaimers of any risks to participants or annotators, etc.? Section 9 Ethical Considerations and Appendix E.

heading: 3 D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students)
text: and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? Section 9 Ethical Considerations.

heading: 3 D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to
text: crowdworkers explain how the data would be used? Section 9 Ethical Considerations.

heading: 3 D4. Was the data collection protocol approved (or determined exempt) by an ethics review board?
text: Section 9 Ethical Considerations.

heading: 3 D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data?
text: Section 9 Ethical Considerations.

heading: Future Work
text: As an exploration of table question answering over implicit and multi-type structures, this work focuses on Lookup questions and we leave annotations of Aggregation and other numerical reasoning questions in the future work. The proposed RGCN-RCI framework is not an end-to-end model. We look forward to seeing more end-to-end models which can simultaneously learn table structures and question answering with the help of our dataset. In addition, it takes more time to convert very large tables into graph structures which are used to recognize cell functional roles