Venue: NeurIPS 2021 Track Datasets_and_Benchmarks Round1

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: The study presented a domain-specific corpus that adopts a hierarchical row and column headers, which makes a good contribution to Table Question Answering Systems. 
strengths: I believe the paper has made a new contribution, given the following points:
•	Well written, clear and nicely structured. It is self-contained with background to the study and methods used.
•	The notations used are consistent.
•	The evaluation to the study are clearly communicated.
•	The ablation to the study is clearly exposed.

weaknesses: The data produced is too small, a dataset of 515 questions are indeed inadequate. What are the plans to improve this?
correctness: The paper is clear and concise.
clarity: The paper is well written.
relation to prior work: In relation to previous works, the ideas were clearly communicated.
additional feedback: What are plans put in place to improving the dataset?
What are future contributions to this work?
review_title: An Airline QA dataset 

final_score: rating: 5: Marginally below acceptance threshold
review_content: summary and contributions: This paper proposes a new QA dataset which consists of tables in domain-specific documents. These tables often have more complex layout such as hierarchical row and columns, and specialized vocabulary, compared to the table collected from open-domain content which usually have a flat layout. The authors provide questions that requires understanding of the hierarchical structure of the tables. Experiments show that current state-of-the-art table QA approaches do not perform well on the proposed dataset as they are not designed for complex table layout. The authors also demonstrate different pre-processing methods to improve the performance of the existing methods, which include flatting the table and transposing the tables.
strengths: This paper reveals that the current state-of-the-art table QA approaches do not handle the table with more complex layout well which is important in many real-world applications. This dataset may motivate future research in this direction.
weaknesses: This dataset is very small and domain-specific as it only contains 116 tables from 13 documents and 5 airline companies. So the practicality of the current dataset seems to be very limited and it is not yet suitable to use it to accurately evaluate the performance of a model.
correctness: The claims seem to be correct.
clarity: Yes
relation to prior work: Yes
additional feedback: N/A
review_title: Complex table QA dataset

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: While there are already multiple table QA datasets, this paper presents a new domain-specific table QA task. It has the following unique properties:

1. The tables have hierarchical row and column structures.
2. It involves domain specific vocabularies which might introduce additional challenges than Wikipedia. 

strengths: 1. Domain-specific and complicated tables are practical in many industry applications and this dataset is a step towards the wide applications of table QA methods. 
2. The analysis on different aspects is interesting and informative to future work. 
3. It is unknown whether the small domain-specific dataset could be valuable for other industry domains. 
weaknesses: 1. The dataset is small which can only be used for zero-shot experiments. 
2. It is unclear where the main challenges come from. Is it the domain gap or mainly the structures of the tables. 
correctness: The data and the baselines are reasonable. 
clarity: The details of the dataset and the baselines are clear. 
relation to prior work: The distinctions from existing work are discussed. 
additional feedback: N/A
review_title: The first Table QA dataset that explicitly captures tables with complex structure, including column and row header hierarchies.

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: summary and contributions: The paper presents a new evaluation dataset for models that answer questions over tables. Both the questions and the complex tables - with a hierarchy of row and column headers - are collected from a narrow domain and all questions are authored by human experts, which makes the benchmark more realistic but also rather small because such data are expensive to collect.
Current SOTA table QA models trained on Wikipedia tables perform rather poorly on the new benchmark, showing the particular challenge of the benchmark.
The dataset features many interesting additional annotations, allowing for further analysis, such as human-generated question paraphrases (how consistent is a model in answering questions correctly?) and a boolean value indicating whether or not the header hierarchy is necessary to find the right answer unambiguously.

(modified: 20 Jul 2021)
The rebuttal confirms my impression that this paper presents a valuable dataset with various possibilities for evaluation for an important task whose community has been focusing on simple table structures too much. My recommendation stays 'clear accept'.
strengths: - Good explanation of the need for a QA dataset over *complex* tables
- All questions are human-authored, both with and without looking at the table. This makes the task very realistic.
- Even though the benchmark only contains "simple" lookup questions, SOTA models do not achieve high performance, which shows the potential of this new benchmark to drive progress for this task.
- Asking for paraphrases of the questions is a neat trick to test for model consistency, a very important model property. In other words, the dataset provides high-quality human-authored paraphrases to test for consistency.
- Additional annotation saying whether the header hierarchy is needed for answering the question or not allows for interesting analysis.
- High impact experiment on consistency of evaluated models, an important topic in natural language processing - especially in the context of pretrained transformer language models, such as BERT.
weaknesses: - rather small; so no training/fine-tuning is possible. Few-shot approaches are not considered.
- very narrow domain; the tables only contain numerical data as far as I could tell. This is not a big weakness but it still makes the evaluation very narrow.
- There is no information on inter-annotator agreement. But I suppose that the second round of validating annotators simply reached consensus because there is an easy way to tell what is the right answer for a given question-table pair. Am I correct?
- The baselines are all evaluated on transformed versions of the data. There is no empirical evidence that the transformations help. Why did you not evaluate the baselines on the raw tables for comparison? L.281 says "... This *should* help the models ..." (emphasis mine). Did you test that?

after rebuttal: All weaknesses were adequately addressed in the rebuttal. Most can be solved by clarifying the description in the text. The size is not a major weakness but remains the only mentionable one.
correctness: The data collection process and the evaluation seems correct. Just a few minor points (see Weaknesses for details):
- information on inter-annotator agreement
- no baseline evaluation on the raw tables

All of the above has been addressed and explained in the rebuttal. It just needs to be clarified in the final version of the paper.
clarity: The paper is very well written and understandable. The arguments for the need of the new dataset are presented eloquently.
relation to prior work: The related works section explains previous work well and table gives a detailed overview. It is especially important to look at other domains than Wikipedia.
additional feedback: - It would be nice to see an evaluation counting how often a model changes the predicted answer when evaluated on different paraphrases of the same question. This measures consistency directly (and independently of QA performance).
review_title: Small evaluation dataset to challenge SOTA table QA models

Meta-review: Reviewers are mixed. Reviewers note the novel aspect of the dataset as a strong positive, but are concerned about the small size. The AC finds the data size be of significant concern, even if it is just used for evaluation purpose. The test/validation set used for evaluation should have sufficient samples to cover diverse cases, to allow statistically meaningful comparisons, and to facilitate analysis of failure modes. It is not clear if 515 is sufficient even for this purpose. Therefore AC does not recommend acceptance in its current form. 
Decision: Reject