Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a new table pre-training approach. Different from previous masked language approaches for table pre-training. The authors propose to pre-train (or train) BART on the synthetic SQL corpus. The input for pretraining is the concatenation of SQL query and table while the output is the results by executing the SQL query through the SQL execution engine. Results show that the proposed pretraining approach reduces the gap between pre-training and fine-tuning on table understanding tasks including table-based question answering and table-based fact verification -- it outperforms BART and baselines on four datasets.
main_review: Strength:
- Formulating table pre-training through learning via synthetic SQL corpus is a novel idea. 
- The proposed new pretraining paradigm of table pretraining is simple while effective. Moreover, the authors show its efficiency on a much smaller corpus.

Weakness:
- Compared with naming it as 'pre-training', the proposed approach should be named as data augmentation from constructed synthetic table corpus. It is intuitive to see the improvement of performance since the input and output format between synthetic corpus and the downstream task (QA, fact verification) are the same. Therefore, there is no wonder a huge improvement is received. This work is less interesting and novel from this perspective.
summary_of_the_review: I lean to accept this work since it provides a new view of pretraining and the proposed approach receives significant improvement on four datasets.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposed `TAPEX`, a table pretraining method that conducts pretraining via learning a neural SQL executor over a synthetic corpus. Such corpus is obtained by automatically synthesizing executable SQL queries and execution results. By selecting a diverse, large scale and high quality synthetic corpus, the method claimed to address the data scarcity challenge. Then the executor could be further fine-tuned for downstream tasks, and demonstrate significant improvements over previous state-of-the-art work on table question answering and fact verification datasets.
main_review: Strengths:
1. The paper is very clear and informative, well-written with abundant figures & tables for better understanding. The introduction to pretraining / fine-tuning methods is well arranged. Thanks for the effort!
2. The idea of learning a SQL executor for pretraining is simple yet effective. It’s kinda novel to use it as a way of pretraining and the authors conducted many experiments to demonstrate its effectiveness.
3. The authors conducted many ablation studies about how much the volume of pretraining data impacts the final result, where does the performance gain come from and how does the dataset impact the results. The experiments further confirmed the effectiveness of the method.
4. The model and datasets proposed in this paper are publicly available, making it easy to reproduce.
5. The results demonstrate a significant improvement over the previous works.

Weaknesses:
1. The paper has a kinda novel idea on the pretraining method but the problem setting is limited to table understanding, limiting the method to only table related problems.
2. The paper experimented using SQL style query language and the examples seemed simple. While TableQA is a challenging problem, some analysis on the impact of difficulty of queries on the final performance would be appreciated.
3. The claim that the model can reproduce the table structure seems to be a bit questionable. With the attention mechanism, the attention weights are very commonly calculated and shown as in Figure 4. It would be supported if the weights can be accumulated on some rows or columns.
4. While SQL language is structured, natural language questions can be very flexible. Would appreciate it if there are more comparisons on the change of performance if we generate the SQL in a different way or use paraphrase of question.

summary_of_the_review: This paper is well-written and very informative, introducing `TAPEX`, a pretraining method learning SQL executor, with significant performance improvement. The novel design of pretraining is very extensive and proven to be effective on TableQA setting. While some more experiments could be conducted to further understand the performance gain, the current paper has done a lot of ablation study, showing the impact of pretraining data. I recommend this paper to be accepted.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposes a new approach to perform table-pretraining, this new approach is hugely different from the previous approaches like TaPas, TaBERT, etc in two ways: 1) the model architecture does not need to be designed to fit the table structure, no new objective functions have been invented, a simple BART model can be used to keep pre-training with its original encoder-decoder loss, 2) the algorithm does not require large amount of mined table-text parallel corpus, which avoids huge amount of data preprocessing and heuristics to do noise canceling. 

The main technical part of this paper is "small amount of tables" while "large amount of synthesized SQL". The basic idea mimics the previous papers by (Yu et al. 2021) to utilize the clean dataset and templates to generate arbitrary amount of SQL queries and simulate various environments. The BART model is essentially used as a universal approximator to estimate the logical operations for SQL queries. The massively pre-trained TAPEX model indicates its superb SQL execution capability as indicated in Table 6, outperforming the BART without such SQL-approximation pre-training by a huge margin of 20%+. This is quite impressive, and demonstrates the strong approximation capability of BART model even without specially-designed logic-units.

The empirical results are also very impressive, the model matches or surpasses the previous SoTA on all the four datasets, which is definitely a plus for the proposed model.
main_review: The paper is in general very easy to follow with a quite clear structure. The main technical part is quite simple yet effective. I believe simple is a good feature to have. 

The proposed method is similar to the previous papers on "demonstrating transformers are universal approximator". The paper is motivated to  train the BART model to mimic the complex logical operations done by SQL query language. Though the accuracy is still not perfect, with "comparative/arithmetic" only achieving 50% accuracy, the other logical operations like group/select/filter are already pretty good.

I especially like the idea of "not using huge amount of mined noisy data for pre-training". As the huge amount of data can be very noisy and could waste large amount of time filtering and preprocessing. The paper demonstrates that small amount of tables are already quite good enough since the lexical/semantic variance is already captured during the BART pre-training, the model only needs to learn how to reason over the logical operations. This idea is very neat and proves effective. 

There is some minor issues for the paper:
1. the paper mentions in multiple places that their model can address the problem of "how to design an effective pre-training task to exploit the table structure", I do have strong reservation for this point. First of all, the paper still uses the linearization to represent the table in the encoder side, no real structural information is leveraged in the model. The pre-training objective function is also irrelevant to the table structure. Therefore, this claim is flawed. I would prefer "how to design effective pre-training task to approximate the structural/arithmetic reasoning of formal language" would be a much better motivation. 
2. Besides, the proposed algorithm is not specific to tables. I think KB can also be applied as long as there is formal language which can be synthesized. The gist of the paper is "mimicking logical operations in formal language" rather than "incorporating tables structures in pre-training". I would appreciate if the authors could revise their wording in the paper.
summary_of_the_review: To sum up, the paper has strong empirical evidence to support its simple yet effective approach. Though the wording and claims are sometimes not quite accurate, it's still a very good paper. I believe it can arouse lots of interest in the community and persuade more people to abandon "mined large noisy corpus" and turn to "accurate and small corpus". Based on these contributions, I would recommend accepting this paper.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The authors propose a pre-training strategy for Table (structured data) grounded tasks (question answering and fact verification). The main idea is to teach the model to execute SQL queries, by predicting the execution of the query, rather than using a pre-training strategy over the tabular data (e.g., Masked LM pre-training, or masked column pre-training). To elaborate, the authors synthetically generate SQL queries, execute them over tables, and collect the execution output. Then, the model is trained to generate (in an autoregressive way) the execution output given the concatenation of the table and SQL query (Figure 3 for more details). In this way, the model is forced to reason over the table to generate an answer, rather than using a reconstruction loss (MLM) that improve the representation of the input. 

Experiments
The authors use BART as the base model and use the SQL templates extracted in SQUALL (Shi et al. 2020) executed over 1500 randomly selected tables from WikiTable-Questions (Pasupat & Liang 2015) (from the training set and not overlapping with dev and test). The generated input-output pairs are 5M and the accuracy on a held-out is 89.6%. Then the pre-trained model is fine-tuned to 4 datasets (3 tables QA and 1 TableFact), and the results show a consistent and significant improvement on the existing state-of-the-art. Finally, the authors show an interesting analysis on the attention (Figure 4), the efficiency of the pre-trained task (Figure 5&6) and a manual evaluation on 500 samples from WikiTables showing how TAPEX improved over BART on different kinds of QA types (Table 6). 
 
main_review: Strengths
- the proposed pre-training strategy is simple and effective. 
- the performance in the 4 evaluated benchmarks are significantly above the state-of-the-art. 
- the paper provides many insights on the results (Figure 4&5&6, Table 6) and the paper is clearly written. 

Weaknesses
- Table + Query vs text-to-SQL. This is not a weakness for this particular model but more for the general methodology of providing the entire table as input to the model. As the authors mention in the limitation section, the model has a limited input length, thus cannot handle big tables. This could be solved by generating SQL queries rather than learning how to execute them (which for a neural net is very hard), but yet again as mentioned by the authors in the limitation section, the TAPEX pre-training is not effective in this setting. I would not be surprised if the performance of the table as input + query setting is not as good as the semantic parsing one (e.g., text-to-SQL) in scenarios with very complex queries or very large tables. Anyhow, the proposed pre-training is very effective in this setting, so overall, this is not a major weakness in my view. 
- Template for the synthetic generation. Although the performance with template-based data generation is good, I believe this is quite a restriction over the multitude of possible queries that can be generated using grammars. 

summary_of_the_review: A simple and effective pre-training strategy for Table QA. The authors proposed TAPEX, a model trained on executing SQL query rather than MLM, a new SOTA in 4 benchmarks, and many analyses and visualizations.
review_title: null

Meta-review: Reviewers are positive overall -- the is a general consensus towards acceptance. Reviewers viewed the simplicity, novelty, and effectiveness of the propose pre-training approach as strengths. Further, reviewers praised the draft as very clearly written, and viewed experimental ablations as relatively in-depth -- e.g. two reviewers found the additional analysis of impact of data size to be valuable. A few concerns about additional ablations and claims were brought up, but all were adequately addressed in author response.
Decision: Accept (Poster)