Learning Relational Decomposition of Queries for Question Answering from Tables (2024)

Abstract: Table Question-Answering involves both understanding the natural language query and grounding it in the context of the input table to extract relevant information. In this context, many methods have highlighted the benefits of intermediate pre-training using SQL queries. However, while most approaches aim at generating final answers directly from inputs, we claim that there is better to do with SQL queries during training. By learning to imitate a restricted subset of SQL-like algebraic operations, we demonstrate that their execution flow provides intermediate supervision steps that allow for increased generalization and structural reasoning compared to classical approaches. Our method, bridges the gap between semantic parsing and direct answering methods, offering valuable insights into which types of operations should be predicted by a generative architecture and which should be executed by an external algorithm. Our code can be found at https://github.com/ RaphaelMouravieff/Partial-Exec.

heading: 1 Introduction
text: The field of Table Question Answering (QA), which encompasses complex content manipulation tasks like projection, sorting, grouping, and aggregation, presents considerable challenges for Natural Language Processing (NLP). Its complexity and growing relevance across diverse sectors, from business to academic research, have attracted widespread attention. This domain has evolved quickly with the rise of Pretrained Language Models (PLMs), but this field remains challenging for current models (Jin et al., 2022).
Former studies focused on Semantic Parsing (SP) techniques tailored for well-structured and clean table data, as highlighted in (Shi et al., 2020). However, real-world scenarios often involve heterogeneous resources, for example combining both numerical and textual content in some cells, like
in WikiTableQuestions (Pasupat and Liang, 2015). Among the proposed solutions, (Liu et al., 2021) tried to generate directly the answer and therefore bypass the generation of logical forms. Despite this advantage, these methods exhibit limitations, particularly when executing numerical operations (e.g., computing a mean, counting). To cope with this, a natural solution is to propose hybrids that stand as intermediary solutions between semantic parsing and direct generation. For instance, (Herzig et al., 2020; Zhou et al., 2022b) have combined basic table selection methods (e.g., selecting rows and columns, or cells) before computing aggregations or performing basic numerical operations. However, they often fail to address intricate queries necessitating the synthesis of diverse table views and interactions because of the limited expressivity of their underlying algebra.
In this work, we propose to study the continuum between semantic parsing-based and direct generative methods, to leverage the strengths of both. Going beyond previous works, we propose a novel framework that facilitates reasoning over heterogeneous table resources. This framework relies on the definition of an algebra over tables inspired by relational algebra. Based on this algebra, each question in natural language and its corresponding table can be translated into a computational graph. By varying a cut-off criterion that specifies which part of the graph should be computed directly by the model (i.e. direct generation) and which part should be computed outside of it (i.e. semantic parsing), we can study different trade-offs and their effect in terms of effectiveness.
Beyond providing a stronger interpretation of the user query in the context of the table compared to semantic parsing-based (SP) methods, our framework addresses the common execution challenges associated with SP methods, which require clean tables to allow full SQL execution. Our approach predicts operators with associated "clean" operands
10471
from the input, thanks to the generative ability of the Transformer architectures.
To train our model, we leverage a pre-training procedure (Pruksachatkun et al., 2020; Geva et al., 2020; Yu et al., 2020) that helps neural architectures manipulate tabular data before dealing with complex Table QA tasks. This is done by first learning to generate from SQL queries rather than from natural language. On top of that, our main contribution focuses on the fine-tuning of such models when dealing with natural language questions, through the development of a flexible framework that allows partial execution of corresponding SQL queries, enabling effective handling of real-world tables while still obtaining good performance on numerical reasoning questions (Figure 1). Our research addresses the following key questions :
1. Is there an intermediate supervision for table QA that bridges the gap between semantic parsing and direct answer generation, thereby enhancing generalization and structural reasoning capabilities (Table 2)?
2. What types of algebraic operations should be predicted by a generative architecture, and which should be preferably executed by an external algorithm (Tables 3a and4a) ?
3. Does partial execution of SQL graphs help to improve robustness to perturbations in the input tables (Table 3b and 4b) ?
Through our experiments, we demonstrate that, for some intermediate cut-off levels, our approach generalizes better and is more robust compared to direct answer methods, which are usually limited in their structural reasoning capacities. This framework provides insights into the trade-off between different levels of supervision, paving the way for future research and innovation in the field of Table QA.

heading: 2 Related Work

heading: 2.1 Table Question Answering Architectures
text: Table question answering is a very active field with many recent developments. This ranges from specifically designed transformer architectures, with sparse (Eisenschlos et al., 2021) or biased (Golchin and Surdeanu, 2023) attention matrices that capture table structures, or specialized table embeddings as in TUTA (Wang et al., 2021) and GENTAP (Shi et al., 2022), to large
language models (LLMs) that leverage in-context learning to deal with table structures (Chen, 2022; Cheng et al., 2022; Wang et al., 2024). While our study, orthogonal to these directions, could be applied in the context of any family of architectures including LLMs (e.g., fine-tuned using low-rank adaptation (Hu et al.; Dettmers et al.)), we chose to build on compact architectures based on reasonably-sized pre-trained language models (PLMs) such as BERT or BART, as considered in popular recent works TAPEX (Liu et al., 2021) or OmniTab (Jiang et al., 2022). Beyond scalability, such architectures, which do not require specific prompt design that could bias conclusions, offer easier comparison opportunities.1 Finally, we believe that our developed approach, which consists of predicting and using external programs as tools when generating answers (e.g., in the vein of Toolformer (Schick et al.)), are still fully valuable in the context of LLMs, providing increased inference speed and stability. Our work is a step towards showing how tools can be used with structured data like tables, which can be transferred to LLMs in future works.
In the following, we focus on differences between table question answering approaches regarding their output strategies, which is more strongly related to the study of this paper.

heading: 2.2 Output Strategies in Table QA
text: Table Question-Answering models can be distinguished on their answer generation, which is either a formula operating on the table (semantic parsing) or a direct answer (direct generation), or a hybrid of both.
Semantic Parsing Semantic parsing aims to transform natural language into executable queries, primarily SQL. Sketch-based models decompose SQL query construction by breaking down and classifying query components, enhancing structured SQL generation (Jin et al., 2022). Generationbased methods like RAT-SQL (Wang et al., 2019) directly produce SQL queries using an encoderdecoder architecture that considers both the question and the table context for generation. Under weak supervision, (Min et al., 2019) optimize the probability of the correct answers over a set of possible latent representations, facilitating the model’s
1We also note that it has recently been shown in a broader context that LLMs are usually contaminated by evaluation benchmarks (Golchin and Surdeanu, 2023), which could alter the results of our study.
ability to infer correct responses without explicit answer mappings. Another stategy is to use reinforcement learning where the execution result is used as rewards to train models (Zhong et al., 2017). Despite SQL’s effectiveness in QA over tables (Shi et al., 2020), its limitations with non-database tables and question translation are a major drawback. Our approach seeks to transcend these bounds by introducing a logical form independent of the table during execution.
Direct Answer Generation In contrast to semantic parsing, direct answer generation produces final answers, bypassing the step of converting questions into formulas. This directly addresses the limitations of SQL-based systems, enabling the processing of various table formats. For instance, (Mueller et al., 2019) use a GNN-based encoder to encode the table structure and a decoder to output the answers conditioned on the graph and the query. An additional benefit of this method is its compatibility with advanced data augmentation techniques (Eisenschlos et al., 2020). This includes transformations from SQL to its result as in TAPEX (Liu et al., 2021), or from Excel formula to its execution as in FORTAP (Cheng et al., 2021). However, a notable challenge for transformers in this domain is handling numerical reasoning queries effectively (Zhou et al., 2022a).
Hybrid Methods Hybrid methods extract pertinent tokens from tables to create responses, typically employing an aggregator to associate with and route these tokens to a specifically designed executor. TAGOP (Zhu et al., 2021) uses sequence
tagging for extracting relevant cells and a classifier for assembling them into coherent symbolic reasoning programs. TAPAS (Herzig et al., 2020) employs a classifier layer at the end of a BERTlike encoder for selecting content from tables and determining the aggregation operation to apply to it. These methods have good numerical abilities, but however, unlike other output strategies, they have limited expressiveness and struggle with complex multi-aggregation queries (Herzig et al., 2020). Our proposed supervision using intermediate logical form addresses this issue by enabling complex multi-aggregation representations.

heading: 3 Model
text: The goal of Table QA is to find the answer A given a natural language question q posed on a table T . We consider the setting where, at train time, queries in natural language are associated with corresponding SQL queries.
In this section, we describe 1) the manipulated data structures; 2) the algebraic operators that correspond to nodes of SQL execution graphs we consider; 3) our way to build intermediate supervision through partial execution of that graph; and 4) our way for linearizing the resulting graph to form the target of our transformer architecture.

heading: 3.1 Tabular algebra
text: In this section, we describe the algebra, inspired by the relational one (Codd), that we use to represent any operation on tables.
Structures Table Question Answering is the task of finding an answer A from a table T ∈ T , where
T = ( (xr,c)c=1...NTcol ) r=1...NTrow is a matrix of values xr,c, which can be numbers or strings. Differently from relational algebra, we view tabular data as a sequence of tuples which we suppose to be ordered. A table can have a header, which corresponds to a sequence of column names c1 . . . cNTcol . When no header is given, each ci corresponds to the column index, and hT = {ci}N T col
i=1 stands as the set of column names from T . Views on the original table, that correspond to results from algebraic operations, are also considered as table T ∈ T .
Classically, tables only include atomic values. To cope with set aggregations (i.e., involving a group-by operation), we also manipulate group-by tables G ∈ G, where G =( (gr,c)c=1...NGcol ) r=1...NGrow , with each component gr,c corresponding to a set of values. We also note columns boolean matrices as B = (br,1))r=1...NBrow , with br,1 ∈ {0, 1}.
Operators Table 1 describes the different operators that we use to manipulate tables T or group-by tables G, whose behavior can be conditioned on parameters (e.g. “order by” can be ascending or descending). These operators follow roughly standard relational algebra operators and cover a broad range of SQL queries. A notable difference with classical relational algebra, which was dictated by the fact we want to further decompose operations for analysis purposes, is the fact that the selection operation simply corresponds to a filter given a column of boolean values produced by a separated comparison operator and that the order of tuples
is used for comparisons (e.g. >, <) and operations (e.g. +, -).
Translating from SQL to our algebra is straightforward. We rely on the SQLGlot library2 to obtain a parse tree from any SQL query. This parse tree is then translated into a computational graph. Each node n of this graph is denoted as ϕ (xn, [n1, . . . , nK ]) where xn is either a table in T , a group-by table in G or an operator in O (an operator is both the operation, e.g. "limit", and its parameters, e.g. k). In the case of operators, n1, . . . , nK correspond to the arguments of the operators, i.e. other nodes in the computation graph corresponding to its operands, and xn(.) the application of the operator on the corresponding list of child nodes. By abuse of notation, in the following we note n = ϕ (xn, [n1, . . . , nK ]) ∈ X , with X a given set, to denote xn ∈ X .

heading: 3.2 Partial Execution of the computational graph
text: Now that we have defined the data and the algebra, we can present how this can be leveraged to produce various representations. For this, we rely on a graph transduction function v operating recursively on any node n of the graph. That is, given a set of operators O∗ we allow to be executed, v(n) = ϕ(xn(v(n1), . . . , v(nK))) if xn ∈ O∗ ∧ ∀i ∈ 1 . . .K, v(ni) ∈ T ∪ G, and v(n) = n otherwise. In other words, we execute from any leaf to the root of the computational tree every allowed operation in O∗ until execution is blocked (because xn not in O∗ or one of its depen-
2https://github.com/tobymao/sqlglot
dencies cannot be executed). The computation graph can hence be partially executed through this transformation v, allowing for flexible handling of SQL operations, by applying v on the root node.

heading: 3.3 Linearizing the Graph
text: As the computational graph must be generated sequentially, we need to define how to transform it into a sequence of tokens, i.e. how to linearize it. To do so, we use a linearization function that we denote l, which takes a node n = ϕ (xn, n1, . . . , nK) in input and returns a sequence of tokens.
In the case of tables (i.e., when xn ∈ T ∪G), we use a simple markup where we separate rows with the symbol “|” and columns with a comma “,”. In the case of operators, i.e. xn ∈ O, the linearization l corresponds to the name of the operator followed by its parameters. For instance, the sequence LIMIT 1 corresponds to the limit operator with 1 as its parameter.
For operator nodes, we define various linearization schemes depending on the order (pre- or postorder) and the usage of aliases to avoid duplicating the same information (the graph is a directed acyclic graph, but there can be different paths between two nodes since results might be re-used).
Pre-order vs post-order We can either use a pre-order linearization scheme where the operator appears before its operands: lpre(n) = l(xn) ⊕ ⊕i(|| ⊕ lpre(ni)) or a post-order one: lpost(n) = ⊕i(|| ⊕ lpost(ni)) ⊕ || ⊕ l(xn). In both cases, ’||’denotes a separator token and ⊕ concatenation.
Using aliases In the above linearizations, re-used results will be linearized several times. This happens frequently with queries with some aggregation. The problem is that this can result in longer sequences, which in turn might be harder to generate. To tame this problem, we associate each node with a given alias the first time it is linearized (e.g. N13) and use this reference instead of its linearization in subsequent occurrences (see appendix A.1.1 for details).
Finally, tables are linearized either before or after the operators. After some preliminary experiments, we chose this to make the grammar of the sequence more regular for a transformer (not mixing operators and content).

heading: 4 Experiments

heading: 4.1 Dataset and Evaluation Metrics
text: In our experiments, we used the WikiTableQuestions (WTQ) dataset (Pasupat and Liang, 2015), which fulfills all predefined criteria for our study: it is characterized by its provision of complex numerical reasoning questions, tables with missing information, mixed cell types (e.g. text and numbers), and availability of SQL supervision. The SQL annotations supplied by SQuALL (Shi et al., 2020) enable the coverage of approximately 80% of the questions from WTQ in the training and validation sets only. We also experimented with the WikiSQL (Zhong et al., 2017) dataset. As with WikiTableQuestions, WikiSQL provides the necessary supervision for training our model. However, unlike WikiTableQuestions, the questions are much simpler (no order-by, group-by, or arithmetic operators), and the tables are already clean.
Results are reported using the Denotation Accuracy (DA) metric as our primary evaluation criterion. DA checks if the execution of the predicted answer is equal to the target answer. When the answer is a list of results, DA disregards the order (i.e., set equality). We decomposed this metric into two categories: the Strict Denotation Accuracy (SDA), which is the traditional one used, and the Flexible Denotation Accuracy (FDA), which compares results after removing units (years, $, kg, etc.). The choice to employ both SDA and FDA stems from our dependence on external tools’ APIs for execution. As a result, our execution outcomes are unit-less, and using SDA would hide the improvements brought by our model – note that we could extend our method to generate an arbitrary sentence containing the result in future works.

heading: 4.2 Inputs and outputs
text: The query encoding is straightforward but table encoding presents a challenge due to its inherent structure. We follow TAPEX and OmniTAB (Liu et al., 2021; Jiang et al., 2022), and represent the transformed table as T ∗ = [HEAD], c1, ..., cN , [ROW], 1, r1, [ROW], 2, r2, ..., rM . The tokens [HEAD] and [ROW] delimit the table’s header and row sections, respectively, with subsequent numbers indicating row indices. Additionally, we use a vertical bar | to delineate headers or cells in separate columns. We then concatenate the query with the linearized table as the input to the encoder.
Outputs in our model correspond to linearized
computational graphs. We considered 42 experimental conditions. First, we use one of the following seven sets of operators as O∗: (P) Only projection operators; (+C) P with comparison operators; (+S) +C with selection operators; (+GB+H) +S with group-by and having; (+A) +GB+H with aggregations; (+OP) +A with operators; (Full) with all operators, i.e. as TAPEX (Liu et al., 2021). Second, we used six possible linearizations: pre-order, post-order, and pre/post-order-alias-start/end. Examples of different linearizations, with different partial executions, are given in the Appendix A.1, tables 6, 7, 8 and 9.

heading: 4.3 Training pipeline
text: Our training methodology employs a sequence-tosequence (seq-2-seq) framework, utilizing BART as the fondational architecture (Lewis et al., 2019). We initialize our parameters using the TAPEX (Liu et al., 2021) checkpoint, as preliminary experiments have demonstrated improved performance. Subsequently, we fine-tune the model using natural language questions, replacing the SQL queries used in the pre-training. Our additional hyperparameters correspond solely to the choice of operators in O∗, as discussed below.

heading: 4.4 Overall performance
text: In this section, we compare our model with the state-of-the-art ones, on the test split of the WTQ Dataset. Results are shown in Table 2, distinguishing between those employing fine-tuning techniques from BART-like architectures, and those considering in-context learning of LLMs, using specific prompting strategies. We report SDA as well as FDA for the model for which we reproduced the results. We report in Table 2 the results of the best-performing set of operators we experimented, namely O∗ = {P,C, S}, as well as our ensemble model that leverages various granularities O∗.
The semantic parsing baseline SQuALL does not perform well, especially if tables are not manually cleaned up (dropping from 54.3% to 27.2% for FDA), while other methods do not require this costly cleaning step. Our models showcase notable achievements, with our best one (selected on the validation set) reaching an FDA of 61.4%. This is comparable to OmniTab which relies on sophisticated data augmentation techniques. Our performance can even increase to 66.3% when leveraging ensemble methods (see Section 4.7). We also show later that besides obtaining state-of-the-art results
(for similarly sized architectures), our models are also more robust (Table 3b).
Notably, prompting approaches based on LLMs, including the chain-of-thought method (Wei et al., 2022), demonstrate superior performance without necessitating model adaptation. It is important to note that (1) these approaches rely on much more parameters (GPT-3.5 has 175 billion parameters (Brown et al., 2020)), (2) our training approach could also be used to fine-tune LLMs, and (3) this paper contributes to the ongoing discussion in the community about leveraging external tools vs. relying solely on model capabilities. Recent studies, which highlight difficulties encountered by GPT-4 for counting rows of tables, are in line with this argument (Sui et al., 2024).

heading: 4.5 Sensitivity over questions types
text: In table 3a and 4a, we show the performance for different query types, distinguished by whether they contain operators such as Projection, Comparison, Selection, Group By, Order By, Aggregation, Operator, and Limit. It is important to note that queries containing a group-by are limited, so results reported in this column should be interpreted cautiously.
Among existing models, Omnitab slightly outperforms our model in some scenarios, showing the importance of its data augmentation techniques compared to Tapex. Tapas does perform worse on these query types, which shows the limits of its aggregation methodology based on column/row selection.
Among our models, PCS exhibits the best overall performance (as on the test set), thanks to its robust handling of query types. However, surprisingly, it performs worse on group-by queries compared to models that include GB in O∗. We suppose that this might be due to the variance due to the limited number of queries of that type. Finally, our model exhibits a pattern where simpler operators (projection, comparison, selection) are better handled when generated directly, while others (order by, aggregation, operators) benefit from being executed externally.
Finally, Table 5 presents the performance of models based on the complexity of the query, as measured by the number of operators in the original computational graph. OmniTab and our models (especially +GB+H) demonstrate resilience with relatively stable performance across various operation ranges. Tapex and Tapas, however, show a decline in performance as complexity increases, with Tapas notably struggling in the 8+ operation category, highlighting the limit of extractive methods.

heading: 4.6 Comparing linearization methods
text: In figure 2, we show the impact of linearization on the performance of the models. We can first observe that differences between our model variants decrease as most of the computational graph is executed, which was expected. Contrary to our expectations, however, using aliases has a negative impact, especially when they are more frequently used (+P to +P+C+S), which shows that having too many aliases is problematic when generating a relational formula. When using aliases, putting the
tables after the operators did somehow improve the results. We think that these results might change with better training procedures (e.g., data augmentation with perturbations): we observed that models using aliases were more robust, but their overall performance was nevertheless below that of nonaliases ones. Finally, we observe that there is a granularity level (+P+C+S) that achieves the best performance, corresponding to cases where only basic table selection is performed; moreover, this level is less prone to overfitting as discussed in Section 4.8.

heading: 4.7 Ensembling
text: Figure 3 illustrates the results that we obtained using different ensembling combinations. The ensemble prediction is given by a majority vote. In case of ties, we use the validation FDA to weight the votes. We experimented with two ensembling settings: going from semantic parsing models to full execution, or in the opposite direction, i.e. from full execution to semantic parsing.
First, performance improves whatever the ensembling method. This improvement can be explained with the analysis presented in Table ??, where we analyzed the performance depending on the operators composing the computational graph. While certain models excel in specific types of operations, others may show superiority in different areas. Such diversity among the models is important for ensembling.

heading: 4.8 Sensitivity over table column cells perturbations
text: The Transformers architecture can easily overfit, especially in the case of a dataset like WTQ. To measure the importance of overfitting, we use the
Table 3: Performance (FDA) of models on the validation set of WikiTableQuestions, grouping results per type of query, for the models based on pre-order linearization (no alias). The column ALL reports FDA averaged over validation queries. The best results are in bold.
Model Projection (ALL) Comparison Selection Group By Order By Aggregation Operator Limit σ # 500 367 363 30 151 206 75 153 Tapas 52.6 51.8 52.3 16.7 53.0 43.7 30.7 52.3 13.5 Tapex 55.2 55.9 56.5 50.0 60.9 38.8 44.0 60.8 7.9 Omnitab 58.8 59.7 59.8 56.7 61.6 47.1 45.3 60.8 6.4 P 44.6 40.9 41.3 40.0 49.7 43.7 28.0 49.0 6.8 +C 51.6 50.1 50.7 23.3 48.3 50.0 38.7 47.7 9.7 +S 58.6 58.0 58.4 40.0 58.3 52.4 52.0 57.5 6.4 +GB+H 57.8 57.8 58.4 23.3 57.0 49.5 49.3 56.2 11.8 +OB 57.6 57.5 57.8 53.3 58.9 51.5 50.7 58.2 3.3 +A 58.0 57.8 58.4 56.7 62.2 47.1 49.3 61.4 5.4 +OP 56.6 57.8 58.4 50.0 60.3 46.1 42.7 60.1 6.8
(a) Using validation data – the row # contains the number of matching queries (see Section 4.5)
Model Projection (ALL) Comparison Selection Group By Order By Aggregation Operator Limit σ Tapas 42.6 41.7 42.2 16.7 38.4 37.9 18.7 37.9 10.6 Tapex 43.4 43.0 43.5 43.3 44.4 35.4 29.3 44.4 5.5 Omnitab 45.4 44.7 44.9 36.7 42.4 39.3 30.7 42.5 5.1 P 43.2 39.8 40.2 36.7 45.0 44.2 28.0 44.4 5.7 +C 49.0 46.3 46.8 23.3 45.7 48.5 38.7 45.1 8.5 +S 53.6 51.0 51.2 40.0 49.0 51.9 50.7 48.4 4.2 +GB+H 51.6 49.6 50.1 23.3 45.0 49.5 48.0 44.4 9.2 +OB 50.6 50.7 51.2 40.0 43.7 48.1 46.7 43.1 4.1 +A 47.2 46.0 46.6 50.0 45.7 41.8 40.0 45.1 3.1 +OP 47.8 47.7 48.2 50.0 45.7 43.2 30.7 45.8 6.1
(b) Using validation data with random permutations of each column (see Section 4.8)
Figure 3: Evolution of FDA (test set) depending on the number of model variations in the ensemble. We either add models to the ensemble starting from the left (green) or the right (blue).
validation set (since the test set has no associated SQL queries) and perform random perturbations, i.e. we permute rows within each column. To avoid problems related to the maximum length of the input, we ensure these perturbations only affect the parts present in the input of the transformer – all models would have been affected, and this would have reduced the sensibility of our measures.
Results are shown in Table 3b using a preorder (no alias) linearization (our best linearization method). We observe that perturbation strongly affects even the best-performing approaches on WTQ, as OmniTAB performance lowers from 58.8 to 45.4 (-13.4), Tapex from 55.2 to 43.4 (-11.8), and Tapas from 52.6 to 42.6 (-10.0). Our models are much less impacted. For instance, our bestperforming approach (PCS) decreases its performance from 58.6 to 53.6 (-5.0), and surpasses the best baseline, Omnitab, by a large margin (53.6 vs 45.4), showing that data augmentation is less effective in preventing overfitting than generating formulas combining content and relational operators.
Among our models, we note that the lesser the amount of executed parts in the computation graph, the lower the decrease. As some models were initially more performant than others, we can note that the "P+C+S" model is the most effective one, with an average FDA of 53.6. Finally, we can see that the impact on some operators (e.g. group by, limit, comparisons) is even higher for models where most
(a) Using validation data – the row # contains the number of matching queries (see Section 4.5)
Model Projection (ALL) Comparison Selection Aggregation Tapex 77.3 77.3 77.3 76.4 P 76.1 76.0 76.0 73.3 +C 86.7 86.6 86.6 80.0 +S 79.5 79.4 79.4 77.6 +GB+H 79.5 79.4 79.4 77.6 +OB 79.5 79.4 79.4 77.6 +A 79.3 79.4 79.4 76.7 +OP 79.3 79.4 79.4 76.7
(b) Using validation data with random permutations of each column (see Section 4.8)
or all of the computational graph is executed. On WikiSQL (Table 4b), we still observe some improvements over methods as Tapex, especially after perturbing the validation tables, showcasing an improved robustness even in such simpler settings.

heading: 5 Conclusion
text: We explored the realm between semantic parsing and direct output generation for table QA, showing that PLMs can leverage an appropriate level of granularity where basic table manipulations (clean-
ing, selection) can be handled by the transformer itself while higher-level operations (e.g. aggregation, arithmetic) are better handled by dedicated tools. We showed that a model, appropriately trained, achieves high performance compared to state-ofthe-art, and that, more importantly, most PLM baselines are prone to overfitting (by using a simple permutation of table cells), while our method is much less affected and outperforms the best baseline, OmniTab, by a wide margin. Future works will include more sophisticated training procedures, a sparse attention mechanism to cope with long tables, and more in-depth error analysis.

heading: 6 Limitations & Risks
text: Our models have not been trained with data augmentation, which would help make them more robust – even if other models could benefit from it (e.g., Tapex or Tapas), we hypothesize that it would have an even bigger impact on our model. The best baseline, OmniTab, was already trained with augmented data. Experimenting with more datasets would also have strengthened our results. However, as for all works on Table QA, WikiTableQuestions is still a resource of reference.
We did not compare our results thoroughly with LLMs but did report the results from the original papers. However, the gap between the bestperforming LLMs and our model is not that high, showing the potential benefit of using partially executed formulas. Future works could include the fine-tuning of LLMs with our proposed supervision.
Our methodology requires SQL annotations. In the case of datasets without any SQL supervision, but with questions in natural language, a simple approach would be to rely on a weak/distant supervision method, i.e., using semantic parsing models’ predictions as SQL annotations.
Risks involved in this research are similar to those arising from any NLP research, as an automatic understanding of data can be used maliciously, e.g., leaking confidential information from tables. However, this work focuses on an exploratory study of learning abilities, which is dedicated to the scientific community only.

heading: 7 Acknowledgments
text: This work was partly funded by the ANR-21CE23-0007 ACDC project. Experiments were performed using HPC resources from GENCI-
IDRIS (Grant AD011014032R1, AD011014110 and A0151014638).

heading: A Appendix
text: A.1 Linearization A.1.1 Using aliases In the case of pre-order, we denote the alias for node n with αn and use the following linearization:
lpre(n) =    l(xn)⊕i αni ⊕ αn ⊕i lcpre(ni) if x ∈ O
∅ else
where ∅ denotes the empty sequence and lc is either empty – if the operator has already been linearized – or || ⊕ lpre(ni) if not. Note that the order of linearization is important, but to avoid more complicated notations we do not make it explicit here.

heading: Future Work
text: Future works could include the fine-tuning of LLMs with our proposed supervision. Our methodology requires SQL annotations. In the case of datasets without any SQL supervision, but with questions in natural language, a simple approach would be to rely on a weak/distant supervision method, i. e. , using semantic parsing models’ predictions as SQL annotations. Risks involved in this research are similar to those arising from any NLP research, as an automatic understanding of data can be used maliciously, e. g. , leaking confidential information from tables. However, this work focuses on an exploratory study of learning abilities, which is dedicated to the scientific community only