Bridging the Semantic Gap Between Text and Table: A Case Study on NL2SQL
Authors: Lin Long
Xijun Gu
Xinjie Sun
Wentao Ye
Haobo Wang
Sai Wu
Gang Chen
Junbo Zhao
Venue: ICLR 2025 Poster (2025)
Keywords: Large Language Model
Table Representation Learning
NL2SQL
Multimodal Learning
Primary area: foundation or frontier models, including LLMs
TL;DR: To bridge the gap between tabular and textual information, we propose TNT, a table-language model that empowers LLMs with the ability to effectively and efficiently extract structure-enriched semantics from tabular data.

Abstract: The rise of Large Language Models (LLMs) has revolutionized numerous domains, yet these models still exhibit weakness in understanding structured tabular data.
Although the growing context window promises to accommodate a larger volume of table contents, it does not inherently improve the model's ability to understand the underlying structure and semantics of tabular data.
To bridge the semantic gap between **T**ext and **T**able, we propose **T**n**T**, a table-language model that features multimodal table representations to empower LLMs to effectively and efficiently abstract structure-enriched semantics from tabular data. 
**T**n**T** also introduces a scalable and efficient training pipeline, featuring novel self-supervised tasks, to integrate abstract tabular knowledge into the language modality.
Extensive experimental results on NL2SQL demonstrate a much better table understanding of **T**n**T**, which achieves up to **14.4** higher execution accuracy compared with traditional text-based table representations.

Published as a conference paper at ICLR 2025
BRIDGING THE SEMANTIC GAP BETWEEN TEXT AND TABLE: A CASE STUDY ON NL2SQL
Lin Long 9x Xijun Gu , Xinjie Sun Wentao Ye Haobo Wang 20, Sai Wu
Gang Chen S Junbo Zhao 9t "Zeejang University, Hangzhou High-Tech Zone (Binjiang) Institute of Blockchain and Data Security
ABSTRACT
The rise of Large Language Models (LLMs) has revolutionized numerous do- mains, yet these models still exhibit weakness in understanding structured tab- ular data. Although the growing context window promises to accommodate a larger volume of table contents, it does not inherently improve the model's 's ability understand the underlying structure and semantics of tabular data. To bridge the semantic gap between Text and Table, we propose TNT, a table-language model that features multimodal table representations to empower L.LMs to ef- fectively and efficiently abstract structure-enriched semantics from tabular data. TNT also introduces scalable and efficient training pipeline, featuring novel self-supervised tasks, integrate abstract tabular knowledge into the language modality. Extensive experimental results on NL2SQL demonstrate a much bet- ter table understanding of TNT, which achieves up 14.4% higher execution accuracy compared with traditional text-based table representations.
INTRODUCTION
The emergence Large Language Models (LLMs) has reshaped the landscape of deep learn- ing (Brown al. 20200 Touvron et al. 2023). Researchers have been actively exploring extend- ing LLMs' capabilities to process images, videos, and audio, broadening their applicability across diverse domains (Liu et al. 2023: Hu et al. 2024 Zhan et al., 2024; Chen al. 2023). Tabular data, given its ubiquity and unique application value, has also attracted increasing interest from the community (Ruan et al. 2024 Lu al. 2024b Fang al. 2024). However, the special properties and structures of tabular data pose distinct challenges for L.Mss, raising a fundamental question: How well can L.MSs truly understand tabular data?
To investigate this, we focus on NL2SQL,, a piv- otal task in table understanding where LLMs translate natural language queries into SQL, given specific database (Pourreza & Rañeii 2023 Gao et al. [2024] Dong et al., 2023).. It requires an abstract semantic understanding of relational tables and often serves as an interface between L.MMs and tables in mainstream table-
EX(%)
154
Set
58
Set
#CellVColumn
related methods (Cheng et al. [2023] Ye et al. 2023 Jiang et al., 2023bb Wang et al. 2024). To harness LLM for tabular tasks, straightfor- ward yet common strategy is to serialize each column with its name, data type, attributes, and sample cell values, into prompts (Li al. 2024b
Figure 1: Performance of GPT-3.5 Turbo on SPI- DER-Realisitc with different sets of cell values se- rialized into the prompt.
Pourreza & Rañeii 2023). However, we find that based on such naive representation, current L.MS struggle form a robust and consistent understanding of comprehensive table semantics. As shown in Figure model performance varies significantly with different cell value selections, rather than generally benefiting from a comprehensive understanding of the given tables (Figure [.. It sug- gests that the model resorts to flat context mapping rather than real table structure-aware semantic comprehension an issue also noted prior work (Li et al. 2024d Yang et al. 2022).
"qqual contribution.
Correspondence: j. 2ha o.z.u.edu.ccn

---

Published as a conference paper at ICLR 2025
Such phenomenon drives us to rethink the limitations of text-based table serialization (Zhang al. 2024 Li et al. 2024c), which can be summarized into three key aspects: 1) Structural Incompat- ibility: The bi-dimensional structure of tabular data is inherently misaligned with the sequential, autoregressive nature of language models, making LLMs less sensitive to structure-related patterns within its context; 2) Redundancy: Common serialization methods for tabular data are highly token- inefficient, especially when dealing with large tables or databases, leading to huge computational overhead and potential performance degradation due long contexts; 3) Bias: Serializing poten- tially uncurated table schemas along with limited number of cell values into the prompt can cause the model to overfit to the local information presented in the context, leading an incorrect or incomplete understanding of the table's semantics.
To bridge this gap and make L.LMs better table readers, we believe that a better table representation needed- - one that can abstract the pructure-enriched semantics of original tables. Although earlier works have explored learning table representations with smaller models like BERT and BART (Yin) et al. 2020: Liu et al. 2022b), these approaches are less competitive today due to their limited lan- guage understanding capabilities. Drawing inspiration from Multimodal Large Language Models (MLLMs) that integrate features from various modalities with text features (Liu et al. 2023 Bai ct al. 2023) Lu et al., 2024a), we introduce TNT, a novel framework that empowers LLMs to extract and reason over high-level representations of tabular data, treating tabular inputs as distinct modal- ity. As depicted in Figure 3 TNT comprises three components: a structure-aware Table Encoder, which captures the abstract semantics of tabular contents and converts it into compact embeddings through a bi-dimensional architecture; a Table-Language Adaptor, which maps these embeddings into the LLM's textual space using learnable queries (Bai al., 2023; Tong et al. 2024); and an LLM decoder, which performs multimodal table reasoning on downstream tasks.
The proposed tabular representation method introduces another critical challenge: How can we fectively train and align these embeddings-basee based representations so that they not only encode valu- able tabular information but are also interpretable and usable by the L.LM? To address this, TNT fol- lows a three-stage training strategy: Encoder Pre-training. Table-Ianguage Feature Alignment, and Instruction Tuning. In the encoder pre-training stage, we introduce column- Solumn-wise contrastive learning, allowing the Table Encoder to learn general column semantics using unlabeled tables. For feature alignment, we create two synthetic table- langulae interleaved datasets, supplemented by adaptations of existing datasets (Nan et al., 2022, Pasupat & Liang 2015 Parikh al.. 2020), to teach the LLMS how to leverage column embeddings to address textual instructions. Finally, in the instruction tuning stage, we fine-tune the model using labeled data enhance its instruction- following capability for specific downstream tasks. For better clarity, we will focus on the NL22QL task as a conerete example, which can also serve as an interface in other general table-related tasks.
Our main contributions are threefold:
(Iniights) We highlight the limitations of LLMs in understanding structured tabular data, introducing novel representations enhance the understanding on abstract table semantics.
(Methodology) We propose TNT, a multimodal framework, along with an efficient training pipeline, paving the way towards scalable, structure- eetrichrr-enriched enriched table semantics learning.
(Experiments) We conduct extensive experiments on NL2SQL under challenging setups that resemble real-world tables, to verify the effectiveness and generalizability of TNT.
MOTIVATION
Understanding table content matters. As discussed above, current LLMs struggle to fully inter- pret serialized tabule contents, while their comprehension often relies heavily hand-crafted meta- information (c.g., column names, descriptions, foreign keys) (Sui et al.] 2024a). However, we argue that such meta-information is insufficient and unreliable to provide comprehensive table semantics, whereas a structure-aware understanding of tabular contents is essential for developing a complete and accurate grasp of table semantics (Yin et al.) 2020). A common challenge in real-world tables is that they usually come with uncurated schemas, where column names may be ambiguous, while detailed deseriptions are missing. For instance, abbreviations are frequently used as column names, yet their meanings can vary depending on the context. Some tables may even include semantic placeholders as column names (Figure In such cases, understanding table structure and fully
2

---

Published as a conference paper at ICLR 2025
Given the following database schema:
What is the number of with more than 4 cylinders? Question (Iretruction) Hybrid Prompt
table cars, data, [cars, iata.Id(see embzi primary key,
Hybrid Representation cars data. Cylindera( emb>> int)....
Learnabe Queries
Crosa Attention
Adaptor
3. LLM Embodiiing
Bi Bi-dimensional Artn
Feedronwerd
Swil Fell-Attention N
Cell Embeddings E(T)
Courm Altantion
FoodForwerd
Row Aftentionn
cel I
ent
Cell Embbeddings B(T)
SELSCT
Sentence Transformer +
MPG
Cylinders
Edispt
Year
FROM caT5 date
.4
cyliiberz
22
22
198.0
1970
1. Table Encoder
23
18
199.0
1970
Answer
Tnpput Table
AAttentionn Scupp
Overall architecture of TNT. The Table Encoder generates cell embeddings from input tables, which are aggregated at the column level by the Adaptor to be further utilized by LLM.
Figure 3:
interpreting the patterns and interrelation- ships within its contents (i.e., abstract ble semantics) are key to accurate table understanding. This not only reduces the model's 's reliance on concrete manual anno- tations but also enhances its robustness and overall table comprehension. However, as shown in Figure LLMs struggle to under- stand the serialization of tables with non- semantic schemas, which is the gap that TNT aims to fill.
Age Heighticm) Weightikg)
Actually to...
Bd
A3
A4
57
103
Figure 2: (a) Table with non-semantic column names. (b) Performance on SPIDER-Realistic, where SFT LLAMA3 uses text-based table representation.
LLMs need better table representations. Although prior studies have explored learning table representations (Yn et al. 2020 Liu et al. 2022b Reimers & Gurevych, 2019), each has limi- tations present generalizdd form of rable-modality knowledge that be effectively utilized by LLMss. Given the constraints of text-based representations discussed above, we helieve that an effective and alignable table representation should ensure: Structure-wwareness. Considering the bi-dimensional structure of tabular data, a comprehensive and robust table representation should capture semanties across cell, row, column, and table dimensions while being invariant permuta- tions of rows and columns. 2) Expressive Efficiency. To handle tables of varying sizes, an effective table representation should also be capable of appropriately abstracting and compressing table infor- mation based on proper semantic segmentation, to ensure expressive efficiency for better utilization by LLMs. 3) Cross-table Generalizability. An effective and robust table representation should not only be able to encode table-specific information but also incorporate common tabular knowledge ensure generalizability across different tables, thus better enhancing I.LMs' abstract table under- standing. To satisfy these requirements, paper introduces dedicated model architecture that progressively derives column- wise semantics, coupled with a specialized training strategy to enable the model to learn effective table representations that be seamlessly aligned with LLMs.
ARCHITECTURE
As illustrated in Figure TNT an end-to-end table-language model consisting of three compo- nents: 1) the Table Encoder, which generates structure-enriched semantic embeddings based
3

---

Published as a conference paper at ICLR 2025
the input tables; 2) the Table-Language Adaptor, which further aggregates the output embeddings from Table Encoder and aligns them with textual features; and 3) the Large Language Model (LLM), which leverages the high-level table representations and reasons within the bi-modal space.
Table Encoding. Tables consist of heterogeneous cells that are formally independent yet logically organized. To more effectively abstract and encode this type of semantics, we introduce Table Encoder, taking the unique structural characteristics of tabular data into consideration. Formally, let T = [C11..... Cini ................................ Cmn] represent table with rows and columns, where Cty denotes the cell content in the i-th row and j-th column. The Table Encoder starts with a sentence transformer I to extract semantic representations from each cell, transforming the variable-length cell contents into uniform, compact "cell tokens":
E(T) (#(cn)..... {(Cmm)] E Rmxxxdd
where d is the dimension of each cell embedding. These cell embeddings are then passed through stack of bi-dimensional attention modules (Zhu et al.) 2023 Somepalli al. 202I). where they interact with other relevant cells to capture global structural semantics:
(T) 2D-Aut 2D-AAm(EE(T)) E R
Within each module, styee bi-directional attention (Devlin al. 2019) is applied first along each row and then along each column. This alternating attention mechanism enables the model to capture both the distributional properties within individual columns and the interrelationships be- tween columns. To maintain the permutation invariance of (relational) tables, positional embeddings are intentionally excluded from these bi-dimensional attention modules.
Remark. Similar to the role of vision encoder in Vision-Language Model (VLM) (Liu et al. 2022a Bai et al. 2023 Tong et al. 2024), the Table Encoder leverages tabular knowledge acquired through large-scale pre- training (discussed in later sections) to transform the input table into series of cell embeddings that encapsulate the structure-ente cennture-enriched global semantics the table, analogous to patch embeddings in image encoding. These cell embeddings serve as the fundamental informa- tive units, which can be aggregated to form more compact and comprehensive table representations.
Table-Language Adaption. Given the varying sizes of tables, the cell embeddings derived from the Table Encoder can be highly redundant and are not naturally aligned with the textual inputs of LLMs. Therefore, aggregation and alignment needed to make more effective and efficient utilization of these embeddings. To ensure expressive efficiency under appropriate semantic seg- mentation, we introduce an Adaptor g. aggregate cell embeddings the column level, columns are considered the fundamental units that define a table's semantics. Specifically, performs cross- attention between k learnable queries (Bai al. 2023 Tong al. 2024 Li al. (023c) and cell embeddings from each column, transforming cell embeddings from tables with an arbitrary number of rows (m) into fixed-length (k), column-wise compact representations that are aligned with the LLM's embedding dimensionality (d'). The embedding of i-th column can be derived by:
(3) Being jointly trained with the LLM on lable-language interleaved data, the Adaptor generates rep- resentations that can be more effectively processed and interpreted by the specific LLM, thereby enhancing its reasoning and generation capabilities for downstream tasks.
C(T);
ET(T) E kxdd"
Dynamic Context Integration. In TNT, we differentiate between the abstract semantics that can be represented by column embeddings and other table information involving concrete details (eg.. column names and foreign keys), which are inherently sequential and better preserved in text form to ensure precise generation by the LLM. To effectively integrate both embedding- based and textual representations, we adapt widely used prompt template (Li et al. 2024b Yin et al., 2020 Pourreza
& Rañei 2023), transforming into hybrid table representation:
"table tab_name, columns=[t wtt.:/ttaaaaaaaaaaaaaaeammmem www.nnaam.moma.cem
During inference, the respective column embeddings are dynamically inserted into the specified slots at the LLM's embedding layer. This hybrid representation relieves the Table Encoder from loss- lessly compressing all table details, while still providing the LLM with valuable high-level insights. seamlessly combines the abstraction of column embeddings with the specificity of text-based in- formation, resulting in more comprehensive and expressive table representation.
4

---

Published as a conference paper at ICLR 2025
Stage 1: Column-wise Contrastive Learning
Stage 2: Multi-task Feature Alignment
Stage Instruction Tuning
Teshaal Feature
LLM [Dacoder]
LLM (Decoder)
Table Encoder
Tabular Feature
Raw Tabular Data
Adaptor
Adaptor
Encoder
Table Encoder
(a)
Multi-takk Table Table-langues language Interleaved Data
Tasksppciific Data
Schemo-independent
Table
Projecton
Gownledh
Identical
<4,4
Table Encoder
Projectoo
(b)
Tables
Snapshots
Column Embeddings
Figure 4: (a) Overall training process of TNT. (b) Detailed depiction of column-wite contrastive learning. e. and e denote positive and negative samples w.r.t. e, respectively.
TRAINING
Similar to other MLLMs, the training logic of TNT falls into three parts: 1) Encoder Pre-training. where we introduce a novel column-wise contrastive learning objective to independently train the Table Encoder on large-scale raw table data, enabling it acquire general tabular knowledge; 2) Feature Alignment, where we leverage multi-task table-language interleaved data to establish conceptual linkage between tabular and textual features; and 3) Instruction Tuning, where we fine- tune the model on task-specific labeled data, to improve TNT's instruction-followin capability for targeted downstream applications.
Data. To support large-scale training in Stages 1 and 2, we collect a dataset D of 86, 046 high- quality business tables from various domains, including finance, education, and medicine, ensuring broad coverage of the complexity and diversity of real-world tables. To facilitate modality fusion, we incorporate table-language interlcaved data from existing benchmarks, including FetaQA (Nan et al. 2022). WikiTableQQustion (Pasupat & Liang 2015). and ToTTo (Parikh et al. 2020).
4.1 COLUMN- WISE CONTRASTIVE LEARNING
Humans naturally identify patterns within columns and distinguish differences between columns, which represents the most essential and general form of table semantics. Inspired by this, employ contrastive learning to explicitly guide the Table Encoder to extract features that unify the intra- column semantics while differentiating between columns, using only unlabeled tables schema- independent manner. As shown in Figure 4 we first apply random row sampling on each table T; within the mini-batch, creating two snapshots, S, and Si which share the same schema but have different cell contents. The Table Encoder then generates an embedding pool P consisting of column embeddings from each snapshot in the mini-batch. During contrastive learning, positive pairs are formed by the embeddings from the same columns across the two snapshots. Following Chen et al. (2020a), we maximize the similarity between positive pairs while minimizing the similarity of other negative pairs using the InfoNCE loss (more details provided in Appendix B.3). which is defined
exp(e Te./7)
Loom (T, P)
Σ PEP
log
P
Eeemm(E) exp(e' ee/T))
5

---

Published as a conference paper at ICLR 2025
where e+ is the embedding of the same column but from different snapshot, and is the temperature. This strategy drives the model to lear discriminative and context-aware column em- beddings, capturing the unique semantics of each column within specific schema.
4.2 MULTI-TASK FEATURE ALIGNMENT
In the pre-training stage, the Table Encoder acquires general tabular knowledge through self- supervised learning on large-scale raw tables. However, without alignment on task-driven data, these pre-trained representations often lack interpretability, making it challenging to provide valid information for LLMs' reasoning and generation. To bridge the gap between tabular and linguistic elements within the embedding space, we curate a multi-task table-language interleaved dataset and jointly train the Table Encoder and Adaptor in conjunction with specific backbone LLM.
We extracted subset of tables from D to create training samples for two synthetic tasks: D Column Prediction predicting which column given cell value belongs to, and @ Cell Prediction identifying which cell value corresponds to a given column. Solving these tasks requires the model to utilize the information captured in the column embeddings to recognize distributional patterns and interrelationships between columns. To promote reasoning based on column embeddings rather than aehema-ddeendenn signals, manually anonymize 50% of the column names, preventing overfitting on schema-specific cues and fostering a more generalized table comprehension.
To further increase data diversity, we also adapt FetaQA (Nan al., 2022). WikiTableQuestion (Pa- supat & Liang, 2015), and ToTTo (Parikh et al. 2020) into formats suitable for training column embeddings. resulting in three adapted tasks: @ Question Generation - generating a question on based on given answer from a specific table, @ Table Titling creating brief title for given summarizing the content
specific row. To align with our goal of semantic table understanding, ensure that these outputs maintain appropriate level of abstraction by minimizing their reliance on detailed cell values. In total, we curate 292, 235 samples for feature alignment. Detailed statistics are provided in Appendix B.2
table, and @) Row Summarization
4.3 INSTRUCTION TUNING
Although we employ diverse prompt templates during the feature alignment stage, some loss of general instruction instruction-folloiing capabilities in the LLM is still inevitable. Therefore, to refine the L.M's comprehension of column embeddings and ensure optimal performance in downstream tasks, we further unfreeze the backbone LLM and conduct supervised fine-tuning using finely labeled data. To optimize training efficiency, only the Adaptor and LLM are trained during this stage.
5 A MULTIMODAL PERSPECTIVE
From the introduction above, existing MLLMs, including: 1) Structure-optimized Encoder: Like Vision Transformers with special partitioning and 2D positional embeddings (Caron et al.) 2021). TNT utilzes Table Encoder with a specialized bi-dimensional attention mechanism that directly captures the row-column struc- tures in tabular data. 2) Hierarchical Feature Extraction: The cell-to-column feature extraction in TNT mirrors the hierarchical feature process in vision and text models, (e.g., patch-to-image and work-to-ssetence) (Bai et al. 2023 Reimers & Gurevych] 2019). This approach combines local details with global features while achieving token efficiency. 3) Scalable Encoder Pre-training: Previous works that apply common MLM or contrastive learning tabular data (Zhu et al.) [023] Wang & Sun 2022) typically impose high requirements on table curation, thereby limiting their gen- eralizability. The proposed column-win contrastive learning objective is instead resilient to most table variations and can well generalize to broader range of tables, showing greater potential in scalability, 4) Knowledge Injection via Modal Fusion: Incorporating additional modalities breaks the limitations of homogenized input processing in uni-modal LLMs from both architectural and training perspectives, which cannot be accomplished by language-dris
that TNT integrates several design principles inspired by
tuning- based methods.
In summary, our work provides pioneering validation of the feasibility of integrating tabular and textual modalities. We hope to offer a perspective from tabular data for future efforts to unify all data modalities. More discussions are in Appendix

---

Published as a conference paper at ICLR 2025
Table Comparative results between TNT and variants using traditional text-based table representa- tions on datasets with non-semantic column names. Best performance on each setup is highlighted.
Dev
Test
DK
Realistic
Backbone
Method
EM
EX
EM
EX
EM
EX
EM
EX
Original
12.8
21.9
18.3
40.5
11.6
31.0
7.1
14.2
LLAMA3-8B-
SFT
30.9
31.7
43.1
50.5
21.7
30.8
19.1
17.9
AI Meta 2024
TNT
46.0
44.6
51.4
59.9
36.3
44.7
35.6
32.3
Original
8.5
17.9
14.0
33.9
6.2
23.2
5.1
7.9
MISTRAL- Instruct
SFT
32.8
32.5
44.2
52.4
23.6
33.8
20.5
16.9
Jiang et al.
2023a)
TNT
43.1
42.6
51.4
57.4
34.0
41.7
36.8
32.9
Original
9.1
22.5
17.9
39.8
8.6
29.0
6.7
12.0
CODELLAMA- - B-Intruct
SFT
32.1
32.6
43.1
51.7
24.3
33.1
21.5
20.9
Rozière et al.
2023
TNT
39.3
40.1
47.9
56.6
31.4
39.6
32.5
28.9
6 EXPERIMENTS
6.1 EXPERIMENTAL SETUP
Datasets. In this paper, evaluate TNT the task of NL2SQL. We adopt three prevalant datasets, SPIDER Yu et al. 2018), SPIDER-DK (Gan et al. 2021a), and SPIDER-Realistic (Deng al. a 2021). SPIDER questions that correspond to 5, 693 distinct SQL. queries and 200 multi-table databases with seman- tie column names. It is divided into three folds: training, development, and test sets. We reserve data (including tables) in development and test sets for evaluation, ensuring they unseen during training (Xie et al., 2024: Qu et al. 2024). SPIDER-DK and SPIDER-Realee variants of the original SPIDER dataset. SPIDER-DK emphasizes the importance of domain knowl- edge, while SPIDER-Reale Realistic removes explicit mentions column names in the questions.
large-scale cross-domain NL2SQL dataset containing 10, 181 annotated
are more challenging
Ideally, robust table understanding should not overly depend on the quality of column names (i.e., whether they are meaningful or not). To assess the models' abilities to understand table contents independently of column names, create non-semantcc version of each dataset, in which 80% of the column names are anonymized. This modification forces the models to leverage structural and contextual clues within table contents, rather than relying on direct meanings of column names, to accurately infer the meaning of each column and thereby generate correct answers.
Metrics. Consistent with prior work (Xie al. 2024: Qu al. 2024), we evaluate our framework using two metrics: Exact Set Match Accuracy (EM) and Execution Accuracy (EX). EM measures the exact match between keywords in the predicted SQL query and the gold query, while EX compares the execution results of the predicted SQL query with that of the ground truth SQL query on some database instances, which provides a more precise estimate the model's performance. We use the official test suite from Zhong et al. (2000) for evaluation.
Implementation Details. The Table Encoder integrates a sentence transformer initialized with all-MiniLM-LL-v22 (Wang et 2020), lightweight BERT-based text encoder. We set the number of learnable queries k in the Adaptor to 5, which yields the most robust performance empirically. We evaluate TNT using three mainstream open-source LLMs: LLAMA3-88 I--Instruct (A Metal (Jiang et al. 2023a 2 and CODELLAMA- 7B Instrruct (Rozière et al., 2023 as backbone models (results are based on LLAMA3--B-Instrre
2024 MISTRAL-7B-Inse
unless otherwise specified). To ensure fair comparison, we set the decoding temperature to 0 to eliminate randomness during evaluation. Instruction tuning is conducted on the training set of SPIDER. More detailed setups are provided in Appendix The code is available at:

---

Published as a conference paper at ICLR 2025
Table 2: Comparative results on standard NL2SQL benchmarks with semantic column names.
Dev
Test
Realistic
Backbone
Method
EM
EX
EM
EX
EM
EX
EM
EX
Original
34.0
54.1
32.5
58.0
29.5
50.8
25.6
54.7
LLAMA3-8B-Instmu
SFT
73.0
71.5
71.9
77.8
57.4
63.7
63.4
59.6
A Meta 2024
TNT
73.2
72.7
72.6
79.0
58.5
65.4
66.1
62.6
Original
26.5
46.7
26.2
53.7
21.5
46.7
21.9
35.6
MISTRAL-7B B--nstut--
SFT
69.7
68.2
68.6
75.6
50.7
58.7
64.2
57.3
Jiang et al.
2023a)
TNT
72.4
71.3
70.1
78.6
57.9
63.6
64.0
59.1
Original
29.0
51.2
31.0
59.7
24.9
51.0
26.6
38.6
CODELLAMA-TT BB-Istruuttt
SFT
65.7
65.0
64.7
72.6
53.3
58.9
53.3
54.5
Rozière et al.
2023
TNT
66.2
66.2
65.3
73.5
54.3
60.6
56.7
55.5
6.2 OVERALL PERFORMANCE
Effectiveness on Structure-aware Semantic Understanding of Table Contents. While humans effortlessly interpret table with ambiguous irrelevant column names, traditional LLMs often struggle under these conditions. To demonstrate the proposed multimodal representation pro- vides LLM with insights on abstract table semantics, we evaluate its performance against the original and fine-tuned versions of backbone LLMs on datasets with non-semante column names, providing no sample cell values but column embeddings. As shown in Table [ TnT dominantly outper- forms all the baselines on both metrics, achieving up to 16.5% % higher EM and 14.4% higher EX. These results indicate that TNT can derive insights from the structure-enriched semantics in col- umn embeddings, accurately recognizing the distributional patterns within each column and their interrelationships, thereby inferring the reference of each column for precise SQL generation. This demonstrates TNT's strengths as robust table interpreter, going beyond mere schema parsing.
Generalizability across Various Setups. We further assess the generalizability of TNT using standard academic NL2SQL benchmarks. As shown in Table 2 TNT consistently surpasses the baselines across most benchmarks for all backbone LLMs, demonstrating strong generalizability. These results indicate that even when table schemas are semantically informative, a deeper structure- enriched semantic understanding of table contents provides additional insights. This underscores the value of extracting abstract semantic features from table contents using high-level column embed- dings, which are not effectively represented through text-based approaches. Additionally, we ob- serve slight variations in performance gains between instruction-tunee LLMs and code-tuned LLMs, which may be attributed to differences in their generalizability. thereby affecting the effectiveness of feature alignment. We leave further optimization of the backbone models for future work.
6.3 ABLATION STUDIES
TNT involves multi- stage training pipeline, therefore, conduct ablation experiments to evaluate the significance of each stage. Specifically, we remove each training step independently from TNT and assess the resulting impact on the model performance. As shown in Table removing any of the three training stages leads to a notable performance decline (up to -23.4 %). underscoring the unique and essential contributions each stage: 1) Encoder Pre-training (PT) equips the model with foundational tabular knowledge through cross-table self-supervised learning. Without this explicit guidance for distinguishing column semantics, the embeddings risk overfitting to spurious task- specific patterns and less diverse tables, resulting in less generalizable and informative embeddings. As shown in Figure without PT, TNT shows sharper decline in the loss during feature alignment under the same learning rate, suggesting that it may be exploiting shortcuts instead of building a robust understanding of table semantics. 2) Multi-task Feature Alignment (FA) bridges the gap between tabular and textual features with table- language interleaved data. Skipping EA and proceeding directly to instruction tuning significantly weakens the robust integration between Table Encoder and L.M,, due the lack of data diversity. The final stage, Instruction Tuning (IT), is
8

---

Published as a conference paper at ICLR 2025
Table Ablation results on training process.
mitout
TNT wit PT
Emb
Dev
Test
Lo33
36.0
55.8
/
x
31.9
53.4
21.2
39.2
31.7
50.5
10 20
50
40
50
44.6
Figure 5: Loss curve during feature alignment.
critical for enhancing the model's instruction-followingg ability, without which the model is not able to generate informed responses even when the earlier stages provide strong column embeddings.
6.4 FURTHER ANALYSIS
Column embeddings are not soft prompts. Due to the introduction of additional learnable parame- ters, one may question whether the column embed- dings function merely as soft prompts rather than be- ing truly tied to table semanties. To address this con- cern, we compare TNT with LLMs fine-tuned with soft prompts, using the same number of learnable parameters as TNT during instruction tunings. As shown in Table 4 the introduction soft prompts do not bring significant performance boost, likely because they do not contribute to understanding ta- ble semantics but instead focus enhancing the model's task-specific abilities, which are already largely addressed by instruction tuning. We also
Table 4: EX Accuracy with different forms of Icarnable representations non-semantic SPIDER.
Method
Dev
Test
SET LLAMA3-8B In-trust
31.7
50.5
SFT (w/ Soft Prompts) TransTab WWag & Sun
30.4
51.5
2022
23.7
33.0
CM2 Ye et al.
2024
17.6
39.2
TNT
44.6
59.9
evaluate the effect of other embedding-based table representations. such as TransTab (Wang & Sun 2022) and CM2 (Ye et al. (024), while both approaches show poor alignment with the backbone LLM. Results above highlight that the column embeddings introduced in TNT not only genuinely capture high-level abstractions of table contents, but also are well alignable in conjunction with LLMs for effective reasoning and comprehension.
Table 5: EX Accuracy SPIDER-Dev with different numbers (#) of example values.
Dataset
Method
#=0
#=1
#=3
#=5
#=20
Original
54.1
57.0
57.6
57.0
57.0
Semantic
SFT
71.5
73.1
73.5
72.3
71.8
TNT (#=1)
74.0
Original
21.9
37.7
39.9
38.4
37.8
Non-semantic
SFT
31.7
46.8
47.5
46.9
46.0
TNT (#=1)
51.5
Column embeddings tell more than serialized contents. We make further comparisons between our hybrid representations and traditional text-only representations. Specifically, we analyze the impact of serializing different numbers of sample cell values into the prompt. To minimize po- tential side effects from changing the prompt template, we include minimal number of example values (#=1) TNT. Three key observations emerge from Table 1) Incorporating tabular content generally improves model performance, indicating the significance of table contents. 2) However, model performance does not consistently correlate with the number of cell values included, as LLMs with limitations in structural understanding, struggle to fully comprehend large volumes of table contents. Worse still, incorporating cell values result in a longer context length, which can nega- tively impact model performance, making the optimization fext- table representations even

---

Published as a conference paper at ICLR 2025
harder. 3) For TNT, using just example value achieves the best performance across all variants, highlighting that compact column embeddings, which are structure-enriche and more semantically interpretable, provide insights beyond serialized table contents. This demonstrates the effectiveness of column embeddings in enhancing the model's ability to process and reason over table contents in more token-efficiee manner, overcoming the limitations of text-based approaches.
Compatibility Analysis. Regarding the potential of TNT, we demonstrate that the proposed column embed- dings are highly compatible with common prompting techniques used in existing NI.25QI. methods. As shown in Table 6 integrate techniques such as Schema Fil- tering (SF) et al. 2024b), Code Correction (CC) (Sun et al.
Table 6: Effects of combining TxT with other techniques on SPIDER-Test.
Method
EX
2023). and Self-Consistency (SC) (Pourreza & Rafiei 2023) into TNT, all of which led to noticeable improvements. We can even approach methods that are based on closed-source LLMs like GPT-4 (Gao et al.] 2024 Sui al. 2024b) (86.6%) and GPT--3.5 (Dong et al. 2023) (82.3%), with much smaller backbone LLM, such as LLAMA3-8B. This, to some extent, also suggests that the structure-enriched semanties captured by column embeddings provide unique value that cannot be effectively substituted by text-based prompting techniques alone.
SF
79.6
TNT (LLAMM3--BB-asee))
CC
84.3
SC
80.9
7 RELATED WORK
Tabular Prompt Design. Harnessing LLMs for tabular tasks first requires converting tabular data into a format compatible with text-based imputs. common method serialize tables into formats like Markdown, JSON, XML (Fang et al. 2024, Singha al. 2023 Sui et al. 2024a). However, prior study reveals that even minor changes, such as row permutations, can lead to substantial per- formance fluctuations (Liu al., 2024). indicating that LLMs lack consistent understanding of table structures. Additionally, due to context length limitations, feeding all table contents into the LLM is often impractical. Some methods retain only schema information (Pourreza & Rafici 2023
Gao et al. 2024). which may hinder generalizability. Retrieval or compression methods selectively include critical table content based on predefined rules, which mitigate context constraints (Herzig iet al. 2020 Li et al. 22024b). Nevertheless, these strategies, like other prompt-based methods, still fail overcome the fundamental limitations of LLMs in comprehending structured data.
Tabular Training. Table tuning focuses on improving (open-source) LLMMs understanding of ta- bles with substantial volume of table-related data. For example, Table-GPT (Li et al.] (024c) employs hatteesisfflllowe---------e
strategy to create datasets for table tuning, while TableLlama (Zhang et al. 2024) uses real-world datasets for instruction tuning on LLAMA2. While effective, these methods still face architectural limitations. Pre LLM-era methods have explored learning table representations with adapted architectures with special positional embeddings (Maj- mundar al., [022] a Yin et al. 2020), attention mechanisms (Zhu et al. 2023 (ooepalli al. 2021) or decoding strategies (Liu et al. (022b), and special learning objectives, including masking (Yin et al. 2020 [Herzi al. 2020), corrupting (Huang al., (2020) and contrastive learning (Wang &
Sun 2022 Ye et al. 2024). However, these methods typically lack enough generalizability and the reasoning and generation prowess of modern LLMs, which reduces their applicability.
CONCLUSION
In this paper, we present TNT, a multimodal framework with novel table representations that en- hance LLMs' ability to comprehend cauucture-enricher semantics from tabular data. TNT features a tailored architecture optimized for tabular structures, along with training strategies specifically de- signed for effective table understanding. Extensive experiments on the NL.2QQL task demonstrate significant performance gains, achieving up to 14.4% % higher execution accuracy, underscoring its effectiveness in semantic understanding of tabular data. Our work paves the way toward an effective modality fusion between tabular and textual data. We hope provides valuable insights for future research.
10

---

Published as a conference paper at ICLR 2025
REPRODUCIBILITY STATEMENT
The implementation details TNT are outlined in a.I further elaborated in Appendix B Due to copyright and privacy restrictions, some proprietary table data used in our training process cannot be publicly released. However, users can train the model within the same framework using their own table data. derivative table expert model available httts://githhi/ com/ tableept/taaleeptt-aa
ACKNOWLEDGEMENT
This paper is supported by NSFC under Grants (No. U24A201401). This work is also partially sup- ported by the Pioneer R&D Program Zhejiang (No. 2024C01035) and the Fundamental Research Funds for the Central Universities (No. 226-2024-00049),
REFERENCES
AetMe Llama 3 model card.
2024. URL httpp://ggttuuu..cc//etttat
11ama //MDD/L/ CARD. md
Rami Aly. Zhijiang Guo, Michael Sejr Schlichtkrull, James Thome, Andreas Vlachos. Christos
Christodoulopoulls, Oana Cocarascu, and Arpit Mittal. FEVEROUS: fact extraction and verifi cation over unstructured and structured information. In NeurIPS Datasets and Benchmarks, 2021. https: http://///aaaaateeeeeeeeeettrtto
URL
benchmarks hwt.seppooeeeiieeeeeeeeee www.:/nss...rrrs....ooo.oo - roundl. hoct: html
2021 hwsh 2aaaa//666c033333662622222222233
Jinze Bai, Shuai Bai, Shusheng Yang. Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A frontier large vision-language model with versatile abili- ties. CoRR, abs/2308. 12966, 2023. URL hhttps://doot www.oo../..o.. 485.0000.0...0...2... 12966
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agar- wal, Ariel Herbert- Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCan- dlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In NeurIPS, 2020. URL ptttps::/procceee/e ngs neurieurppp. wwtpaper.//..cc/ppcc.. (raper/2020/hahht 111.1966611100000011222006 Abst Absttaetaaaaaa-a-aaaaaaaat html
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In ICCV, pp. 9630- 9640. IEEE, 2021. URL
1119//1CV48922.1.. 2021. 00951
Guo Chen, Yin-Dong Zheng, Jiahao Wang, Jilan Xu, Yifei Huang, Junting Pan. Yi Wang, Yali Wang, Yu Qiao, Tong Lu, and Limin Wang. Videollm: Modeling video sequence with large language models. CoRR, abs/2305.13292, 2023. URL https:///oi..ot oropi0..rg11..... Dt:s:://bo..org/I001853.
arXiv.2305 13292
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey E. Hinton. A simple frame- work for contrastive learning of visual representations. In ICML, volume 119 of Proceedings of Machine Learning Research, pp. 1597-1607. PMLR, 2020a. /RLhtt:::/pproceo httt:::/proceeeinngg
mlr. wt..spres//ve
wttp/0eenn202 htmml
Wenhu Chen, Hongmin Wang, Jianshu Chen, Yunkai Zhang, Hong Wang, Shiyang Li, Xiyou Zhou, and William Yang Wang. Tabfact: A large-scale dataset for table-based fact verification. In ICLR. OpenReview.net, 2020b. https:/ tenttts::/opprrreeet /tppn/eeeeeeeeeeeeeeeeeeeeeeeeee iew. net forum waaerrumiddrkeeeeeee d=rke.JRhNYDH
Wenhu Chen, Hanwen Zha, Zhiyu Chen, Wenhan Xiong, Hong Wang, and William Yang Wang. Hybridqa: A dataset of multi-hop question answering over tabular and textual data. In EMNLP (Findings), volume EMNLP 2020 of Findings of ACI., pp. 1026-1036. Association

---

Published as a conference paper at ICLR 2025
for Computational Linguisties, 2020c. URL Findings- eetpss.-mmp.....
Zhoujun Cheng, Tianbao Xie, Peng Shi, Chengzu Li, Rahul Nadkarni, Yushi Hu, Caiming Xiong, Dragomir Radev, Mari Ostendorf, Luke Zettlemoyer, Noah A. Smith, and Tao Yu. Binding language models symbolic languages. In ICLR. OpenReview.net, 2023. URL https:
//openreview. wttp://eeetttooomdddddidia..
Xiang Deng, Ahmed Hassan Awadallah, Christopher Meek, Oleksandr Polozov, Huan Sun, and Matthew Richardson. Structure- -ereture-groneeed pretraining for text-to-qll. In NAACL-HLT, pp. 1337-1350. Association for Computational Linguistics, 2021. URL https:://ot...... org/1C. 18653/v1/202
-ww.clmaaammmmaa...a mtap://liaa221.nnacnnnniiii 105
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding. In NAACL-HLLT, pp. 4171-4186. As- sociation for Computational Linguistics, 2019. URL
n19 019-14233
Xuemei Dong, Chao Zhang, Yuhang Ge. Yuren Mao, Yunjun Gao, Lu Chen, Jinshu Lin, and Dongfang Lou. C3: zero-shot text-to-sql with chatgpt. CoRR, 0as/2307.07306. 2023. URL
https://dott 1tr.iio.rr/11..031.1
2307 07306
Xi Fang. Weijie Xu, Fiona Anting Tan, Ziqing Hu, Jiani Zhang, Yanjun Qi, Srinivasan H. Sen- gamedu, and Christos Faloutsos. Large language models (Ilms) on tabular data: Prediction, generation, and understanding
survey, Trans. Mach. Learn. Res., 2024, 2024. URL
htttps://epenrese
net
Yujian Gan, Xinyun Chen, and Matthew Purver. Exploring underexplored limitations of cross- domain text-to-sql generalization. In EMNLP, pp. 8926 8931, Association for Computational Linguistics, 2021a. TtRLhttps:/// //t::////oo... wtw.....................
ewnlp-mainnnn...m 702)
Yujian Gan, Xinyun Chen, Jinxia Xie, Matthew Purver, John R. Woodward, John H. Drake, and Qiaofu Zhang. Natural SQL: making SQL easier to infer from natural language specifications. In EMNLP (Findings), pp. 2030-2042, Association for Computational Linguistics, 2021b. URL
https://doi. owopi.orgg100 1ttp:/00..66533vvl1
emnlp.174
Dawei Gao, Haibin Wang, Yaliang Li, Xiuyu Sun, Yichen Qian, Bolin Ding, and Jingren Zhou. Text- to-sql empowered by large language models: A benchmark evaluation. Proc. VLDB Endow.,
1325)1113 2024. URL https: /ww. v db. wttpooggglll/ooooo hwppldbbbvllll. http:///////11111//2ggg////
pdf
Jonathan Herzig, Pawel Krzysztof Nowak, Thomas Müller, Francesco Piccinno, and Julian Martin Eisenschlos. Tapas: Weakly supervised table parsing via pre-training. In ACL, pp. 4330-4333. Association for Computational Linguistics, 2020. URL htttss://doi. http://oo..rg.1...100..
01/2020.ac1-maale 398
Yuchen Hu, Chen Chen, Chao-Han Huck Yang, Ruizhe Li, Chao Zhang, Pin-Yu Chen, and Engsiong Chng. Large language models are efficient learners of noise-robust speech recognition. In ICLR. OpenReview.net, 2024. URL https:ttt..... htttps://ppenreviieeee www. oet
Xin Huang. Ashish Khetan, Milan Cvitkovic, and Zohar S. Karnin. Tabrransformer: Tabular
data modeling using contextual embeddings. CoRR, abs/2012.06678, 2020. URL https:
//axxiv.org/abballa... waaxxv..orr/bb//2222.6666 06678
Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de Las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne I Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mistral 7b. CoRR, abs/2310.06825,, 2023a. URL https:://oi.org////
012555/arXiv.33111 26822
Jinhao Jiang, Kun Zhou, Zican Dong, Keming Ye, Xin Zhao, and Ji- Ri-R Wen. Structgpt: A general framework for large language model to reason structured data. In EMNLP, Association for Computational Linguistics, 2023b. URL https:/ 1tt.aa/.oo/g.0..1333111000000000
eemn 1--mallamlll n.577]
12

---

Published as a conference paper at ICLR 2025
Boyan Li, Yuyu Luo, Chengliang Chai, Guoliang Li, and Nan Tang. The dawn of natural language to SQL: are we fully ready? [experiment, analysis lu0026 benchmark J. Proc. VLDB Endow., 17
(11):3318-332 2024a. URL htttps://wwt v rlab. http://ldd//oo11//33333333
pdf
Haoyang Li, Jing Zhang, Cuiping Li, and Hong Chen. RESDSQL: decoupling schema linking and
skeleton parsing for text-to-sql. In AIlI, pp. 13067-13075, AAAI Press, 2023a. URL https:
//doi.org/10 609/aaai......a..aaa...a...a..aa w37i11....... v37111..2533
Haoyang Li. Jing Zhang, Hanbing Liu, Ju Fan, Xiaokang Zhang, Jun Zhu, Renjie Wei, Hongyan Pan. Cuiping Li, and Hong Chen. Codes: Towards building open-source language models for text-to-sql. Proc. ACM Manag. Data, 2(3):127, 2024b. URL http://///..../11111/1110 3654930
Jinyang Li, Binyuan Hui, Ge Qu, Jiaxi Yang. Binhua Li, Bowen Li, Bailin Wang, Bowen Qin. Ruiying Geng, Nan Huo, Xuanhe Zhou, Chenhao Ma, Guoliang Li. Kevin Chen- Chuan Chang, Fei Huang, Reynold Cheng. and Yongbin Li. Can LLM already serve as A database interface? A big bench for large-scale database grounded text-to-qlss. In NeurIPS, 2023b. URL http::/ nips cc wwpppppppppppprrappppppppp.ppp.p Secaper_fflesse fttes//appe/e///////
_nd
Berchmarks. html
Junnan Li, Dongxu Li, Silvio Savarese, and Steven C. H. Hoi. BLIP-2: bootstrapping language- image pre-training with frozen image encoders and large language models. ICML, volume 202 of Proceedings of Machine Learning Research, pp. 19730-19742. PMLR, 2023c. URL https:
//proceedings . mml.
html
Peng Li, Yeye He, Dror Yashar, Weiwei Cui, Song Ge, Haidong Zhang, Danielle Rifinski Fainman, Dongmei Zhang, and Surajit Chaudhuri. Table-gpt: Table fine-tuned GPT for diverse table tasks. Proc. ACM Manag. Data, 273):176, 2024c. URL pottps://doi.org /10 10. 1144/36554999
Shujie Li, Liang Li, Ruiying Geng, Min Yang, Binhua Li, Guanghu Yuan, Wanwei He, Shao Yuan, Can Ma, Fei Huang, and Yongbin Li. Unifying structured data as graph for tata-to-text pre- training. Trans. Assoc. Comput. Linguisties, 12:210-22 22800 2024d. URL https: //tp:::://oo..o. org/
10.1162/tacl a 00641
Aiwei Liu, Xuming Hu, Li Lin, and Lijie Wen. Semantic enhanced text-to-sql parsing via iteratively learning schema linking graph. In KDD, pp. 1021-1030. ACM, 2022a. URL htttps://dd
org/10. 3Tq/10..145/3534
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In haah/aaah// html
NeurlPS, 2023, URL http::///aapppppppp...e.eeeeeeep f5c22776222237511115333666---200
Qian Liu, Bei Chen, Jiaqi Guo, Morteza Ziyadi, Zeqi Lin, Weizhu Chen, and Jian-Guang Lou. TAPEX: table pre-training via learning a neural execute In ICLR, OpenReview.net, 2022b.
URL wwt.:tttt://tt /
net. fo forum? ftte:/oooommm110004433333 10d100040504450
Tianyang Liu, Fei Wang, and Muhao Chen. Rethinking tabular data understanding with large lan- guage models. In NAACL-HLT, pp. 450-482. Association for Computational Linguistics, 2024.
https://: doi 1ttp://1...653/11/11.
-na.clllonn 1ong..2..222. 26
Haoyu Lu, Wen Liu, Bo Zhang, Bingxuan Wang, Kai Dong, Bo Liu, Jingxiang Sun, Tongzheng Ren, Zhuoshu Li, Hao Yang. Yaofeng Sun, Chengqi Deng, Hanwei Xu, Zhenda Xie, and Chong Ruan. Deepseek-vl: Towards real-world rision-languee understanding. CoRR, abs/2403.05525
2024a. URL htttttttt//so.t..... /ttp://////o...///////// 40000.885500000 a0.55 2403. 05525!
Weizheng Lu, Jiaming Zhang, Jing Zhang, and Yueguo Chen. Large language model for table processing: A survey, CoRR, abs/2402.05121, 2024b.
arXiv. 2402. 05121
Karime Maamari, Fadhil Abubaker, Daniel Jaroslawicz, and Amine Mhedhbi. The death of schema linking? text-to-sql in the age of reasonedd language models. CoRR, abs/2408.07700 2024.
URL htl.stts:: //doi .org/ 48550/88555..8 2408 2770000000022222
13

---

Published as a conference paper at ICLR 2025
Kushal Majmundar, Sachin Goyal, Praneeth Netrapalli, and Prateek Jain. MET. masked encoding for tabular data, CoRR, 0bbs/220.08866. 2022. URL
22066085600
Linyong Nan, Chiachun Hsieh, Ziming Mao, Xi Victoria Lin, Neha Verma, Rui Zhang, Wojciech Kryscinski, Hailey Schoelkopf, Riley Kong, Xiangru Tang, Mutethia Mutuma, Ben Rosand, Is- abel Trindade, Renusree Bandaru, Jacob Cunningham, Caiming Xiong, and Dragomir R. Radev. Fetaqa: Free-form table question answering. Trans. Assoc. Comput. Linguisties, 10:35-49, 2022.
URL httpps://doi.ozr//11111 1tt.///11//...1/aa......
00446
Ankur P. Parikh, Xuezhi Wang. Sebastian Gehrmann, Manaal Faruqui, Bhuwan Dhingra, Diyi Yang, and Dipanjan Das. Totto: controlled table-to-text generation dataset. In EMNLP, pp.
1173 1173-1186. Association for Computational Linguistics, 2020.
18653/v1/2020 emn wmnne-nm-mmmmm commlp-mminnt main..8 89
Panupong Pasupat and Percy Liang, Compositional semantic parsing on semi-structured tables. In ACL, pp. 1470-1480. The Association for Computer Linguistics, 2015, URL htttps://///i htg/10.3115/v1/pp
Mohammadreza
DIN-SQL: decomposed in- context learning of text-to-sql with self-correcioon. In NeurlPS 2023, 2023.
Pourreza and Davood Rafiei.
http: http::/papers. ni www.niiss.ccpppppppppeppppppp wwtpcccppppppeeeeeeeeeee www.fllleeppee petpe/ppa/e2/////aaaaeeeee/eee [222ccc6f$icaaaasccccell33332 Costle--ttttttttttCoooeeeeeeeeee
URL
Mohammadreza Pourreza, Hailong Li, Ruoxi Sun, Yeounoh Chung. Shayan Talaei, Gaurav Tarlok Kakkar, Yu Gan, Amin Saberi, Fatma Ozcan, and Sercan Ö. Arik. CHASE-SQLL multi-path reasoning and preference optimized candidate selection in sext-to-sql. CoRR, abs/2410.01943, 2024. URL htt.t:tttt//ttttt
wrg/10..g..... 8550 arxiv 2410 01 943
Ge Qu, Jinyang Li, Bowen Li, Bowen Qin, Nan Huo, Chenhao Ma, and Reynold Cheng. Before generation, align it! A novel and effective strategy for mitigating hallucinations in text-to-sql generation. In ACL (Findings), pp. 5471-5471. Association for Computational Linguistics, 2024.
URL JuRL/attss:/accaattiie http://acaaaattooo...oooo wwwp:lloooooooooooooooooooeeeooe 2024. Findings-aci 324
Nitarshan Rajkumar, Raymond Li, and Dzmitry Bahdanau. Evaluating the text-to-sql capabilities of large language models. CoRR, abs/2204.00498, 2022. URL sattps://dol org/1C.
48850//axiv. 2204.00498)
Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bert- networks. In EMNIP.JJCNIP, pp. 3980-3990. Association for Computational Linguistics, 2019.
URL BtRp:t 1tt.:pp:://o..oooo0001116////// http:/10..1653/111///// 1410
Baptiste Rozière, Jonas Gehring. Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi. Jingyu Liu, Tal Remez, Jérémy Rapin, Artyom Kozhevnikov, Ivan Evtimov, Joanna Bitton, Manish Bhatt, Cristian Canton-Ferree Aaron Grattafiori, Wenhan Xiong, Alexandre Défossez, Jade Copet, Faisal Azhar, Hugo Touvron, Louis Martin, Nicolas Usunier, Thomas Scialom, and Gabriel Synnaeve. Code llama: Open foundation models for code. CoRR, abs//308..295555 2023. arxiv. /ttp://///.....//..../.x... 2308. 12950
URL
48550
Yucheng Ruan, Xiang Lan, Jingying Ma, Yizhi Dong, Kai He, and Mengling Feng. Language mod- cling tabular data: A survey of foundations, techniques and evolution. CoRR, abs/2400.100488 2024.
http:/1111.11111/.1 arXiv 2408 10548
Ananya Singha, José Cambronero, Sumit Gulwani, Vu Le, and Chris Parnin. Tabular represen- tation, noisy operators, and impacts on table structure understanding tasks in llms. CoRR, abs/2310.10355. 2023. URL htttht//s////o// ott. 485.00 arxiv. 2310 10358
Gowthami Somepalli, Micah Goldblum, Avi Schwarzschild, C. Bayan Bruss, and Tom Goldstein. SAINT: improved neural networks for tabular data via row attention and contrastive pre-training. CoRR, abs/2106.013333 2021. URL BtRLhttts::// /tattps///axiv.... arxiv. org/ abs http//ass///...//// 01342
Yuan Sui, Mengyu Zhou, Mingjie Zhou, Shi Han, and Dongmei Zhang. Table meets LLM: can large language models understand structured table data? A benchmark and empirical study. WSDM, pp. 645-654. ACM, 2024a. URL URIpttpss: //oof .rrg 100 11000..115331111 6855.
14

---

Published as a conference paper at ICLR 2025
Yuan Sui, Jiaru Zou, Mengyu Zhou, Xinyi He, Lun Du, Shi Han, and Dongmei Zhang. TAP4LLM: table provider on sampling, augmenting, and packing stmi-struetured data for large language model reasoning. In EMNLP (Findings), pp. 10306-1032) Association for Computational Lin- ogy www.g........o... org/2024. fi ndi inginng nag-nng-mi emn p. 60.3
guistics, 2024b. htt.htts:///cllee
Ruoxi Sun, Sercan Ö. Arik, Hootan Nakhost, Hanjun Dai, Rajarishi Sinha, Pengcheng Yin, and Tomas Pfister. Sql-palm: Improved large language model adaptation for text-to-sql. CoRR. abs/2306.00739, 2023. URL WRLhhttp:://dot org/10r
Shengbang Tong, Ellis Brown, Penghao Wu, Sanghyun Woo, Manoj Middepogu, Sai Charitha Akula, Jihan Yang, Shusheng Yang, Adithya Iyer, Xichen Pan, Austin Wang, Rob Fergus, Yann LeCun, and Saining Xie. Cambrian-l: A fully open, vision-centric exploration of multimodal Ilms. CoRR, abs/2406. 16860, 2024. URL 16860
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurélien Rodriguez, Ar- mand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models. CoRR, abs/2302. 13971, 2023. URL https://doi. wrw/oo.......................... 48550/
arXiv. 2302. 13971
Wenhui Wang, Furu Wei, Li Dong. Hangbo Bao, Nan Yang. and Ming Zhou. Minilm: Deep self-attention distillation for task-agnostic compression of pre-trained transformers. In NeurlPS, 2020. URL http:://proceedings.ennriiii
cc http:/ppapp////22///aaaaa. 1.ff522233....1110003aaaaaaaaaa.
Zifeng Wang and Jimeng Sun. Transtab: transeraa tabular transformers across tables. In NeurIPS, 2022. URL htt.://////aaaea.............o/a
Zilong Wang, Hao Zhang. Chun-Liang Li, Julian Martin Eisenschlos, Vincent Perot, Zifeng Wang,
Lesly Miculicich, Yasuhisa Fujii, Jingho Shang, Chen-Yu Lee, and Tomas Pfister. Chain-of-table: Evolving tables in the reasoning chain for table understanding. In ICLR. OpenReview.net, 2024.
URL http:http://pptt hattpss//openrevieet wot.ee/oooorridddoooo0
Xianjie Wu, Jian Yang. Linzheng Chai, Ge Zhang, Jiaheng Liu, Xinrun Du, Di Liang, Daixin Shu, Xianfu Cheng, Tianzhen Sun, Guanglin Niu, Tongliang Li, and Zhoujun Li. Tablebench: A comprehensive and complex benchmark for table question answering. CoRR, abs/2408.09174, 2024. /tttpp:://oi..or//110 orr/10.../..ooo (w.8855555XXX. 2408 09174
Yuanzhen Xie, Xinzhou Jin, Tao Xie, Matrixmxlin Matrixmxlin. Liang Chen, Chenyun Yu, Cheng Lei. Chengxiang Zhuo, Bo Hu, and Zang Li. Decomposition for enhancing attention: Improving llm-based text-to-sql through workflow paradigm. In ACL (Findings). Association for Compu- tational Linguistics, 2024. TRR/tttpss//alaanttoot www.:/log.o.o.g///2// Findings-acl.
641
Jingfeng Yang, Aditya Gupta, Shyam Upadhyay, Luheng He, Rahul Goel, and Shachi Paul. Table- former: Robust transformer modeling for table-text encoding. In ACI, pp. 528 537, Associa- tion for Computational Linguistics, 2022. htt.::t::///./
acl sell-oong.o 40
Chao Ye, Guoshan Lu, Haobo Wang, Liyao Li, Sai Wu, Gang Chen, and Junbo Zhao. Towards cross-table masked pretraining for web data mining. In WWW, pp. 4449-4459) ACM, 2024.
URL wttp:://///o......11111...
3645707
Yunhu Ye, Binyuan Hui, Min Yang, Binhua Li, Fei Huang, and Yongbin Li. Large language models versatile decomposers: Decomposing evidence and questions for table-based reason- ing. In SIGIR, pp. 174-184, ACM, 2023. URL
3591708
Pengcheng Yin, Graham Neubig, Wen-tau Yih, and Sebastian Riedel. Tabert: Pretraining for joint understanding of textual and tabular data. In ACL, pp. 8413 8426. Association for Computational Linguistics, 2020. URL htR.hhttss://att /ttp://tea.//bboooo/0..111665//3
i 745)
15

---

Published as a conference paper at ICLR 2025
Tao Yu, Rui Zhang, Kai Yang, Michihiro Yasunaga, Dongxu Wang, Zifan Li, James Ma, Irene Li, Qingning Yao, Shanelle Roman, Zilin Zhang, and Dragomir R. Radev. Spider: large- scale human-labeled dataset for complex and cross-domain semantic parsing and text-to-sql task. In EMNLP, pp. 3911 --29... Association for Computational Linguistics, 2018. URL https: //doi.org// 10.
Jun Zhan, Junqi Dai, Jiasheng Ye, Yunhua Zhou, Dong Zhang, Zhigeng Liu, Xin Zhang, Ruibin Yuan, Ge Zhang. Linyang Li, Hang Yan, Jie Fu, Tao Gui, Tianxiang Sun, Yu-Gang Jiang, and Xipeng Qiu. Anygpt: Unified multimodal L.LM with discrete sequence modeling. In ACL, pp- 9637-9662. Association for Computational Linguistics, 2024. Utt.ttps:: http//oii./gg..////////
18653/v1/2024. aci- acw-loog...aaaaaaaaaaaaaacoallls 521
Tianshu Zhang, Xiang Yue, Yifei Li, and Huan Sun. Tablellama: Towards open large generalist models for tables. In NAACL-HLT, pp. 6024 6044. Association for Computational Linguistics, 2024. BRLLttpss: / //w:i///o..////../ http://o...orr/00.../////////22/ www.cel.2..aaacl1.a.mmaaaa naacl-long....... 335
Ruiqi Zhong, Tao Yu, and Dan Klein. Semantic evaluation for text-to-sql with distilled test suites. In EMNLP, pp. 396-411, Association for Computational Linguistics, 2020. URL https://
doi org 10. (/B33v3vlll /2020 emn p-ma in 2.9
Bingzhao Zhu, Xingjian Shi, Nick Erickson, Mu Li, George Karypis, and Mahsa Shoaran. Xtab: Cross-table pretraining for tabular transformers. In ICML, volume 202 Proceedings of Machine Learning Research, pp. 142044. PMLR, 2023. URL https: httpa////...ccccog....... .mlr.
press/v200 22320222hu23.c.. h
16

---

Published as a conference paper at ICLR 2025
Table Detailed statisties of model architecture and computational overhead per inference.
Learnable
Table Encoder
Adaptor
L.M (LL.AMA3-8B-Isttrul
Parameters
102,910,464
68,734,976
8,030,261,248
Computational
Table Encoding
Feature Alignment
Decoding
Overhead (ms)
36.49977
2.37205
693.60648
A LIMITATIONS AND FUTURE WORK
Due to computational constraints, our experiments are conducted on open-source LLMs with ap- proximately 7B parameters. We anticipate further enhancing TNT by scaling up the training data and evaluating it on more comprehensive benchmarks (e.g., NL2Code) more table structures (e.g.. hierarchical tables) to fully explore its potential. We also hope our work can bring insights to other research focused enhancing LLMs' ability to understand other forms of structured data.
B IMPLEMENTATIONS DETAILS
B. ARCHITECTURE DETAILS
We provide detailed statistics on the number of learnable parameters each component of TNT and their computational overhead during inference in Table 7 The results show that the additional learnable parameters and computational overhead introduced by TnT are minimal, indicating that TNT maintains high efficiency while enhancing table understanding.
B.2 TRAINING SETUPS
When it comes to the training details, during the column-wise contrastive learning stage, we pre- train the Table Encoder on the dataset of 86, 046 tables for 20 epochs, using a batch size 64 and constant learning rate of 10-4, For multi-task feature alignment, the Table Encoder is connected to a downstream LLM through the Adaptor. In this stage, we freeze the LLM and only train the Table Encoder and Adaptor cosine learning rate scheduler is used, with a maximum learning rate of 5 x 10-5, a minimum learning rate of 10-6, and a warmup ratio of 0.05. Lastly, the instruction tuning for all methods is performed on the training set of SPIDER for 2 epochs with a batch size of 128. We use cosine learning rate scheduler with maximum learning rate of 10-6 no minimum learning rate, and a warmup ratio of 0.05. Additionally, a weight decay coefficient of 0.1 is applied. For all training stages, we use the AdamW optimizer with 31 0.9, 83 0.999, and c 10-5, The input embeddings of the decoder are always truncated to maximum length of 4096. Baseline implementations follow the setups described in previous works (Li al.) 2024bb Gao et al.) 2024). All models are trained on 8 A100 GPUs with 80GB of memory. Note that we did not specifically optimize the training hyperparameters; thus, the above configuration may not yield the absolute optimal model performance. We provide detailed statistics about the training data used in the Feature Alignment stage in Table
multi-task data for single epoch with batch size of 1024. A
B.3 DETAILS OF COLUMN- WISE CONTRASTIVE LEARNING
In this section, we provide detailed explanation of the calculation of the contrastive loss in Equa- tion During column-wise contrastive learning, we first apply random row sampling on each table T; the mini-batch to generate two snapshots, S, and Sr, which maintain the same schema but differ in cell content. Corresponding columns in the two snapshots are denoted Table Encoder then processes each snapshot to produce column embeddings, which are further nor- malized using projection head. This process yields embedding pool P consisting of column embeddings from each snapshot, across every tables in the mini-batch. Next, we construct positive pairs by matching column embeddings from the same column but different snapshots (i.c., c; and c(), while embeddings from other columns (which can belong to any tables the minibatch) P form negative pairs for given embedding We use e. denote column embedding that can
and c/. The
17

---

Published as a conference paper at ICLR 2025
Table 8: Details of multi-task pre-training data.
Task
Data Sources
Description
Example Prompt Template
Column Prediction
D (Synthetic)
50, 000
Predict which col-
Given the following database achema:
umn cell value the given row from.
Suppose there 1s rows (<ce)) value>, Lralue>, <selLvaiue>, ...) Which col unn ia the cell value <cell Losloeenost likely fron? Ansxer with the corresgonding column name only, Given the following database schima:
Cell Prediction
D (Synthetic)
50, 000
Predict which cell value from the given column.
cell
[<cell value>,,
Koellvaluu>> ...]
likely
col <col name??
Answer with the correspending cell velue name only.
Question Generation
FetaQA, WikiTableQuestion
20, 935
Ask question based the given answer.
Civen the tollewing databare schema: <database.prontt>
Please propose relevant cueation
answered <answer>,
Table Titling
ToTTo
85, 613
Provide brief duction of the given table.
follewing database schimas
prongt Suppose the table pager please wsite an introductosy Lible table.
web
Row
ToTTo
85, 687
Provide summarization the given row.
brief
Given the follewing database schema: <databane.pronpt)
Summarization
sumnacize folloxing rowe sentence columns nay be shufriad). <EOW>
Total
292, 235
w/ PT
w/o PT
Figure 6: Distributions of column embeddings on SPIDER-Dev produced by TNT with pre-training and TNT without pre-training.
form positive pair with Then, we use Equation to maximize the cosine similarity between positive pairs and minimize it between negative pairs. To further enhance the model's learning, we implement a hard negative sampling strategy within each batch. This strategy prioritizes selecting columns from the same table or database, which are superficially similar and thus harder to dis- tinguish. By focusing features and capture table-specific global characteristics, thereby improving its robustness and un- derstanding of column semantics. In this way, observe that the effectiveness of column- wise contrastive learning and feature alignment heavily depends on the quality of the tables, partic- ularly their diversity and the average number of columns per table.
these challenging negatives, the model is encouraged to extract nuanced
18

---

Published as a conference paper at ICLR 2025
1.0
acc
0.5
TNT
SFT LLaMa3
0.0
1.0
recall
0.5
0.0
1.0
d 0.5
0.0
select
where
group
order
and/or
IUEN
keywords
Figure 7: Detailed EM Accuracy of TNT and SFT LLAMA3-Instruck on non-semantic SPIDER-Dev.
C ADDTIONAL EXPERIMENTS
C.I EMBEDDING VISUALIZATION
We visualize the distribution of output column embeddings generated by TnT and TNT without pre-training. As shown Figure embeddings of different columns produced by TnT without pre-training appear to be more clustered and less discriminative, indicating that, in the absence of pre-training, the column embeddings are prone to overfitting homogeneous task-specific patterns during Feature Alignment and Instruction Tuning. This results in less informative and less valuable embeddings, reducing their effectiveness in enhancing the LLM's table understanding.
C.2 CORRESPONDENCE BETWEEN COLUMN EMBEDDINGS AND COLUMN SEMANTICS
We also compare TNT with variant that uses random column embeddings, where the embed- dings in each slot have 30% probability of not corresponding to the actual column names. As shown in Table the effectiveness of TxT is significantly reduced when using mismatched column embeddings, as these embeddings fail to capture valid column semantics and instead mislead the model's reasoning. This demon- strates that the semanties of each column em- bedding are closely linked to the actual contents in the corresponding columns, which explains why column embeddings help the LLM reason more effectively over table semanties.
Table 9: EX Accuracy with different forms of learnable representations.
Method
Dev
Test
SET
31.7
50.5
SET (w/ Soft Prompts)
30.4
51.5
TNT (w/ Random Col Embs)
34.6
Trans Tab (Wang Sun 2022
23.7
33.0
CM2 Ye al. 2024
17.6
39.2
TnT
44,6
59,9
19

---

Published as a conference paper at ICLR 2025
Table 10: Performance of TNT BIRD-dev. The reported metric execution accuracy.
Method
BIRD-dev (w/o knowledge
BIRD-dev (w/ knowledge)
Original (QWEN2.5-7B)
18.97
31.42
SFT (QWEN2.5-7B)
28.49
44.07
TNT (QWFN2.5-7B)
31.42
49.28
ChatGPT
24.05
37.22
Table 11: Performance of TNT adapted to agent workflow on WikiTQ and TabFact. The reported metries in both benchmarks are accuracy (%).
Method
WikiTQ
TabFact
TNT (QWEN2.5-7B)
61.42
77.80
Text-to-SQQ SQL (GPT-3.5)
52.90
64.71
Binder (GPT-3.5)
56.74
79.17
Dater (GPT-3.5)
52.81
78.01
Chain-of-Tabhe (GPT-3.5)
59.94
80.20
C.3 ADDITIONAL BENCHMARKS
To better demonstrate the generalizability of our method, we follow the same model architecture and training recipe introduced previous sections (but with different training data) extend TNT to other general tabular tasks. Specifically, we first evaluate TNT and corresponding variants on BIRD-dev (Li et al., 20236). BIRD, unlike Spider, emphasizes advanced SQL. syntax, the use of external knowledge, and SQL efficiency. Achieving SoTA performance typically requires strong code generation capabilities often gained through large-scale pre-training on code data plex pipeline designs (Li et al.) 2024a, Maamari et al., 2024; Pourreza et al. 2024 Li et al. 2024b), which are beyond the tabular representation focus of our work. Despite that, from the results in Table IO
com-
still observe significant improvement by leveraging better table representations with TNT. which demonstrates its generalizability on challenging tasks. Additionally, following code/SQL-ddrven agent workflow similar to the ones introduced in Wu al. (2024); Rajkumar jet al. (2022), we further extend TNT to a broader set of tabular tasks, including TableQA, fact ver- ification, numerical reasoning, and data analysis. We evaluate it TableBench (Wu et 2024).
HybridQA (Chen et al. 2020cc). FEVEROUS (Aly et al.) 2021). TabFact (Chen et al. (020b)) and WikiTQ (Pasupat & Liang, 2015). compared with additional basclines (Wu al. [2024] Rajkumar et al. 2022] et al. 2023 Ye et al. 2023 Wang et al. 2024). Experimental results on WikiTQ and TabFact (some of the numbers are reported from Wang et al. (2024) Table [T] that TNT achieves performance par with SoTA methods without any task aae-sseccfiic prompting tricks or com- plex pipeline designs. Notably, TNT based on a 7B open-source LLM,, whereas other baselines built GPT-3.5, Evaluation other benchmarks shows similar result in Table [2
Table 12: Performance of TNT adapted to an agent workflow TableBench, HybridQA, and FEVEROUS.
Method
TableBench (PoT@1) 1)
HybridQA (Acc)
FEVEROUS (Acc)
TableLLM (LLAMA3.1-8B)
25.80
27.61
42.30
TableLLM (LLAMA3-8B)
31.96
27.35
51.50
TableLLM (QWEN2-7B)
21.05
27.12
20.10
TableLLM (CODEQWEN-7B)
26.39
20.14
46,90
TableLLM (DEEFSEEK-7B)
28.39
19.53
18.39
QWEN2.5-Instruce
22,58
51.13
63,30
QwEN2.5-Coder-7B
16.15
51.10
60.70
TNT (QWEN2.5-7B)
36.64
53.17
78.05
20

---

Published as a conference paper at ICLR 2025
Table 13: Performance of TNT variant compared with SoTA methods on the Spider leaderboard.
Method
Spider-test (EX)
TNT LLAMA3-8B Code-Correctio
84.3
MiniSeek (Anonymous)
91.2
DATL-SQI. + GPT-4 Self-Comsistency
Gao et al.
2024
86.6
DAIL-SQL++ + GPT-4 Gao et al.
2024
86.2
DIN-SQL + GPT-4 (Pourreza Rafiei
2023
85.3
C3 + ChatGPT + Zero-Shot
Dong
2023
82.3
RESDSQL-3B + NatQQ (Li al.
2023a
Gan
202
79.9
C.4 ADDITIONAL METHOD COMPARISONS
In this section, provide additional method comparisons between TNT variants and other SoTA methods on the Spider leaderboard. Results in Table [3] that TNT shows competitive perfor- mance with simple prompting techniques and relatively small-scale model.
D CASE STUDY
To more intuitively demonstrate the effect of TNT, we summarized cases where the standard SFT model failed but TNT succeeded. The differences between the two models primarily reflected in four areas: schema linking, column hallucination, content matching, and content reasoning. Representative examples are shown in Figure Notably, the SFT LLAMA3-Instruc model, lacking a strong understanding of database contents, often struggles to align SQL conditional statements with the actual database representations. As illustrated by Figure TNT consistently outperforms the SFT model on critical keywords such as "SELECT" and "WHERE", which are essential for show- casing the model's schema linking capabilities. Besides, the SFT model frequently fails to recognize special representations in the database, such as using "T" to denote "True" in the bottom-right ex- ample. It may also hallucinate column names that do not exist in the schema. These phenomena highlight the SFT model's weak grasp of the overall structural understanding of tables.
MULTIMODAL PERSPECTIVE (ELABORATED)
From the above introduction, we can see that TNT integrates several design principles inspired by existing MMLMs, including:
A Brructurally-essignee Encoder. Similar how Vision Transformers (ViT) in VLMs utilize image partitioning and 2D positional embeddings O. TNT leverages a Table Encoder with a spe- cialized two-dimensional attention mechanism that directly captures the unique structural properties of tables. This approach better encodes row-column correspondences, leading to more accurate semantic representations of the table's contents.
Progressive Feature Extraction. Existing MMLMs typically follow a hierarchical encoding pro- cess (e.g., from patch to image, from word to sentence) (). which integrates local details with global features while maintaining token efficiency. TNT follows a similar strategy by first using the Ta- ble Encoder to produce cell-level embeddings that capture structure-enriched semantics and then aggregating these embeddings through an Adaptor into compact and token-efficient column-level representations.
Scalable Encoder Pre-training. Achieving scalable pre-training requires robust training objec- tives that handle diverse input quality. Previous works that apply common MLM or contrastive learning to tabular data ( typically impose high requirements table curation, thereby limiting their generalizability. To address this, we propose a novel column-wiss contrastive learning objec- tive that imposes constraints the semantic clarity of column names the presence of explicit
21

---

Published as a conference paper at ICLR 2025
Q: Which city has most number of arriving flights?
Q: Which model of the car has the minimum horsepower?
Gold: ESLECT ti city FROM MIRFORTS B8 TI SCIN
FLIGH25 AB 12 on 11 AlrgortCode 12.DestAipprr LIMIT
Gold: SELECT 71 Hodel FACH CAR HAMES AB 1I JOIN 22, hocoepooo ABC LIXT
GRcUp BY ci,city CRDER BY
72 Makeld
OROGR
Pred S80807 clitity FROM airporte AS JOI8 Light An 12 me Ti Cade 19 Deetti GR00P BY DestAirport ORDER BY count(t) DESC
Pred SELECT Model Borseposeε LINI2
OIDER
MPG Cylrdex, Edspl. Horapower. Waght,
Pred SEE3CT m1 oiity FROM airporte AS JOI8 iight on code "port clicity GRDZR BY count (*) RESD L
Pred SELECT Model FROM car namea AB 11 CCIX caze data AB 72 oN 21.MakeId 12.2d GEDER 3Y Borsepowee LINIO
Schema Linking
II. Column Hallucination
What is the average GNP and total population nations whose govemment US territory?
What is official language spoken the country whose head of state Beatrix?
Gold: SRLIEn axg (GNP) , sun Igogu ion) FROM
Gold: SELECT Languugg
country
country WHEAS Rovermenn 1CB Dezcitozy"
sounnsyyangge AS 02 OK 22. Ct,Code 72. counou Cade MHEBR 11, HeedDFState "Bestrix" ANI IxCFficiell
Pred SEL svauNPP) nar Poom ationi FROM
country WHERK Governeen bloon US bec tozy"
Pred SELLECT
J0IN country OK Countzyyooe 12. Code 12. Hesd0EState "Bastrix" AND
SEL3C2 ava( cantpopalation) ERCH country WHEFS govermenttom Perritory"
21. 160feicial "Official"
SELECT Langu FDOM sounttylangueg 21 J018 country As 12 oK T2 CounteyCoda 12, Code WHERE 12, HeadDS3tate "Bastrix" AME Ieofficial
Content Matching
Content Reasoning
Figure 8: Case study on Dev. Pred is from SFT LLAMA3-Instruct and Pred is from TNT.
inter-column relationships. This makes more suitable for scaling across larger and more diverse tabular datasets.
Knowledge Injection via Modal Fusion. Different modalities of data have distinct characteristics and require different approaches for effective understanding. Introducing additional modalities al- lows us to integrate the modality-specifie comprehension capabilities of custom cestom-designed encoders with the superior language understanding and reasoning abilities of L.MMs. This multimodal inte- gration helps overcome the limitations of homogenized input processing in uni-modal LLMs and is fundamentally distinct from traditional tuning-based methods. We argue that this approach is more cost-efficient and offers higher potential for performance improvements.
F LIMITATIONS OF EXISTING TABLE REPRESENTATION LEARNING
1) Structure-awareneet Some methods directly model serialized tabular data using language mod- els (Liu et al. 20226), resorting to a linear comprehension that fails to capture the structural informa- tion inherent in tables. 2) Expressive Efficiency, Some methods model the table content along with other contexts, producing token-level embeddings (Ye et al.) 2024). However, these embeddings often lack a coherent hierarchical structure be compressed effectively, which become fragmented and lack efficiency on larger tables. 3) Generalizability. Prevalant tabular pre-training objectives, such as Masked Language Modeling (Zhu et al., (023) and Column Name Prediction (Yin et al.) 2020), rely on associations between cells or between cells and column names. However, they strug- gle to generalize to tables with more heterogeneous and noisy elements. Lacking any of the above characteristics undermines the effectiveness of table representations, while TNT achieves balance with a dedicated model architecture and tailored training strategy. To address these issues, TNT in- troduces dedicated model architecture that progressively derives column-wise semantics, coupled
22

---

Published as a conference paper at ICLR 2025
with specialized training strategy to enable the model to learn effective table representations that can be seamlessly aligned with LLMs.
23


Reviews:
[{"review_id": "icytfYYMHm", "reviewer": "Reviewer_Y53T", "summary": "The paper proposes a new module for LLMs to better understand the tabular data. The new module, TNT, is inspired by the multi-modal LLMs and encodes the tabular information through a BERT-size encoder. The encoded dense features will be transformed by an adapter and then be inserted during the decoding stage of LLMs. \n\nWhen building the whole pipeline, TNT first pre-trains the encoder and then trains the encoder and the LLM together in an alignment task. Finally, it conducts instruction-tuning for the specific tasks.\n\nThe performance on NL2SQL improves remarkably with the newly proposed method.", "questions": "I am curious to see the performance of TNT over tables of different sizes compared with baselines. This will better support the claim that a longer context may not perfectly solve the table understanding by LLMs.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Table understanding is important to LLMs. And it is vital to find an efficient and effective way to enable LLMs to understand the information represented in the tabular structure.\n2. The proposed method can potentially solve really huge tables regardless of the context size.\n3. The whole training pipeline is well explained and is inspiring to related work.", "weaknesses": "The main issue with this paper is the limited evaluation tasks and baselines. The paper only considers NL2SQL while ignoring many other table understanding tasks, such as Table QA (e.g. TabFact, WikiTQ). It is true that it's challenging to solve table-based problems with LLMs. It is still necessary to compare TNT with other methods where only LLMs are used, such as agent-based methods [1,2,3].\n\nThe comparison is also potentially unfair. TNT is only compared with SFT while TNT requires much more training than SFT. I assume that SFT models are only trained on the training set of the target benchmark, while TNT needs more training during the encoder training and alignment stages.\n\n[1] Cheng, Zhoujun, et al. \"Binding language models in symbolic languages.\" arXiv preprint arXiv:2210.02875 (2022).\n[2] Ye, Yunhu, et al. \"Large language models are versatile decomposers: Decompose evidence and questions for table-based reasoning.\" arXiv preprint arXiv:2301.13808 (2023).\n[3] Wang, Zilong, et al. \"Chain-of-table: Evolving tables in the reasoning chain for table understanding.\" arXiv preprint arXiv:2401.04398 (2024).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Q2FeiSFFcs", "reviewer": "Reviewer_rghh", "summary": "The paper proposes TNT, a multi-modal inspired, efficient training pipeline for structure-enriched table semantics learning, given the current limitations of LLMs in understanding structured data. The proposed method shows improved performance on NL2SQL tasks.", "questions": "1. What is the unique importance of the “column-wise contrastive learning” task? Previous pre-training have tackled objectives at varied granular levels, such as table-level, row-level, sub-table level, cell level; what make the column-wise objective stand out in this work?", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The proposed technique is built upon three pressing challenges of tabular data understanding: the structural information, the varying sizes, and generalize across tables. The proposed TnT method consists of modules that target corresponding modules, such as table encoder and abstract table representation. \n\n2. The training objectives, especially the “multi-task feature alignment” covers potential usage for multiple common table-related tasks, which presumably enhances task generality in downstream.", "weaknesses": "1. Lack of baseline comparison: There have been some efforts in training a general table-oriented LM, such as TableGPT and TableLLaMa. Particularly, it would be informative to compare TableLLaMa as a baseline, to the LLaMa version of TNT method proposed. Currently the baselines are purly out-of-the-box model checkpoints and two training variants (SFT and TNT), so it is kind of obvious that further training would bring increases. Adding additional baselines (such as TableLlama) to prove the effectiveness of the proposed training  recipe would be informative.\n\n2. Lack of table tasks: the only evaluated task is NL2SQL. However, to prove that the proposed model is general across tabular-related tasks, it is highly recommended to evaluate a series of most common tabular tasks, such as table fact verification, table-based analysis generation, and table-based question answering. Particularly because the proposed method TNT claims to apply to bi-directional table structures, it would be helpful to actually test on some benchmarks that features bi-directional tables, such as HiTab and AIT-QA.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vRhM6nQ4Sg", "reviewer": "Reviewer_WAuU", "summary": "This paper proposes a so-called TNT model, which comprises three major components, a table encoder similar to the image encoder that encodes the table; a table-language adaptor, which maps the representations from the table encoder to LLM's textual space; and the standard LLM decoder.\nTo train these components, the paper conducts encoder pre-training, table-language feature alignment, and instruction tuning.\nThe authors conduct experiments on three text-to-SQL datasets, including Spider, Spider-DK, and Spider-Realistic (the latter two are variants built upon the standard Spider dataset).\nThe authors have improved the performance compared to the standard supervised fine-tuning on the three datasets and conducted ablation studies.", "questions": "- Why do you limit your studies to text-to-SQL datasets, especially the Spider dataset and its variants? How does TNT perform on other table-related tasks? \n- Why do you claim `bridging the semantic gap between text and table` in your title? Do you consider it as an over-claim considering the scope of your experiments?\n- Have you checked the performance listed on Spider leaderboard and include those papers in the related works section?", "limitations": "", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- There seems to be lots of training efforts done by the authors, as they need to pre-train the encoder, align it with the decoder and instruction-tune the model.\n- There is some improvement compared to the standard supervised fine-tuning on these datasets.\n- There are ablation studies conducted to help understand the contributions of each component.", "weaknesses": "- The authors only conduct their experiments on three datasets, all focused on text-to-SQL, which extremely limits the scope of this paper, which undermines their claim in the title as `bridging the semantic gap between text and table`. To be more specific, the authors could have conducted their experiments on more datasets from table QA, table-to-text datasets, etc. Even in the domain of text-to-SQL, they do not conduct any experiments on BIRD benchmark.\n- The performance improvement is not significant compared to the SFT baseline. For instance, in Table-2, Llama-3-8B-Instruct achieves 73.0 on Dev while TNT achieves 73.2. This holds for test and other models as well, the improvements are around 0.2, 0.7, 0.6, etc., which makes me wonder whether the improvement indeed is significant.\n- The overall performance on Spider is low compared to the state-of-the-art method. On the Spider leaderboard: https://yale-lily.github.io/spider, the best performed method is 91.2 on test set. Even consider the fact that the authors are using the 8B model, the RESDSQL-3B + NatSQL [1], which is a 3B method, achieves 79.9 on the test set, higher than the number reported in the paper (79.0). In addition, RESDSQL-3B + NatSQL came out in 2023.\n\n## References\n[1] Li, Haoyang, et al. \"Resdsql: Decoupling schema linking and skeleton parsing for text-to-sql.\" Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 37. No. 11. 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "eU0fWEjKaL", "reviewer": "Reviewer_HfF2", "summary": "The paper discusses the limitations of Large Language Models (LLMs) in processing structured tabular data, highlighting that broader context windows in next-generation LLMs do not necessarily enhance their capability to comprehend the structural and semantic aspects of tables. To address this gap, the authors introduce TNT, a table-language model designed to integrate the structural information of tables into LLMs. TNT features a robust and scalable training framework, employing contrastive learning to merge tabular data knowledge with linguistic modalities efficiently. The efficacy of TNT is especially showcased through its performance on the NL2SQL task, where it significantly outperforms traditional text-based table serialization methods, achieving up to 14.4% higher execution accuracy.", "questions": "Q1: What is the metric used in Figure 1 (what does Ex(%) refer to?) and what do set1 and set2 refer to?\n\nQ2: For the contrastive learning, it's not very clear how the positive and negative pairs are collected. I'm thinking whether you mean that (1) you first sample two snapshots $S_i$ and $S_i^{'}$ using random row sampling, then for each snapshot, you calculate the embeddings for each column, for example denoted as $c_j$ and $c_j^{'}$, and then you consider the positive pair as ($c_j$, $c_j^{'}$) if they share the same column, and consider the pair of $c_j$ with other column embeddings as negative pairs?\n\nQ3: Perhaps a question of rigor. I'm curious about whether the proposed method can be applied to some more challenging tasks like HiTab, which are hierarchical tables. It seems the proposed column-based contrastive learning cannot be simply applied to hierarchical tables.\n\nQ4: What are the datasets used for instruction-tuning mentioned in Section 4.3?\n\nQ5: What are the distributions of the datasets you used for pre-training? It seems the portion of anonymized column prediction samples is quite relevant to why TNT performs better on non-semantic versions of datasets.\n\nQ6: What does \"original\" refer to in Table 2? Does it mean few-shot inference or some other prompting methodologies? This question is critical since I refer to the leaderboard of Spider (https://yale-lily.github.io/spider). It seems like when equipping GPT-3.5 with chain-of-thoughts or self-consistency, it can already achieve quite similar or even better performance than the proposed TNT. If so, the major benefits of TNT is that it can support non-semantic column names; however, maybe this can be easily solved by adding schemas or retrieving relevant documents as supplementary information. And I also have concerns whether this can be easily solved with more advanced models using inference like GPT-4 or O1 models as well.\n\nQ7: For Table 3, it's puzzling why the performance trends on the development set do not match those on the test set. For example, when only instruction tuning (IT) is used, omitting column embeddings, the scores are 31.7 on the dev set versus 50.5 on the test set. However, when both feature alignment (FA) and IT are applied, the performance is nearly the same on the dev set at 31.9 but improves to 53.4 on the test set.\n\nQ8: For the hybrid table representation mentioned in line 210, is there any ablation study? Is \"dtype\" an essential property for LM understanding, especially for the NL2SQL task?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "* The idea of treating tabular inputs as a distinct modality is interesting, however, it may somehow be overlapped with conventional text features encoding since the majority of tables are plain text, and the only difference is that tabular data has structure which may inherently be unaligned with the current autoregressive nature of LLMs.\n* The motivation for addressing non-semantic column names is sound, although I have concerns about whether these ambiguity issues can be easily resolved by adding schemas or by retrieving relevant documents as supplementary information in LLMs' inference stage.\n* The paper is well-written and easy to follow.", "weaknesses": "W1: I suggest you rephrase the InfoNCE loss mentioned in Section 4.1 and Figure 4 in case not all readers are familiar with contrastive learning.\n\nW2: The performance of the proposed TNT seems to be limited, especially when considering the leaderboard of Spider (https://yale-lily.github.io/spider). It seems like when equipping GPT-3.5 with chain-of-thoughts or self-consistency can already achieve quite similar or even better performance than the proposed TNT. If so, the major benefits of TNT is that it can support the non-semantic column names; however, maybe this can be easily solved by adding schemas or by retrieving relevant documents as supplementary information. And I also have concerns whether this can be easily solved with more advanced models using inference like GPT-4 or O1 models as well.\n\nSeveral relevant papers should be included in the references:\n\n* Chain-of-Table: Evolving Tables in the Reasoning Chain for Table Understanding (https://arxiv.org/abs/2401.04398)\n* TAP4LLM: Table Provider on Sampling, Augmenting, and Packing Semi-structured Data for Large Language Model Reasoning (https://arxiv.org/abs/2312.09039)\n* StructGPT: A General Framework for Large Language Models to Reason over Structured Data (https://arxiv.org/abs/2305.09645)\n* Large Language Models are Versatile Decomposers: Decompose Evidence and Questions for Table-based Reasoning (https://arxiv.org/abs/2301.13808)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)