Improving Table Understanding with LLMs and Entity-Oriented Search
Authors: Thi-Nhung Nguyen
Hoang Ngo
Dinh Phung
Thuy-Trang Vu
Dat Quoc Nguyen
Venue: COLM 2025 (2025)
Keywords: table understanding
llm
TL;DR: We introduce an entity-oriented search method to enhance table understanding in LLMs, reducing preprocessing and achieving state-of-the-art results.

Abstract: Our work addresses the challenges of understanding tables. Existing methods often struggle with the unpredictable nature of table content, leading to a reliance on preprocessing and keyword matching. They also face limitations due to the lack of contextual information, which complicates the reasoning processes of large language models (LLMs). To overcome these challenges, we introduce an entity-oriented search method to improve table understanding with LLMs. This approach effectively leverages the semantic similarities between questions and table data, as well as the implicit relationships between table cells, minimizing the need for data preprocessing and keyword matching. Additionally, it focuses on table entities, ensuring that table cells are semantically tightly bound, thereby enhancing contextual clarity. Furthermore, we pioneer the use of a graph query language for table understanding, establishing a new research direction. Experiments show that our approach achieves new state-of-the-art performances on standard benchmarks WikiTableQuestions and TabFact.

Published as a conference paper at COLM 2025
Improving Table Understanding with LLMs and Entity- Oriented Search
Thi-Nhung Nguyen', Hoang Ngo², Dinh Phung', Thuy-Trang Vu', Dat Quoc Nguyen²
Monash University, 2Qualcomm Al Research* (nhung. thinguyen, dinh. hrung,phung... trang. vu1 Jemonash. edu, dqt qual com
Abstract
Our work addresses the challenges of understanding tables. Existing meth- ods often struggle with the unpredictable nature of table content, leading a reliance on preprocessing and keyword matching. They also face limitations due to the lack of contextual information, which complicates the reasoning processes of large language models (LLMs). To overcome these challenges, we introduce an entity-oriented search method to im- prove table understanding with LLMs. This approach effectively leverages the semantic similarities between questions and table data, as well as the implicit relationships between table cells, minimizing the need for data preprocessing and keyword matching. Additionally, it focuses entities, ensuring that table cells are semantically tightly bound, thereby enhancing contextual clarity. Furthermore, pioneer the use of graph query language for table understanding. establishing a new research direc- tion. Experiments show that our approach achieves new state-of-the-art performances on standard benchmarks WikiiTbleeQuesioos and TabFact.
table
1 Introduction
Tables are widely used to systematically ganize and present data. Understanding tables is key to addressing various down- stream tasks, such as table-based question answering (Wang et al., 2023a; Lin et al., 2023). As illustrated in Figure 1, the goal is to extract the relevant information from the table to provide accurate answers to users' questions. Recent research has explored us- ing Large Language Models (LLMs) to solve tabular data problems by leveraging their strong performance with prompting (Yang et al., 2023; Nguyen et al., 2023; Xie et al., 2023). One common approach is to convert a natural language question into a struc- tured query (e.g., SQL) and then execute the query on tables to retrieve the final an- swer (Lin et al., 2023; Gemmell & Dalton, 2023; Wang et al., 2024; Nahid & Rafiei, 2024; Liu et al., 2024; Kong et al., 2024).
Year
Title
Episodes
2010
Loose Women
Alternating
2010
Zoo Story
All
Morning
Occasional
2014
Women
same year as Loose Woree but did it infrequently?
Question: Which
Answer: "This morning"
Figure A table question answering example "show" table.
Although tabular data allows users to organize and display information logically in real- life scenarios, it presents unique challenges for LLM-based methods. One of the major challenges the unpredictable nature of the content and formatting within table cells. For instance, in column labeled "address", one cell may contain a full address with the street, city, and zip code (e.g,, "123 Main St, Springfield, 12345"), while another cell may only list the
"Qualcomm Vietnam Company Limited. Qualcomm Al Research is initiative of Qualcomm Technologies, Inc.

---

Published as a conference paper at COLM 2025
city name (e.g,, "Springfield") or even be blank. This inconsistency hinders the performance (errinssfeedt
of query search-based methods. To mitigate this challenge, some works have implemented preprocessing techniques, such normalizing numbers and dates, inputting missing values, removing outliers, and transforming tables into more suitable formats (Gemmell & Dalton, 2023; Zhao et al., 2023; Liu et al., 2024; Nahid & Rafiei, 2024). However, these preprocessing techniques demand significant effort in data analysis, are difficult to adapt to unseen tables, and can lead to unintended consequences if not applied properly. Improper preprocessing may result in information loss, creation of sparse matrices, and disruption of the table's original structure.
Another challenge with tabular data is the requirement for complex reasoning due to limited contextual information. This issue arises because table cells often contain only keywords or short phrases rather than full sentences, and the relationships between cells are frequently implicit rather than explicitly stated. For instance, consider a table that includes "employee" column containing names like "Alice Johnson" and "Bob Smith", along with "status" column indicating their employment status (e.g,, "active," "on leave", or "lerminated"". Without additional context, interpreting the precise meaning of these statuses can be difficult. To address this challenge, some recent studies have focused on chain-of-thougee reasoning, however, this approach requires multi-step reasoning across the entire table, which can be computationally intensive and resource-demanding (Wei et al., 2022; Yao et al., 2023; Chen al., 2023; Wang et al., 2024). Alternative approaches include rephrasing questions, breaking them into sub-queries, or retrieving related columns or rows (Kong et al., 2024; Patnaik et al., 2024; Ye et al., 2023). However, these methods often rely heavily on keyword matching. As result, the retrieved table cells may be irrelevant to one another, increasing the effort required for LLMs to extract the final answer.
(COT)
In this paper, we propose a novel approach based on entity-oriented search to enhance table understanding with LLMs. Our method effectively leverages the semantic similarity between the question and the data stored in the table, along with the implicit relationships between cells. Our approach alleviates the strict data format requirements and the reliance on keyword matching found in related works (Gemmell & Dalton, 2023; Kong et al., 2024; Chen et al., 2024b).
Firstly, we focus identifying entities stored within the table by prompting the LLM. We assume the table contains entities and relations, which may be organized differently depending on the table's structure. An entity can represent real-world object, person, place, or concept, each defined by its attributes-dale cells that provide detailed information. For example, in Figure 1, each show is an entity with attributes like "Year", "Title", and "Episodes". A relationship example is the relation between each entity and its "Year", indicating when a show aired. By structuring data this way, we aim to establish stronger relationships and constraints among relevant table cells, thereby clarifying the context of each entity. This approach contrasts with the original table data, where excessive information often introduced noise and confusion, making it difficult to determine context.
Secondly, we propose an entity-oriented search approach to extract relevant entities and attributes. Specifically, we integrate methods from full-text search, semantic search, and graph search. Full-text search allows for keyword-based searching, while our semantic search method focuses on the semantic similarities between the entities and attributes in the questions and those stored in the table. Additionally, we use graph queries to represent questions and to query the graphs formed by entities and relations, effectively leveraging these relations to achieve more accurate search results. We expect that this approach will minimize the need for data preprocessing and keyword matching, adapting effectively to various ways of phrasing questions since the underlying meanings and relationships between entities and attributes remain consistent, even when different terms are used. Furthermore, we are pioneering the use of graph query language (Cypher) for table understanding by providing the LLM with both the graph schema and the question as input, enabling it to transform the question into a Cypher query statement."
htwpa///o...oo//dd/////////aaaaa (Francis et al., 2018).
2

---

Published as a conference paper at COLM 2025
Our contributions are summarized as follows: (l) We propose an entity-oriented search that effectively leverages the semantic similarities between the entities and attributes in the questions and those in the table, along with the implicit relationships between table cells. This minimizes the need for data preprocessing and keyword matching. (II) Our search approach focuses on table entities, ensuring that table cells are semantically tightly bound. This enhances contextual clarity and strengthens relationships between relevant cells. (III) We are the first explore graph query language (Cypher) to enhance table understanding, introducing a new research direction. (IV) Comprehensive experiments show that our approach achieves state-of-the-art performances.
Related Works
Table understanding encompasses a wide range of tasks such as question answering (QA). Many early works focus on fine-tuning BERT (Devlin et al., 2019) to serve as a table encoder for these tasks, such as TAPAS (Herzig et al., 2020), Table-BERT (Chen et al., 2020), TABERT (Liu et al., 2022), TURL (Deng et al., 2022), TUTA (Wang et al., 2021), and TABBIE (lida et al., 2021). Recently, the superior performance of large language models (LLMs)) with prompting has shifted research focus towards exploring their potential in processing tabular data. A straightforward approach is concatenate task descriptions with the serialized table as string and input them into a LLM to generate a text-based response (Marvin et al., 2023; Cheng et al., 2023; Sui et al., 2024). Additionally, some works have further enhanced performance by utilizing few-shot and curated examples (Cheng et al., 2023; Narayan et al., 2022; Chen, 2023).
To effectively address table-based tasks with large language models (LLMs), recent research increasingly employs external tools instead of relying solely on general text processing. Some works propose generating Python programs and then executing them to extract relevant data (Chen et al., 2023; Gao et al., 2023). Similarly, some works propose using text-to-SQL conversion to extract answers (Rajkumar et al., 2022; Cheng et al., 2023; Ni et al., 2023). However, this approach struggles with complex cases involving intricate tables due to the limitations of the single-pass generation process. In this setup, LLMs cannot modify the table in response to specific questions, necessitating reasoning over a static table. In contrast, mhain-of-though (CoT)-based approach reasons step by step before providing an answer (Chen et al., 2023; Zhao et al., 2024; Yang et al., 2024). To enhance COT,, several methods have been proposed, such breaking down the question into sub-problems (Zhou et al., 2023; Khot et al., 2023), employing a table-filling procedure (Ziqi & Lu, 2023), and incorporating preprocessing operations and SQL execution (Wang al., 2024; Nguyen et al., 2025).
Additionally, self-consistency (SC), proposed by Wang et al. (2023b), is another widely adopted technique in recent sta state-of-the-art research. SC involves sampling diverse set of reasoning paths from ILMMs and selecting the most consistent answer by marginalizing over these paths (Ye et al., 2023; Cheng et al., 2023; Liu et al., 2024). However, a common limitation of both CoT and SC methods is their requirement for substantial number of reasoning samples from LLMs. For example, Cheng et al. (2023) generate 50 samples for each question, while Ye et al. (2023); Liu et al. (2024) require over 100 samples. This results in slower performance and higher computational costs, making them almost impractical for
real-world implementation.
3 Our Approach
In this section, we introduce a novel framework, TUNES, improve table understanding with entity-oriented search. Given a table and related question, the task of TUNES is to generate an answer based on relevant information from the table.
Figure 2 illustrates the architecture of our TUNES, which consists of three main components:
(1) Entity Identification, (2) Entity-Oriented Search, and (3) LLM-based Answer Generation. Entity Identification: To begin, we focus on extracting entities from the table. We utilize
LLM to analyze the table's s structure, such primary keys, column names, and row names,
3

---

Published as a conference paper at COLM 2025
Ausrlant Which show Plosen tet silvsuor?y
Dg LLM fased Ansoee Generatin
Menwee This Nening
Worme Til:: Looa Wenne
Full Text Seardht
Ice: Loose Whrad)
2010
hoa 2314 triawde All
Looas Women (2:10)
tin Titer (aose Worao
Senuntic Search
prinary
DasA
Ranodax
The Herning 20100
Tra Thi
Ece Seryy 2015
deo Mary
Tai schoolicce
offend og
Salocder.A All
2012
Verning
Docua
Tile This Monking
wnty
2214
Worer
14
Gragh Search
Thex 2010
The SWA
2015
Erety ldvetfication
Locar
Wirner
Jchedue
Figure 2: Overview of our proposed framework TUNES. The embedding space and the graph are simplified for illustration purposes.
StowED PG21
ShowlD PDC2
Loose
Losse
200
200n
Lo00e
Atbman
Hopy schoodale
Veer
2010
2010
2014
Fyteodden Altenrttin Alterniing All
Tite
Altenakinng Altenatinn
Atribiut
Namoea
Skary
lcaniicaion
Untoning tmplich Rael motoraaaa
Figure 3: Entity Identification Example.
to identify attributes and relationships, thereby identifying entities. Entity-Oriented Search: We then construct graph from these entities, attributes, and their relationships. During this process, we remove attributes without values and merge those sharing values to streamline the graph. Simultaneously, we embed the entities, attributes, and user questions into an embedding space, enabling an entity-oriented search that integrates full-text, vector, and graph search. LLM-based Answer Generation: Finally, we extract the top K most relevant entities to the question and input them into the LLM to generate accurate answers.
3.1 Entity Identification
This subsection describes our approach to detecting attributes and relationships, which enables the identification of entities within data table.
The first step is to find the primary key, which can be a single attribute or combination of attributes that uniquely identifies each entity, typically found in the column or row names. To locate the primary key, we simply use the first I rows and h columns input to prompt the LLM. This process enables us to identify the corresponding attribute column or row and subsequently recognize the entities. For example, in Figure/ 1, combining "fitle" and "year" could serve as the primary key to distinguish the entities within the table. In this case, attributes, and each entity is identified by its attributes across
"year," "tille, " and "episodes" these columns. Note that, if the primary key is absent from the table, the LLM responsible for generating it and specifying its position (See our prompt in Table 4 in Appendix A).
To unbox the implicit relationships in the table, we aim to explore how each entity is related to its attributes. We achieve this by providing the primary key and attribute names to the LLM, prompting it to generate the relationships (See our prompt in Table in Appendix A). Figure 3 illustrates example of the entity identification process.
4

---

Published as a conference paper at COLM 2025
Question How many more times does Loose Women then This Merning?
Cyphar Quary
Creata
Rasuun
Zoo Story 2010
occational
schotle
This Morning 2010
Eriles
Hov maro year das aoca Rocas
retaled 3D
Loose Women 2014
2010
question
has sohodulo
2014
colouiines
aternating
Women
2010
Figure 4: Cypher Query Execution Process. To answer the question "How many more times does Loose Women air than This Morning?", the Cypher query execution process first calculates the number of times "Loose Women" airs, creating new attribute called CoonttoooeeWomnn" It then performs similar calculation for "This Morning', generating an attribute called "countThisMorric "countLooseWWomet" and "countThiisMoringg" creating an attribute named "difference". Note that the relationship "is "s_calculated" is used solely to illustrate the calculation process. Next, the process creates new entity representing the question and determines how this entity relates to the newly created attributes, using relationships such as "related_to _to" and "has answer". Finally, the process returns a complete entity that effectively answers the question by providing a comprehensive analysis of the attributes' impact. See the corresponding Cypher query for the input question in Appendix B. We pioneer the use of the graph query language Cypher to improve table understandding.
The process calculates the difference between
3.2 Entity-Oriented Search
Graph Search: For each table, we construct a graph g N,, where N is the set of nodes and & is the set of edges. N represents the entities and attributes stored in the table, formed by the union of entity nodes and attribute nodes. To preserve the position of nodes in the table, we store their table addresses as properties of each node. E represents the relationships within the table, established by the relationships between the entity and its attributes.
Node disambiguation: We exclude attributes that lack values, which results in sparse data in the original table. Also, some attributes may refer to the same value or have synonymous meanings. For example, Table 1, cells (1,0), (2,0), and (3,0) all correspond to the year "2010", and these should be merged into single node. We accomplish this by filtering based on the semantic similarity score between two node names in an embedding space, as defined in the next paragraph. To query the graph, we provide the LLM with both the graph schema and the question as input, prompting it to convert the question text into a Cypher query statement (See our prompt in Table 6 in Appendix A). We then execute this Cypher query on the graph to search relevant entities and attributes. In addition to searching for relevant entities and verifying the constraints that must be satisfied in questions, the Cypher query execution process enables complex calculations on attributes and entities. It allows for the automatic generation of new entities along with their associated attributes, as illustrated in Figure 4, making graph search a crucial component of TUNES.
Semantic Search: While constructing the graph, we simultaneously map both the entity nodes and the user's question into an embedding space using a text embedding model. This is to calculate the semantic search scores by determining the similarity between the question and the entities. We measure this similarity using the cosine similarity between their representation vectors. Specifically, we obtain the question's 's representation by directly inputting the question text into the text embedding model. For each entity node, we generate a representation by feeding the embedding model a concatenation of node names from a sub-graph of depth d, with the entity node as the root.
5

---

Published as a conference paper at COLM 2025
Full-text Search: Full-text search determines relevance by considering the number of keyword matches and their frequency. We use the BM25 algorithm to rank table entities based on how well they align with the given question (Robertson & Zaragoza, 2009). First, we create a search document for each entity by combining its primary key and attributes. Then, we calculate search score for each entity using BM25, comparing the question to its search document, with higher scores indicating a stronger match to the question.
Entity-Oriented Search Approach: Entity-Oriented search combines full-text search, se- mantic search, and graph search to provide more relevant search results. Specifically, we extract the top K relevant entities by calculating a weighted sum of their full-text and seman- tic search scores, while also retrieving entities and attributes output from the Cypher query execution process. Full-text search quickly identifies exact terms or closely related ones, even in large datasets. Semantic search focuses understanding the underlying context or meaning of the question. Graph search addresses constraints and implicit relationships within the question. This approach ensures that even if a question does not precisely match a relevant entity, the relevant meanings and satisfied relational constraints within the graph will still allow it to be retrieved.
3.3 LLM-based Answer Generation
The top K relevant entities are incorporated into prompt and input into the LLM to generate response (See our prompt in Table in Appendix A). Each entity is treated as paragraph, with the primary key serving as a heading that introduces the main topic, while the attributes and relations provide further details and elaboration. Combining these entities is akin to constructing a document, where each paragraph, representing entity, can stand alone but may also connect to other paragraphs two entities share the same attribute. This approach aligns well with LLMs, which are extensively trained on text documents, reducing the need for complex reasoning and resulting in a more accurate answer.
Experiment Setup
Following previous works (Wang al., 2024), we conduct experi- ments on the benchmark datasets WilkilableQuestions-aa question answering dataset over semi-structured tables (Pasupat & Liang, 2015) and TabFact-a dataset for table-based fact verification (Chen et al., 2020). See Appendix C for the statistics of their test sets.
Datasets & Metric:
We employ the binary classification accuracy for TabFact and the official denotation accuracy for WikittableQuesttons (Pasupat & Liang, 2015).
Baselines: We compare TUNES with strong baselines, including: (1) Methods based on self- consistency (SC) chain-of-thought (CoT), which require a substantial number of reasoning inferences with LLMss: Dater (Khot et al., 2023), BINDER (Cheng et al., 2023), CHAIN-OF- TABLE (Wang et al., 2024) and DP&PYAGENT (Liu et al., 2024). (II) Methods without SC and COT,, which only require a few number of LLM inferences: TEXT2SQL (Rajkumar et al., 2022), ChatGPT (liang et al., 2023), StructGPT (Jiang et al., 2023), TableRAG (Chen al., 2024b) and TabSQLify (Nahid & Rafiei, 2024).
Implementation Details: Following previous works, we use GPT-3.5-turbo as the LLM. Additionally, we report our scores with other LLMs, including GPT-4o-mini and the open- weight LLMs Llama-3.1-88 1ia---3.-------nttruc and the OpenAl API to run inferences with GPT-3.5-turbo and GPT-40-mini, while vLLM (Kwon et al., 2023) is used for inference with -.ama--.1-8B-Insttte Instruct and Llama-3.1-700-Insttt For all these models, we set the temperature values to 0.4 for Text-to-Cypher generation and 0.0 for answer generation.
(Dubey et al., 2024). We use
To extract primary keys (see Section 3.1), we set h to 5. We use Neo4] to interact with graphs (see Section 3.2). We merge attribute nodes through exact matching and filter them by
2https://neo43 http://oow...oooooos//////eaaaaa

---

Published as a conference paper at COLM 2025
Approach
Q
TabFact
Dater (Khot et al., 2023) [SC] (*)
52.8
78.0
BINDER (Cheng et al., 2023) [SC]
56.7
79.2
GPI-3.5-turbo
CHAIN-OF-TABLE (Wang et al., 2024) [CoT] (*)
59.9
80.2
DP&PYAGENT (Liu et al., 2024) [SC] (*)
65.5
80.0
Our TUNESGFT335-uboo [CoT]
68.5
81.5
StructGPT (Jiagg et al., 2023)
48,4
ChatGPT (fiang al., 2023)
51.8
TableRAG (Chen et al., 2024b)
57.0
TEXT2SQL (Rajkumar et al., 2022)
52.9
64.7
TabSQLify (Nahid & Rafiei, 2024)
64.7
79.5
Our TUNESGPT-355ttumm
64.9
79.8
CHAIN-OF MIIN-OF-TABIE TABL BLL----T-LECC---m-I ECP--to-min (*)
70.4
85.8
DP&PYAGENT coT--omnnni mmn-oommmi [SC]
74.7
89.9
Our TUNESGPTT4o-minn
72.3
84.7
Our TUNESGPT-O--mml [CoT]
75.4
90.4
CHAIN-OFIIALLES Laama--33 3.0.-0.1111Bsttrec [CoT]
70.1
85.6
Others.
DP&PYAGENT Llama-3.11700
Our TUNES 1-0---111------ttrctt [SC]
67.9
85.1
75.4
85.6
Our TUNES
[CoT]
75.7
87.5
[CoT]
56.0
49.6
DP&PYAGENT Llama-31168--Imstt [SC]
57.3
63.8
Our TUNES TUNESLLama-B.. Llama-3.1-8--nstteee
54.1
68.1
Our TUNES La-a-3-1-80-Insseee
[CoT]
57.8
71.9
Table 1: Performance results on the WikiiTableQuestionss (WikiTQ) and TabFact test sets. [SC] and [CoT] denote approaches based on self-consistency and thain-of-though respectively. In rows 2-15, results for previous methods are taken from their respective works, except for Dater, BINDER, CHAIN-OI-TABLE and DP&PYAGENT, which are marked with (*), are taken from Nguyen et al. (2025). In the last rows, we run the official implementa- tions of CHAIN-OF-TABLL (https:/ //gi thub. com/ htt//google-ooe googlo--esearceeee DP&PYAGENT (https://a :ttttp:://iiii //gituub.....t www.///o///oeeaeeeeoooooo http///tttt//bblllnn using Llama-3.1 3.1-ma--.-----nsttcccc and Llama-3.1-700-Instuuct
wwwaarchcchiii-eee table) and
cosine similarity greater than 0.95. For text embeddings, we use bge-m3 (Chen et al., 2024a) as our embedding model, setting d 2 when embedding nodes (see Section 3.2 Semantic Search). For the entity-oriented search approach (see Section 3.2), the weight of each search component
1, and for the top K most relevant entities, we set K 50.
We also develop variant of TUNES with COT. Specifically, we require the model to answer the question step by step by iteratively executing the search and answer processes. Simultaneously, the LLMM-based Answer Generation is required to reason step by step before delivering the final answer. The maximum number of iterations is set to 3.
TUNES [CoT]:
Results
5.1 Main Results
Using GPT-3.5-turbo as the base LLM, as shown in Table 1, when compared to baselines employing SC and COT, our TUNESGrT335 turboo [CoT] outperforms the previous state-of- the-art (SOTA) model DP&PYAGENT by 3.0% points on WikiTTaleeQuetioon and 1.5% on TabFact. Notably, TUNESGPTA [CoT] requires fewer intermediate responses from the LLM (the total is 8, including for table analysis, 3 for searching and for answering), compared to CHAIN-OI-TABLE (25), Binder (50), Dater (100) (Wang et al., 2024), and DP&PYAGENT (50-150) (Liu et al., 2024). Note that the average time for performing both semantic search and full-text search is very small, only at 0.06 seconds per query CPU. Thus, in TUNES, almost all of the running time is spent on prompting LLMs. As result, using ontity-oriented search has allowed to reduce the LLM inference cost per
7

---

Published as a conference paper at COLM 2025
question by factor of 2 to 150/8 ~ 18. In addition, our original SUUNEEGPT-3.-turbbe surpasses all baselines not utilizing CoT and SC, outperforming TEXT2SQL by 12% on WikiTableQQeetiions and 15.1% on TabFact, while achieving 0.2+% improvement over the previous SOTA model Tab on both datasets.
In the last 12 rows of Table 1, benchmark TUNES against SOTA methods, including CHAIN-OF-TABLE and DP&PYAGENT, across different LLMs, including Llama-3.1-8 Instruct, 1lama-3.1-70B-nnas Instrucc, GPT-4o-mini to further evaluate the adaptability of our approach. The results show that the original TUNES demonstrates competitive performance, while TUNES [CoT] consistently outperforms both CHAIN-OF-TABLE and DP&PYAGENT across all examined LLMs and datasets.
The performance of TUN remains consistent with the results previously observed on GPT-3.5-turbo In detail, compared to TPPPPYGGENGrT-4mmeee TUNESGFT44-min is 2.4% points lower on WikiTQ and 5.2% points lower on TabFact. How- ever, TUNESGPT-O-ninn surpasses
[SC],
[CoT] on WikiTQ by 1.9% points, despite being 1.1% points lower on TabFact. When augmented with CoT reasoning, TUNES [CoT] surpasses both baselines, exceeding
[CoT] by 5.0% points on WikiTQ and 4.6% points on TabFact, while also outperforming DP&PYAGENTGPT---om [SC] by 0.7% points on WikiTQ and 0.5% points on TabFact.
For open-source LLMs, TUNES demonstrates better adaptability than the baselines. L 1---aa3-1----Imsttueeee notably outperforms CHAIN-OF-TABLEIj 1.m---L--LLam-1111---sttttu [CoT] by 18.5% points and Lama tion, Llmaaaa..1-000-mmstttte even surpass both baselines on both datasets, demonstrating
[SC] by 4.3% points on TabFact. In addí-
a 5.3% points improvement on WikiTQ over CHAIN-OF- BL.Biaam--31-00--nnaae [CoT]
LJama-Jaaa3..... --8---11-st-tttccc [CoT] and TUNES L1 and 7.5% points on WikiTQ and 0.5% on Tabfact over 31.ma-3.111100 7-77--nnsttc [CoT] con- [SC]. Meanwhile, sistently achieve SOTA performance on both datasets. All obtained results demonstrate the effectiveness of our proposed approach.
Overall, TUNES without CoT requires 6x 36x fewer LLM calls than the baselines, re- sulting in significant computational savings. Despite this efficiency, TUNES still achieves competitive performance across four different LLMs, demonstrating its generalizability and effectiveness. Meanwhile, when combined with CoT, TUNES requires 3x-18x fewer LLM calls, while significantly outperforming state-of-the-aae baselines such as CHAIN-OF-TABLE and DYGGYNGE,, with statistical significance (p < 0.01)3 on both the WikiTQ and TabFact datasets.
5.2 Ablation Study
To assess the impact of each proposed com- ponent of TUNÉS, we conduct an ablation study by evaluating different ablated ver- sions of TUNESGPT-t-minni Given budget constraints, we evaluate these ablated vari- ants on randomly selected subset of 1,000 questions from the WikkiTableQuestions test set. Table 2 presents the results for the full-
Approach
WikiTQ
TUNESCPT-4-min
72.0
without Entity Identification
61.3
without Graph Search
62.0
without Semantic Search
69.7
without Full-text Search
70.2
component TUNESGPT alongside the ablation study results on this subset.
Table 2: Ablation results for TUNES
We exclude the entity identification component from TUNES. Instead of searching for entities, the search strategy shifts from entity-oriented approach to a row-oriented one. That is, each row becomes a search document for both full-text and semantic searches and serves as node in the graph without any relationships. The results indicate that this shift from entity-oriented to row-oriented search reduces accuracy, with drop of 72.0% 61.3% 10.7% compared the full TUNES. This shows that clarifying the
Without Entity Identification:
Based one-sided McNemar tests from in-house experiments.
8

---

Published as a conference paper at COLM 2025
Error Type
Description
%
Exa.
[Entity] Table structure
The LLM incorrectly identifies the positions primary keys and attributes.
4%
D.1.1
identification
[Search] Insufficient entity retrieval
Entity-oriented search fails to retrieve ad- equate number of entities required to answer the question.
8%
D.2.1
[Search] Incorrect self- generated entity
Cypher execution process generates inaccu- rate information.
50%
D.2.2
[Answer] Comparative er- ror [Answer] Numerical error [Answer] Logical error
The LLM incorrectly compares quantities such
8%
D.3.1
distance or time.
The LLM performs incorrect calculations.
24%
D.3.2
The LLM incorrectly extracts and utilizes pro- vided information in the reasoning process. N/A
20%
D.3.3
[Answerl Others
2%
Table 3: Error types by components-Ennte Identification (denoted as [Entity]], Entity- Oriented Search (denoted [Search]], and LLM-based Answer Generation (denoted as (Answer]--in TUNES 1-770B-Instruct The total percentage does not add up to 100% as some samples contain more than one error. See error examples (Exa.) in Appendix D.
context for table entities, along with the relationships between entities and their attributes, enhances performance, underscoring the effectiveness of the proposed approach.
Graph search is excluded from the search strategy, leaving the task of searching solely to full-text and semantic searches. As shown in Table 2, the removal of graph search reduces performance, resulting in a 10% decrease in accuracy. This decrease shows that utilizing Cypher, graph query language, to query the graph effectively leverages the relationships between entities and attributes, along with the constraints that must be relevant entities in the search results.
Without Graph Search:
satisfied in the question. As result, it produces
Without Semantic Search:
Semantic search is excluded from the search strategy. As shown in Table removing semantic search decreases TUNES's performance by 2.3%.
Without Full-Text Search: Full-text search is excluded from the search strategy. Although this removal negatively impacts TUNES's performance, the decrease of 1.8% is not as large as with the removal of other components. This shows that the integration of graph queries and semantic search in TUNES reduces reliance keyword matching.
5.3 Error Analysis
Table 3 reports the types of errors across each component from TUNES Jama-3.1-70B-Innstt on WikiTTableQuestion.
Entity Identification Errors: The LLM excels in entity identification within tables, maintain- ing an error rate of just 4%. Most errors occur in tables with complex structures, especially those with duplicated row and column names.
Entity-Oriented Search Errors:
Here, 58% of the errors are related to the quality of the entity-oriented search. Among these, only 8% are due to an insufficient number of retrieved entities, which can result from Cypher query syntax errors or questions requiring large number of entities. The main issue lies in our Cypher execution process, which generates inaccurate new entities and attributes due to mistakes in intermediate calculations, such as incorrectly selecting entities calculating functions.
The biggest challenge for the LLM is performing calculations, such addition and subtraction, which have an error rate of 24%. The second major challenge is logical errors, with an error rate of 20%. These errors occur because the LLM does not fully understand the question or the table's content, leading to incorrect information extraction. Other challenges include errors in comparing complex quantities, such determining which athlete finishes race the fastest.
LLM-based Answer Generation Errors:

---

Published as a conference paper at COLM 2025
Overall, TUNES still faces challenges related to inaccurate information generated from Cypher queries and the limitations of the LLM in calculations, comparisons, and reasoning.
Conclusion
We propose novel approach TUNES to tackle the challenges of table understanding, with three main goals: (1) effectively leveraging the semantic similarities between questions and table data, along with the implicit relationships between table cells, reduce the need for data preprocessing and keyword matching; (2) ensuring that table cells are semantically tightly connected to enhance contextual clarity; and (3) pioneering the use of graph query language (Cypher) to improve table understanding. Experimental results show that TUNES achieves a state-of-the-art performance. In the future, we plan to extend TUNES to address other complex downstream tasks related to table understanding. Our TUNES implementation is publicly available at: https: / http///gtttt.... com/nhungntt/TUNES.
Acknowledgement
This work was supported by Monash eResearch capabilities, including M3.
This work was completed while Hoang Ngo and Dat Quoc Nguyen were at Movian AI, Vietnam.
References
Jianlyu Chen, Shitao Xiao, Peitian Zhang, Kun Luo, Defu Lian, and Zheng Liu. M3- Embedding: Multi-Linguality, Multi-Functionality, Multi-Grauulrrity Text Embeddings Through Self-Knowledge Distillation. In Findings of the Association for Compulational Linguistics: ACL 2024, pp. 2318-2335, 2024a.
Si-An Chen, Lesly Miculicich, Julian Martin Fisenschlos, Zifeng Wang, Zilong Wang, Yanfei Chen, Yasuhisa Fujii, Hsuan-Tien Lin, Chen- Yu Lee, and Tomas Pfister. TableRAG: Million- Token Table Understanding with Language Models. In Proceedings of The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024b.
Wenhu Chen. Large Language Models are few Table Reasoners. In Findings of the Association for Computational Linguistics: EACL 2023, pp. 1120-1130, 2023.
Wenhu Chen, Hongmin Wang, Jianshu Chen, Yunkai Zhang, Hong Wang, Shiyang Li, Xiyou Zhou, and William Yang Wang. TabFact: A Large-scale Dataset for Table-based Fact Verification. In Proceedings of the 8th International Conference Learning Representations, 2020.
Wenhu Chen, Xueguang Ma, Xinyi Wang, and William W. Cohen. Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Tasks. Transactions on Machine Learning Research, 2023.
Zhoujun Cheng, Tianbao Xie, Peng Shi, Chengzu Li, Rahul Nadkarni, Yushi Hu, Caiming Xiong, Dragomir Radev, Mari Ostendorf, Luke Zettlemoyer, Noah A. Smith, and Tao Yu. Binding Language Models in Symbolic Languages. In Proceedings of the 11th International Conference on Learning Representations, 2023.
Xiang Deng, Huan Sun, Alyssa Lees, You Wu, and Cong Yu. TURL: Table Understanding
through Representation Learning. ACM SIGMOD Record, 51(1):33-40, 2022.
Jacob Devlin, Ming-Was Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers), pp. 4171 47816 2019.
10

---

Published as a conference paper at COLM 2025
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The Llama Herd of Models. arxiv preprint, arXiv:2407.21783 2024.
Nadime Francis, Alastair Green, Paolo Guagliardo, Leonid Libkin, Tobias Lindaaker, Victor Marsault, Stefan Plantikow, Mats Rydberg, Petra Selmer, and Andrés Taylor. Cypher: An Evolving Query Language for Property Graphs. In Proceedings of the 2018 International Conference on Management of Data, pp. 1433-1445, 2018.
Luyu Gao, Aman Madaan, Shuyan Zhou, Uri Alon, Pengfei Liu, Yiming Yang, Jamie Callan, and Graham Neubig. PAL: Program-aided Language Models. In Proceedings of the 40th International Conference on Machine Learning, pp. 10764-10799, 2023.
Carlos Gemmell and Jeff Dalton. ToolWriter: Question Specific Tool Synthesis for Tabular Data. In Proceedings of the 2023 Conference Empirical Methods in Natural Language Processing, pp. 16137-16148, 2023.
Jonathan Herzig, Pawel Krzysztof Nowak, Thomas Mueller, Francesco Piccinno, and Julian Eisenschlos. TaPas: Weakly Supervised Table Parsing via Pre-training, In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Pp. 4320-4333, 2020.
Hiroshi lida, Dung Thai, Varun Manjunatha, and Mohit lyyer. TABBIE:
Pretrained Rep- resentations of Tabular Data. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 3446-3456, 2021.
Jinhao Jiang, Kun Zhou, Zican Dong, Keming Ye, Wayne Xin Zhao, and Ji-Rong Wen. StructGPT: A General Framework for Large Language Model to Reason over Structured Data. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 9237-9251, 2023.
Tushar Khot, Harsh Trivedi, Matthew Finlayson, Yao Fu, Kyle Richardson, Peter Clark, and Ashish Sabharwal. Decomposed Prompting: A Modular Approach for Solving Complex Tasks. In Proceedings of the 11th International Conference on Learning Representations, 2023.
Kezhi Kong, Jiani Zhang, Zhengyuan Shen, Balasubramaniam Srinivasan, Chuan Lei, Christos Faloutsos, Huzefa Rangwala, and George Karypis. OpenTab: Advancing Large Language Models as Open-domain Table Reasoners. In Proceedings of the 12th International Conference on Learning Representations, 2024.
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and lon Stoica. Efficient Memory Management for Large Language Model Serving with FagedAttention In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023.
Weizhe Lin, Rexhina Blloshmi, Bill Byrne, Adria de Gispert, and Gonzalo Iglesias. An Inner Table Retriever for Robust Table Question Answering. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp- 9909-9926, 2023.
Guang Liu, Jie Yang, and Ledell Wu. PTaa: Using the Pre-trained Language Model for Modeling Tabular Data Xiv preprint, arXiv:2209.0806) 2022.
Tianyang Liu, Fei Wang, and Muhao Chen. Rethinking Tabular Data Understanding with
Large Language Models. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pp. 450-482, 2024.
Ggaliwango Marvin, Nakayiza Hellen, Daudi Jjingo, and Joyce Aakkatumba-Nabende Prompt Engineering in Large Language Models. In Proceedings of the International Confer- ence Data Intelligence and Cognitive Informatics, pp. 387 402, 2023.
11

---

Published as a conference paper at COLM 2025
Md Nahid and Davood Rafiei. TabSQLify: Enhancing Reasoning Capabilities of LLMs Through Table Decomposition. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pp. 5725-5737, 2024.
Avanika Narayan, Ines Chami, Laurel Orr, and Christopher Ré. Can Foundation Models Wrangle Your Data? Proceedings of the VLDB Endowment, 16(4):738-746, 2022.
Thi-Nhung Nguyen, Hoang Ngo, Kiem-Hieu Nguyen, and Tuan-Dung Cao. A Self- enhancement Multitask Framework for Unsupervised Aspect Category Detection. In Proceedings of the 2023 Conference on Empirical Methods Natural Language Processing, pp. 8043-8054, 2023.
Thi-Nhung Nguyen, Hoang Ngo, Dinh Phung, Thuy-Trang Vu, and Dat Quoc Nguyen. Planning for Success: Exploring LLM Long-term Planning Capabilities in Table Under- standing. In Proceedings of the SIGNLL Conference on Computational Natural Language Learning, 2025.
Ansong Ni, Srini Iyer, Dragomir Radev, Ves Stoyanov, Wen-tau Yih, Sida I Wang, and Xi Victoria Lin. LEVER: Learning verify language-to-code generation with execution. In Proceedings of the 40th International Conference on Machine Learning, pp. 26106-26128 2023.
Panupong Pasupat and Percy Liang. Compositional Semantic Parsing Semi-Structured Tables. In Proceedings of the 53rd Anmual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 1470-1480, 2015.
Sohan Patnaik, Heril Changwal, Milan Aggarwal, Sumit Bhatia, Yaman Kumar, and Balaji Krishnamurthy. CABINET: Content Relevance-based Noise Reduction for Table Question Answering. In Proceedings of the 12th International Conference on Learning Representations, 2024.
Nitarshan Rajkumar, Raymond Li, and Dzmitry Bahdanau. Evaluating the Text-to-SQL Capabilities of Large Language Models. arXiv preprint, arXiv:2204.00495 2022.
Stephen Robertson and Hugo Zaragoza. The Probabilistic Relevance Framework: BM25 and Beyond. Found. Trends Inf. Reir., 333 2009.
Yuan Sui, Mengyu Zhou, Mingjie Zhou, Shi Han, and Dongmei Zhang. Table Meets LLM: Can Large Language Models Understand Structured Table Data? A Benchmark and Empirical Study. In Proceedings of the 17th ACM International Conference on Web Search and Data Mining, pp. 645-654, 2024.
Dingzirui Wang, Longxu Dou, and Wanxiang Che. A Survey on Table-and- HybridQA: Concepts, Methods, Challenges and Future Directions. arv preprint, arXiv:2212.13465, 2023a.
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-Consistency Improves Chain of Thought Reasoning in Language Models. In Proceedings of the 11th International Conference on Learning Representations, 2023b.
Zhiruo Wang, Haoyu Dong, Ran Jia, Jia Li, Zhiyi Fu, Shi Han, and Dongmei Zhang. TUTA: Tree-based Transformers for Generally Structured Table Pre-training. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, pp. 1780-1790, 2021.
Zilong Wang, Hao Zhang, Chun-Liang Li, Julian Martin Eisenschlos, Vincent Perot, Zifeng Wang, Lesly Miculicich, Yasuhisa Fujii, Jingbo Shang, Chen-Yu Lee, and Tomas Pfister. Chain-of-Taan Table:: Evolving Tables in the Reasoning Chain for Table Understanding. In Proceedings of the 12th International Conference on Learning Representations, 2024.
12

---

Published as a conference paper at COLM 2025
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed H. Chi, Quoc Le, and Denny Zhou. Chain of Thought Prompting Elicits Reasoning in Large Language Models. In Proceedings of the 36th International Conference Neural Information Processing Systems, 2022.
Tingyu Xie, Qi Li, Jian Zhang, Yan Zhang, Zuozhu Liu, and Hongwei Wang. Empirical Study of Zero-Shot NER with ChatGPT. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 7935-7956, 2023.
Bohao Yang, Chen Tang, Kun Zhao, Chenghao Xiao, and Chenghua Lin. Effective Distillation of Table-based Reasoning Ability from LLMs. In Proceedings of the 2024 Joint International Conference Computational Linguistics, Language Resources and Evaluation, Pp. 5538-5550, 2024.
Xianjun Yang, Yujie Lu, and Linda Petzold. Few-Shot Document-Level Event Argument Extraction. In Proceedings of the 61st Annual Meeting of the Association for Compulational Linguistics (Volume 1: Long Papers), Pp. 8029-8046, 2023.
Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L. Griffiths, Yuan Cao, and Karthik R Narasimhan. Tree of Thoughts: Deliberate Problem Solving with Large Lan- guage Models. In Proceedings of the 37th International Conference on Neural Information Processing Systems, 2023.
Yunhu Ye, Binyuan Hui, Min Yang, Binhua Li, Fei Huang, and Yongbin Li. Large Language Models are Versatile Decomposers: Decomposing Evidence and Questions for Table-based Reasoning. In Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 174-184, 2023.
Bowen Zhao, Changkai Ji, Yuejie Zhang, Wen He, Yingwen Wang, Qing Wang, Rui Feng, and Xiaobo Zhang. Large Language Models are Complex Table Paarsers. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 14786-14802, 2023.
Yilun Zhao, Yitao Long, Hongjun Liu, Ryo Kamoi, Linyong Nan, Lyuhao Chen, Yixin Liu, Xiangru Tang, Rui Zhang, and Arman Cohan. DocMath-eval: Evaluating math reasoning capabilities of LLMs in understanding long and specialized documents. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volune 1: Long Papers), pp. 16103-16120, 2024.
Denny Zhou, Nathanael Schärli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuurmans, Claire Cui, Olivier Bousquet, Quoc V Le, and Ed H. Chi. Least-to-Most Prompting Enables Complex Reasoning Large Language Models. In Proceedings of the 11th International Conference on Learning Representations, 2023.
Jin Ziqi and Wei Lu. Tab-CoT: Zero-shot Tabular Chain of Thought. In Findings of the Association for Computational Linguistics: ACL 2023, pp. 10259-10277, 2023.
13

---

Published as a conference paper at COLM 2025
Prompt
Task: Given table, your task is to identify the primary key and its position, and then identify the position the attribute names within the table.
If the table does not contain a primary key, generate one and return its position. The table positions should be referenced as a 2D Python array, with indexing starting at [0, 0j. Negative indices, such -1 or -2, may be used for the inserted primary key. Output Template:
Primary Key: |<a list of primary key>]
Primary Key Position: l'column': (<column numbers>]| or ('row': |<row numbers>]|
Attribute Names Position: column or row
Examples:
(example)
Complete:
(table)
Output:
Table Prompt to identify primary key.
Prompt
Your task is to find the relationship between primarykkey and attributes, along with description.
Output Template:
<attribute name>: <relationship with primary key> description
Input and Output example: (examples)
Complete:
attributes: (attributes)
primary_key. (primary_key) key)
relationships:
Table 5: Prompt to generate relations.
A Prompts
Tables 4, 5, 6 and 7 show prompts used in our TUNES framework.
B Cypher code example
Cypher code to retrieve data for query "How many more times does Loose Women air than This Morning?" is shown in Figure 5.
Prompt
Schema:
(schema)
Your task is to extract subgraph containing all necessary nodes to answer to question.
Please return Cypher code only.
Note that:
Value of all properties is string (convert to number needed).
2. If the question is to the order of the value in the table, please answer based on the properties column column_address (int) and row ww__adrres (int) of the table.
3. If the question requires compare strings please use toLower to compare both. Input and Output example: (examples)
Complete:
Question: (question)
Cypher code:
Table 6: Prompt to generate Cypher query.
14

---

Published as a conference paper at COLM 2025
Prompt
Context:
[context]
Question:
[question]
Your task is answer the question based on given context. Please provide the answer extracted only, do not include any rewrite or new content.
the question is related to the location of the data in the table, please answer based on the column address row address. Note that -1 mean all the column row.
You can explain the answer in a short context in the next row and show the confidence score.
Table Prompt generate the final answer.
MATCH E:Entity (title: 'Loose Women ()::irrirrirnn---(> Year) WITH collect vylauee
yearsLooseWomen MATCH (:Entity (title: 'This Morning ">-(:airr_in->>(yyWe co
WITH yearsLooseeWoenn collect (y. value) vyllee AS yearsThisMorning
WITH size (yearsLoosewomen) as countLooseWomen, size (yearsThhisMrniing) size (yearsLoosewomen))
countThisMorningg
(ixesssssshi))))))nnngg as difference
CREATE (lw: Attribute (countLooseeWomen: countLoosewomen!)
CREATE (tm: Attribute (countThisMorning: countThismorning))
CREATE (diff: Attribute (difference: difference))
CREATE (result: Entity [query: "How many times does Loose Women air
more than This Morning?"])
CREATE (result)--::rllattddtoo-----
CREATE (reautt)-C:relaatedtto----
CREATE (result) -:: hna-:=hhaaa_nswrr=>>
RETURN result, 1w, tm, diff ffff
Figure 5: Cypher code example to retrieve data for query "How many more times does Loose Women air than This Morning?"
C Dataset statistics
Table presents the statistics the WikiTableQuestions and TabFact test sets.
Statistics
WikiTQ
TabFact
Number of Questions
4343
2024
Number of Tables
421
298
Min/Max Max Number of Rows
6/518
5/49
Min/ Max Number Columns
5/20
3/21
Table 8: Statistics of the WikiTableQuestionss (WikiTQ) and TabFact test sets.
D Example of errors
D.1 Entity identification error
D.1.1 Table structure identification error
Figure 6 illustrates table structure identification error.
15

---

Published as a conference paper at COLM 2025
Encrypted
Encrypted
Encrypted
Encrypted
Encrypted
0
57
19
108
38
113
57
91
76
79
1
109
20
125
39
116
58
37
77
65
60
21
3
48
22
89
41
102
80
51
79
67
16
128
35
93
54
90
73
50
92
115
17
120
38
56
93
98
18
68
37
103
56
44
75
104
encryption
Extracted entities:
Primary Key
Attributes Names
Encrypted
Encrypted
Encrypted
Encrypted
Encrypted
109
20
125
39
118
58
37
77
65
Predicted
Golden Answer:
V
primary key, the search engine can only retrieve entiy Result 1' based on primary key, leading insufficient information answer the question.
Description:
represent Consequently,
Note that: representing the entiy in labular form for illstrationn purposes salety.
Figure 6: Illustration of Table structure identification error.
D.2 Search errors
D.2.1 Insufficient entity retrieval error
Figure 7 and Figure 8 illuusrate insufficient entity retrieval errors.
D.2.2 Incorrect self-generated entity error
Figure 9 illustrates an incorrect self-generate entity error.
D.3
Answer generation errors
D.3.1 Comparative error
Figure 10 illustrates a comparative error.
D.3.2 Numerical error
Figure 11 illustrates a numerical error.
D.3.3 Logical error
Figure 12 illustrates a logical error.
16

---

Published as a conference paper at COLM 2025
Chart
Artist
Album
Song
Billboard Hot 100
2014
Puff Daddy
Fothcoming Album
Big Homie
New Single
2014
Ross Ijaa
Folhcoming Album
Devil Lie
Single
2013
Chris Brown
Fine China
31
2013
Ludacris
Forthcoming Album
Raised the South
New Single
2012
Chris Brown
Fortune
Don't Judge
18
2009
Justin Bieber
My World
Love Me
37
2009
Trey Songz
Ready
Invented Sex
42
2010
Clara
Basic Instinct
Ride ft. Ludacris
Question: How many releases not single?
Extracted entities:
Chart year
Artist
Album
Song
Billboard Hot 100
2013
Chris Brown
Fine China
31
2009
Trey Songz
Ready
Invented Sex
42
2010
Ciara
Basic Instinct
Ride ft. Ludacris
45
Predicted Answer:
Golden Answer:
Description:
The search output missing 'Don't Judge and 'Love Me', resulting insufficient data the question accurately.
Note that: representing the entity in tabular form is for illustration purposes solely.
Figure 7: Illustration of Insufficient entity retrieval error.
17

---

Published as a conference paper at COLM 2025
Tie No.
Home team
Score
Away team
Date
Attendance
Alhletic
January
11
Brentford
0 0-3
Burnley
25 January 2003
9,563
12
Manchester United
West Ham United
26 January 2003
67, 181
Norwich City
Dagenham Redbridge
January
14
Crystal Palace
0-0
Liverpool
26 January 2003
26,064
Farbborough Town
Arsenal
25 January 2003
35,108
Stoke City
Boumemouth
January
Question: Fulham and Stoke City both with how many points?
Cypher:
MATCH
WHERE (oLooer(n.hooe_ttemn CONTAINS toLower( (edower('Fulllle FFFlhamm ) OR toLower(n (n. hww.eoomeeeateeeeeee CONTAINS toLower( 'Stoke
City') )
WITH
WHERE n.score CONTAINS '-'
WITH split(n. score, '-') AS score
WHERE toInteger toInteger(score(0]] (sSote))r(score(1)
Error:
Query cannot conclude with MATCH (mnss be RETURN clause, FINISH clause, an update clause, a unit subquery call, or procedure call with no VIELD)
Description: Cypher query does not include RETURN clause, so it falls to produce any result.
Figure Illustration of Insufficient entity retrieval error.
18

---

Published as a conference paper at COLM 2025
Year
Class
1972
$30
Ligier JS2 Maserali 3.0L V6
Automobiles Ligier
1973
3.0
Ligier JS2 Maserati 3.0L V6
Automobiles Ligler
1974
$.3.0
Ligier Maserati 3.0L V6
Automobiles Ligier
1977
S +2.0
Renault Alpine A442 Renault 2.0L Turbo V6
Renaull Sport
1978
$ +2.0
Mirage M9 Renault 2.0L Turbo VB
Grand Touring Cars Inc.
1990
C1
Porsche 962C Porsche Type-935 3.0L Turbo Flat-6
Joest Porsche Racing
1993
GT
Venturi 500LM Renault PRV 3.0 Turbo V6
Jacadi Racing
1994
GT2
Porsche 911 Carrera RSR Porsche 3.8 L Flat-6
Larbre Compétition
1996
GT1
McLaren F1 GTR BMW S70 6.1L
Team Bigazzi SRL
Question: How many consecutive years did team automobiles ligier run the maserati in the 70's?
Cypher query:
MATCH (n)
WHERE to team) CONTAINS toLower( automobiles ligier')
AND toLower (n. CONTAINS toLower( maserati')
AND n.name '1988' AND n. n.name '1980'
WITH
ORDER n. row_adedssss
WITH collect(n) AS years
UNWIND range(1, size(years) 1) previous
WITH years[1] current, years(1
WHERE current. row_addess_dd previous. row_address RETURN current
Extracted entities:
Year
Class
Team
1973
$.0
Ligier JS2 Maserati 3.0L VS
Automobiles Ligier
1974
3.0
Ligler JS2 Maserati 3.0L v6
Automabiles Ligier
Predicted Answer: 2
X
Golden Answer:
The query only adjacent years, allowing identify pairs of consecutive years, the first year in this sequence.
Description:
Note that: representing the entily in tabular form is for illustralion purposes solety.
Figure 9: Illustration of Incorrect self-generated entity error.
19

---

Published as a conference paper at COLM 2025
Year
Name
Label
1982
He's Gonna Take You Home
Becket
1982
Thanks to You
Becket
Need You Now
Jive
1984
Thin Line
Power House
1986
Say It Again
Spring
1987
Send It C.O.D.
New Image
Question: Which singles are newer? "thin line" or "say it again"?
Extracted entities:
Year
Name
Label
1984
Thin Line
Power House
Say Again
Spring
Predicted Answer: Thin Line
Thin Line was released in 1984, while Say Again was released in 1986. Therefore, Thin Line is newer than Say it Again because 1984 1986.
Golden Answer: Say Again
The LLM incorrectly claims that "Thin Line" is newer than "Say Again," Since 1984 1986, "Thin Line" is actually older, not newer.
Description:
Note that: representing the entity in tabular form for illustration purposes solely.
Figure 10: Illustration of Comparative error.
20

---

Published as a conference paper at COLM 2025
Date
Opponents
Venue
Result
Scorers
Attendance
Aug
Reading
Sep 1920
Bristal Rovers
A
2-3
Walker, Wolstenhoime
10,000
Sep 1920
Reading
A
0-4
10,000
Sep 1920
Plymouth Argyle
A
1-5
Wolstenholme
12,000
Sep 1920
Plymouth Argyle
H
0-0
8,000
Question: what was the total attendance of the first five games season?
Extracted entities:
Date
Opponents
Venue
Result
Scorers
Attendance
28 Aug 1920
Reading
H
0-1
14,500
Sep 1920
Brietol Rovers
A
2-3
Walker, Wolstenhoime
10,000
Sep 1920
Reading
A
0-4
10,000
Sep 1920
Bristol Rovers
0-2
8,000
11 Sep 1920
Plymouth Argyle
A
1-5
Wolstenholme
12,000
Predicted Answer:
The total attendance of the first five games (14,500 10,000 + 10,000 8,000 12,000) calculated adding the attendance figures each game.
Golden Answer: 54500
V
Description: The LLM incorreclly predicts the lotal atlendance as 500,, despite the correct calculation showing that the sum first live games' allendances (14,500 10,000 10,000 8,000 12,000) equals 54,500, not 60,500. representing entiyy
purpases solely.
Figure 11: Illustration of Numerical error.
21

---

Published as a conference paper at COLM 2025
Chord
Root
Minor third
Perfect fifth
Major seventh
CmM7
E b
G
C#mM7
Cf
E
G#
B (C)
D mM7
D
F b (E)
A
EmM7
G
B
AmM7
Question: the chords minor major seventh and a minor major seventh have which note in common?
Extracted Entities:
Chord
Root
Minor third
Perfect fifth
Major seventh
EmM7
E
G
B
D#
AmM7
A
C
E
Gz
Predicted Answer:
X
Golden Answer:
V
Description: The LLM incorrectly predicts the root of the chord as though correct root for the EmM7 chord is provided as E.
Note that: representing the entity in tabular form is for illustration purposes solely.
Figure 12: Illustration of Logical error.
22


Reviews:
[{"review_id": "cBaEurmZgr", "reviewer": "Reviewer_G4kS", "summary": "This paper introduces an entity-oriented search method to solve the table-base QA with LLMs. Unlike existing approaches that rely heavily on preprocessing and keyword matching, this method leverages semantic similarities and implicit relationships between table cells. The authors pioneer the use of a graph query language for table understanding and demonstrate state-of-the-art performance on WikiTableQuestions and TabFact benchmarks.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This paper presents a novel entity-oriented approach that advances table understanding by moving beyond preprocessing and keyword matching. It introduces a graph query language for table analysis—establishing a new research direction in the field. The method effectively captures semantic similarities and implicit cell relationships, demonstrating superior performance on standard benchmarks WikiTableQuestions and TabFact with state-of-the-art results.", "reasons_to_reject": "1. While the paper claims their method's superiority by reducing computationally intensive chain-of-thought reasoning, it lacks experimental evidence demonstrating this computational efficiency difference.\n\n2. The paper overlooks relevant prior work on retrieval-based methods for table understanding, specifically Chen et al.'s \"TableRAG: Million-token table understanding with language models\" (NeurIPS 2024), which should have been included in the baselines or related work section for proper contextualization.", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zc4OZFbn85", "reviewer": "Reviewer_eigU", "summary": "This paper presents TUNES, a novel approach for table understanding that addresses the limitations of existing methods, such as heavy reliance on preprocessing and lack of contextual information. TUNES leverages semantic similarities between questions and table data, as well as implicit relationships among table cells, to enhance contextual clarity and reduce preprocessing needs. It also pioneers the use of a graph query language (Cypher) for improved reasoning over tables. Experimental results demonstrate that TUNES achieves state-of-the-art performance on standard benchmarks, and the authors plan to extend its application to more complex downstream tasks.", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper presents an effective and well-designed pipeline for table understanding using LLMs, with experimental results showing that the proposed method consistently outperforms strong baseline approaches.\n2. The paper is generally clear and well-written, making complex ideas accessible and easy to follow.", "reasons_to_reject": "1. The motivation is unclear. For example, the claim regarding the \"unpredictable nature of the content and formatting within table cells\" is vague and lacks specificity. \n2. The proposed method appears to be more of an engineering solution rather than a technical innovation. It involves three retrieval strategies and numerous hyperparameters, but it is unclear whether the choice of hyperparameters would significantly impact the results in practical use.\n3. The experiments are only conducted on two public datasets.", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BaJS2QNPs7", "reviewer": "Reviewer_uSxF", "summary": "The paper proposed an entity-oriented search method to improve table understanding with LLMs. This approach focuses on table entities and pioneers the use of a graph query language for table understanding. Experiments show that the approach achieved a new state-of-the-art performance on standard benchmarks WikiTableQuestions and TabFact.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The proposed approach focuses on table entities and pioneers the use of a graph query language for table understanding, establishing a new research direction. \nExperiments show that the approach achieved a new state-of-the-art performance on two standard benchmarks.", "reasons_to_reject": "1. It is better to show the evaluation of the entity identification module. While there is an error analysis that shows the error rate is 4%, we cannot rely on it, because the total number of instances for the error analysis was not mentioned at all. If the error analysis tries to indicate the performance, the authors should first show the statistics of the dataset for the analysis.\n2. From the description in Sec. 3.2, it is not clear how the results from the full-text and semantic search and those from the graph search are combined together to be used in later modules.\n3. It is better to clearly describe how the authors fixed the value for various hyperparameters in Sec. 4.\n4. Without CoT, the performance differences of the proposed approach from the previous SOTA model are rather small.\nWhile the authors criticized CoT as computationally intensive and resource-demanding, they combined their approach with CoT, finally yielding better performances. However, it is not clear whether the differences are really significant without any significance test.", "questions_to_authors": "Could you show another ablation when both full-text and semantic search are removed?", "extra_scores": {}}, {"review_id": "qxYuyHA8zf", "reviewer": "Reviewer_k1Zo", "summary": "This paper explores LLM-based table understanding and proposes an entity-oriented approach. The method leverages LLMs to analyze tables and construct a graph, which is subsequently used for entity matching through a graph query language. This process aims to simplify the table understanding process. Experimental results on the WikiTableQuestions and TabFact datasets show that the proposed method outperforms previous approaches.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* The paper is clearly written and well-structured, making it easy to follow.\n* Using a graph-query language to facilitate table understanding is a new idea and seems reasonable.\n* The ablation study effectively highlights the importance of the individual components in the proposed method.", "reasons_to_reject": "* Limited performance improvement: The results in Table 1 show only marginal improvements over baseline methods. For example, TUNES-GPT3.5 demonstrates only a 0.2 point improvement over TabSQLify on WikiTQ and a 0.5 point improvement over DP&PYAGENT.\n* Lacking justification for using graph representation: The paper’s argument for using a graph-based representation and graph query language for table understanding is not fully convincing to me. Since the performance difference is not very large, why do the proposed graph-based methods offer significant advantages over SQL-based methods or more complex symbolic languages, such as the one employed in Binder?", "questions_to_authors": "* It would be better if the LLM query counts (lines 263 to 266) were included in Table 1, which would better illustrate the inference cost reduction.\n* What's the error type distribution before applying TUNES? Now Section 5.3 mainly discusses the error type for TUNES, but there's no clear evidence on how it improves over original LLM.\n* I'm curious about how recent long thinking models like DeepSeek-R1 or GPT-O4 perform compared with the proposed methods.", "extra_scores": {}}]

Author rebuttal:

Meta-review: The authors propose TUNES, an entity-oriented search method for table-based QA which uses the Cypher graph query language for table understanding. They leverage semantic similarities and implicit relationships between table cells, achieving SOTA performance on two benchmarks while requiring fewer LLM calls than existing methods.

Reviewers had initially raised concerns about (i) lack of computational efficiency evidence and missing TableRAG baseline comparisons (ii) insufficient evaluation of entity identification components and unclear result combination strategies (iii) limited performance improvements and insufficient statistical significance testing — but the authors satisfactorily addressed these concerns through the rebuttal period responses including computational efficiency quantification, TableRAG comparison, statistical significance testing, and additional experiments with DeepSeek-R1-Distill demonstrating consistent improvements.

Overall, the paper makes good contributions with their graph-based approach and demonstrate meaningful empirical progress with computational efficiency gains (albeit with modest performance improvements in some configurations)
Decision: Accept