Compositional Condition Question Answering in Tabular Understanding
Authors: Jun-Peng Jiang
Tao Zhou
De-Chuan Zhan
Han-Jia Ye
Venue: ICML 2025 poster (2025)
Keywords: Multimodal Large Language Models; Tabular Understanding; Tabular Question Answering
Primary area: deep_learning->generative_models_and_autoencoders

Abstract: Multimodal Large Language Models (MLLMs) for tabular understanding have made significant progress in tasks such as financial report analysis and public data tests. However, our comprehensive analysis shows that these models are still limited in certain simple scenarios, particularly when handling compositional conditions in QA. Further investigation reveals that the poor performance can be attributed to two main challenges: the visual encoder's inability to accurately recognize the content of a row, and the model's tendency to overlook conditions in the question.
To address these, we introduce a new Compositional Condition Tabular Understanding method, called {\sc CoCoTab}. Specifically, to capture the structural relationships within tables, we enhance the visual encoder with additional row and column patches. Moreover, we introduce the conditional tokens between the visual patches and query embeddings, ensuring the model focuses on relevant parts of the table according to the conditions specified in the query.
Additionally, we also introduce the Massive Multimodal Tabular Understanding (MMTU) benchmark, which comprehensively assesses the full capabilities of MLLMs in tabular understanding. Our proposed method achieves state-of-the-art performance on both existing tabular understanding benchmarks and MMTU.
Our code can be available at \url{https://github.com/LAMDA-Tabular/MMTU}.

Compositional Condition Question Answering in Tabular Understanding
Jun-Peng Jiang 12
Tao Zhou 12 De-Chuan Zhan Han-Jia Ye
Abstract
ous types of tables, such images or screenshots. It is critical task in numerous fields, including financial anal- ysis (Zavitsanos et al., 2024), experimental data interpre- tation (Konopka et al., 2023), and public service record management (Engin & Treleaven, 2019).
Multimodal Large Language Models (MLLMs) for tabular understanding have made significant progress in tasks such financial report anal- ysis and public data tests. However, our com- prehensive analysis shows that these models are still limited in certain simple scenarios, particu- larly when handling compositional conditions in QA. Further investigation reveals that the poor performance can be attributed two main chal- lenges: the visual encoder's 's inability to accurately recognize the content of row, and the model's s tendency to overlook conditions in the question. To address these, we introduce a new Composi- tional Condition Tabular Understanding method, called COCOTAB. Specifically, to capture the structural relationships within tables, we enhance the visual encoder with additional row and col- umn patches. Moreover, introduce the con- ditional tokens between the visual patches and query embeddings, ensuring the model focuses on relevant parts of the table according to the conditions specified in the query. Additionally, we also introduce the Massive Multimodal Tab- ular Understanding (MMTU) benchmark, which comprehensively assesses the full capabilities of MLLMs in tabular understanding. Our proposed method achieves -fee-oof--hhrr-t- performance on both existing tabular understanding benchmarks and MMTU, Our code can be available at ht tps: //gittt/ue cww/LAMDA Tabular" /MMTU.
Recent advancements Multimodal Large Language Mod- els (MLLMs) (Zhu et al., 2023; Achiam al., 2023; Liu et al., 2024b;a; Yin et al., 2023) have significantly pro- gressed the field of tabular understanding. These models, which integrate pre-trained large language models (LLMs) with vision encoders through projections, provide an end-to- end framework. While conventional tabular understanding tasks, including table detection and structure recognition, have been successfully addressed by previous studies (Xu et al., 2020; Hu et al., 2021a; la; Huang al., 2022; Tang et al., 2023; Wei et al., 2024), our work focuses on the more chal- lenging task of table question answering (TQA), particularly in scenarios where tabular data is presented in image format.
For simple TQA tasks, MLLMs may only need to recog- nize and interpret the table's structure to provide accurate answers. For instance, in a grade report, MLLMs can easily extract specific student's score for course. However, such tasks do not engage with the table's 's inherent struc- ture, where rows represent samples and columns represent attributes. The complexity of rows and columns often leads to compositional queries (Oh et al., 2011; Lei et al., 2018; 8; Talmor et al., 2021; Dang et al., 2024), like identifying the mathematics score of the student with the highest overall total. Such tasks demand a more profound understanding of the table's 's content and structure, significantly increasing the complexity and requiring enhanced interpretative capabili- ties from MLLMs. This raises the question: When provided with a table image, can MI.LMs understand the structural information and handle such complex TQA tasks?
1. Introduction
Tabular understanding (Deng et al., 2022; Jin et al., 2022; Shigarov, 2023; Wan et al., 2024) aims to automatically extract, analyze, and comprehend information from vari-
To answer the question, we conducted a preliminary investi- gation. Based on the characteristics of tabular data, cate- gorize the existing benchmark into four aspects: understand- ing individual elements (IE), interpreting rows and columns (RC), comprehending compositional conditions (CC), and performing basic calculations or reasoning (CR). Our analy- sis, shown in Figure 1, reveals that current MLLMs perform well on IE and RC tasks, which mainly require structural recognition. However, they face significant limitations on
'School of Artificial Intelligence, Nanjing University, Nanjing, China. "National Key Laboratory for Novel Software Technology, Nanjing University, Nanjing, China. Correspondence Han-Jia Ye <<cch@@ @ andd..nu..dducn>..
Proceedings of the 4nnd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 20255 by the author(s).

---

Compositional Condition Question Answering in Tabular Understanding
IE
cooditoon
condition
masing conditions
Incommedly abgmment
0.38
An example
Question: Waich village has a segattive growth rate that is the most close to U2
Villege
Congraalie
Rood
Ground Truth Casselran with a growth rite of -22%%
4ooe
171
140
RC
0.13
0.
CC
Leeds and
ToxMonkkyy [Bark's South River (5 (235,654).
Docthil2 each
Burks Falls: 3.4%, Casselmon:
2.2% ... Merrickville Lower dier. 7,6%, .... Westport: 6.1%. Thirdomm the Village w negeitiva growth rate
cloces 0 Westppptt will a rate (l.86.118)
10,45
14200
CR
Quea-V V LaMa Stepl: Identify the vallages with a negative growth rate. (Point Iddward: -8.0% Westpocts 66.1% Oil Syrings: -8.0%
Stepd: The village with the degative growth rsto closest to Westport, juilss grow male oc.-6.1%.
QueenYLL-Maa Max
GPT-4
GPT-tu: The village regalive gowlh esle closed is Oil Springs (hill growi rile (0.1%)
TextMonkey
DecOwi1
Figure 1. An overview of the capabilities of some current MLLMs subsample of WiliTableQuestionss (Pasupat Liang, 2015) dataset in the four aspects of tabular understanding, including understanding individual elements (IF), rows or columns (RC), compositional conditions(CC), and calculations or reasoning (CR). An example on the CC is also provided. Almost all models perform poorly, with the two main types of errors being: incorrectly aligned with the row or column, and overlooking a condition specified in the question.
CC and CR tasks, which require deeper engagement with the information in rows and columns. MLLM performance varies notably, even on simple IE tasks. Furthermore. all models, including GPT-4o (OpenAI, 2024), perform poorly on tasks involving basic compositional conditions.
understanding of contextual dependencies. Moreover, we introduce the conditional tokens between the visual patches and query embeddings. This ensures that the model can focus on the relevant parts of the table based on the con- ditions specified in the question, allowing it effectively reason across different rows, columms, conditions. Exper- imental results demonstrate the effectiveness of CoCOTAB in improving the model's 's ability in complex TQA tasks, particularly those involving compositional conditions.
To understand what exactly causes current MLLMs to per- form poorly in tabular understanding, especially in compo- sitional conditions. we investigate several aspects according to the three components of MLLMs: vision encoder, projec- tion, and LLM. Our analyses suggest that the primary reason for the observed poor performance is the vision encoder and the projection. Specifically, the patch division strategy of the vision encoder causes inability to accurately recog- nize the content of a row, leading to positional errors in the table's structure. Moreover, when handling complex QA, MLLMS tend to overlook all the relevant conditions in the question, leading to incomplete or incorrect answers.
To better assess MLLMS ability in tabular understanding, we identify gaps in current benchmarks, such as a lack of focus on specific question types and the presence of factual errors. We propose evaluating four key aspects: element, row/column, compositional condition understanding, and basic calculations/rcasoning. Introducing the Multimodal Tabular Understanding (MMTU) benchmark, with 8921 QA pairs across 4 categories and over 10 domains, we aim to overcome existing dataset limitations. The key contributions of this paper are as follows:
To address this, we propose Compositional Condition Tab- ular Understanding (CoCoTAB), a new MLLM aimed at improving tabular understanding. In CoCOTAB, the visual encoder is enhanced to better capture the full structure of tables, processing row and column information without the distortions from dividing the table image into patches. We also improve the alignment between the table's structure and the query conditions, ensuring key information is at-
Evaluating MLLMs weaknesses in tabular under- standing: Our evaluation with multiple MILMs on benchmarks demonstrates the significant disparities and limitations of MLLMS in tabular understanding.
Analyzing causes of limited understanding perfor- mance: Through testing various hypotheses, we find that the primary reasons for MLLMs' poor performance in tab- ular understanding are misalignment in vision encoders and overlook of conditions in questions.
tended. Our approach offers lightweight enhancement to the original model without adding complexity.
Specifically, COCOTAB works by augmenting the visual encoder with row and column patches, which allows the model to better capture the structural relationships within tables. These patches provide contextual information in specific rows or columns, helping the model focus on the specific sample or attribute of the table, and improving its
Enhancing MLLMS with structural information: By incorporating row and column patches and attention mech- anisms, we improved the performance of MLLMs in tab- ular understanding. The new benchmark also provides a more comprehensive evaluation.
2

---

Compositional Condition Question Answering in Tabular Understanding
2. Related Work
2023; Wan et al., 2024; Zhao et al., 2024), have made sig- nificant strides in recognizing the structure and content of tables. However, we focus on more challenging TQA tasks.
2.1. Mutlimodal Large Language Models
The field of multimodal large language models (MLLMs) has seen remarkable progress, especially in the integration of visual and textual processing. Modern MLLMs typi- cally combine visual encoders (Radford al., 2021; Sun et al., 2023; Zhai et al., 2023; Han et al., 2022), large lan- guage models (LLMs) (Brown et al., 2020; Ouyang et al., 2022; Chang et al., 2024; Tianhao et al., 2025; Lian et al., 2025), and various fusion modules (Li et al., 2024; Sun et al., 2025b;a; Zhang et al., 2024; 2025). Recent devel- opments, such as Flamingo (Alayrac et al., 2022), have advanced visual representation by utilizing the Perceiver Re- sampler alongside vision encoders. Models like BLIP-2 (Li et al., 2023) and InstructBIIIP (Dai et al., 2023) employ Q-Former to bridge the gap between vision encoders and frozen LLMs. MiniGPT-4 (Zhu et al., 2023) introduces a combination of Q-Former and a linear projector to align vision and LLM modules more effectively.
Previous methods have tried a lot but the performance is not ideal. Donut (Kim et al., 2021) proposes a new task and a synthetic document image generator to pre-train the model to mitigate the dependencies on large-scale real document images Monkey and TextMonkey (Li et al., 2024; Liu et al., 2024c) adopt shifted window attention and use similarity to filter out significant tokens without redundancy. They expand the model's capabilities to encompass text spotting and grounding, and incorporate positional information into responses to enhance interpretability. mPLUG- GoGGooOl (Ye et al., 2023b) adjusts mPI.UGOWl for OCR-free document understanding. Tabpedia (Zhao et al., 2024) constructs a low-resolution vision encoder and high-resolution vision encoder, with concept synergy mechanism for visual ta- ble understanding. Deng et al. (2024) focuses on exploring various table representations and prompts LLMs directly, In addition to these advantages, they often suffer from the limitations of the ViT architecture. The patch- based divi- sion of the image into blocks causes the model to ignore the inherent relationships in the table, and it also tends to overlook certain conditions in the question.
In contrast, LLaVA (Liu et al., 2024b) utilizes straightfor- ward MLP projector to enhance the alignment between the vision encoder and LLM. mPLUG-OW (Ye et al., 2023a) takes different approach by fine-tuning the vision encoder first and then using LoRA (Hu et al., 2021b) to adjust the LLM. The Qwen-Vi VI. model (Bai et al., 2023) increases the visual module's resolution to 448 in order to improve visual processing performance. Alongside these open-source ad- vances, proprietay models such GPT-4V/4o (OpenAI, 2024; Achiam et al., 2023), Gemini (Team et al., 2023) and Qwen-VL-Plus/MAX (Bai et al., 2023) have demonstrated outstanding pusstanding erffrmmanee performance in both benchmarks and real-world applications. Given the simplicity and effectiveness of the LLaVA architecture, we adopt a similar framework for our model design in this work.
3. Analyze the Ability of MLLMs in Tabular Understanding
We first introduce some notations for MLLMs in tabular understanding tasks. Then, we analyze the performance of current MLLMs on existing benchmarks and examine the reasons behind their performance limitations.
3.1. MILMs for Tabular Understanding
A notable contribution in the field of MLLMs is LLaVA (Liu et al., 2024a), which presents a straightforward yet efficient approach to align the vision encoder with a pre-trained L.M., Specifically,, given an input image X., LLaVA uses the pre- trained CLIP vision encoder ViT-L/14 g (Radford al., 2021) to extract visual features Z, = gX.). The LLM, which is denoted as f-) parameterized by Φ, is then em- ployed to generate textual embeddings H, from instruction X₃- To ensure alignment between the vision encoder and the LLM, LLaVA learns a projector, represented by a multi- layer perceptron (MLP) denoted as W, which transforms visual features Zu, into language embedding tokens H., This enables the seamless integration of multimodal information within the LLM framework.
2.2. Tabular Understanding
Traditional tabular data learning primarily addresses stan- dard tasks such as classification and regression (Ye al., 2024; Meng et al., 2024; Ye et al., 2025b; Jiang et al., 2025). In recent years, increasing attention has been devoted to tab- ular data learning in open- upen-world settings (Jiang et al., 2024; Cai & Ye, 2025; Hollmann et al., 2025; Liu & Ye, 2025; Ye et al., 2025a). Among these, tabular understanding involves comprehending the information contained within tabular data and can be broken down into several tasks, such as Table Structure Recognition (TSR) (Schreiber et al., 2017; Salaheldin Kasem et al., 2024), Table Detection (TD) (Gi- lani et al., 2017; Li et al., 2020), and Table Question An- swering (TQA) (Chen et al., 2020; Talmor et al., 2021; Jin et al., 2022). Traditional methods, whether OaR-d (Ap- palaraju et al., 2021; Da et al., 2023; Gu et al., 2022) or OCR-free (Nassar al., 2022; Kim et al., 2021: Feng et al.,
H = W. Zn, with Z. =9: 9X.))
(I)
Then, by inputting H. and H. into the language model (.), the model generates the response X, = (H,, Hg).
3

---

Compositional Condition Question Answering in Tabular Understanding
In the tabular understanding task, particularly the TQA task we focus on, X,, represents the input table image, and X3 represents the question. Our goal is for the final answer, X.. to be as consistent as possible with the correct answer ..
Table I. Accuracy on four types of table understanding tasks. The results show significant gap between open-source and closed- source MLLMs. Additionally, the performance on the composi-
tional condition QA is generally poor across the board.
IE
RC
CC
CR
N min Σ HL).
LLaVA-7B
0.37
0.28
0.03
0.05
LLaVA-13B
0.47
0.32
0.07
0.10
l' is the loss function that measures the discrepancy between prediction and ground-truth.
Monkey
0.52
0.37
0.17
0.22
TextMonkey
0.52
0.40
0.25
0.18
mPlug-Ow.
0.11
0.08
0.14
0.05
In our paper, based on the rows and columns of tabular data, we categorize TQA into four key aspects: understand- ing individual elements (IE), interpreting rows or columns (RC), comprehending compositional conditions (CC), and performing basic calculations or reasoning (CR). A more detailed introduction is as follows:
DocOwl
0.55
0.50
0.20
0.22
BLIP
0.05
0.07
0.07
0.25
Donut
0.27
0.03
0.02
0.07
GPT-4o
0.88
0.80
0.47
0.77
Owen-VL-Max
0.85
0.82
0.48
0.70
IE: This refers to the task of understanding and extracting specific cell values within a table, such as identifying the value at particular row and column intersection. For example, "What is Student A's math score?"
nience in statistics, we randomly sampled 60 questions from each category and evaluated them this smaller dataset.
Since this is an open-ended question- answering process, directly assessing the correctness of the model's answers is challenging. Additionally, relying solely on GPT-4 to evaluate the correctness of the QA pairs is not ideal, as GPT-4 is not always accurate. For instance, in Figure 1, GPT-4 overlooked the condition of "negative". However, when input the question, ground truth, and the generated answer into LLMss, such as Qwen2.5-72B- Instruct (Yang et al., 2024), the evaluation becomes more accurate compared to using MLLMs alone. This strategy en- sures a more precise assessment of MLLMs by transforming multimodal questions into a single modality for evaluation, thus reducing the likelihood of misjudging the model's ca- pabilities and resulting in a more accurate evaluation.
Evaluation Protocol.
RC: This involves comprehending specific samples or attributes within table. For instance, "Which course does Student A have the highest score in?" or "Which student has the best math score?"
CC: This pertains to understanding table content that sat- isties compositional conditions. Examples include, "What is the math score of the student with the highest total score?" or " Among the top three students in total score, how many have an 'A' in physical education?"
CR: This refers to performing basic calculations or log- ical reasoning on specific cell values within a table. For example, "How much higher is the total score of the top student compared to the lowest-scoring student?"
Results. Table I shows the performance of different MLLMs on the tiny tabular understanding dataset. MLLMs perform poorly overall, with a significant performance gap compared to human experts, where all tasks are easy for hu- mans. For instance, nearly all open-source models achieve less than 55% accuracy on the simple IE task, while closed- source models perform better, with GPT-4o reaching 88%, compared to just 55% for the best open-source model. On more challenging CC and CR tasks, the performance gap between open-source and closed-source models becomes even more pronounced. Even within the open-source mod- els, there a significant performance disparity. Document models, such as DocOwl and TextMonkey, outperform the more general large models like LLaVA all tasks.
3.2. Analysis on MILMMs in Tabular Understanding
Models. We selected several widely used state-of-the-ar MLLMs, covering different architectures, training methods, and data. These MILMs include two proprietary ones, GPT- 4o (OpenAI, 2024), and Qwen-VL-Max (Bai et al., 2023), and seven public ones, LLa 1.6 Vicuna7B/13B (Liu et al., 2024a), Monkey (Li et al., 2024), TextMonkey (Liu et al., 2024c), mPlug-Owl (Ye et al., 2023a), DocOwl (Ye et al., 2023b), InstructBI.IP LI-rcctBLIP--iccuaaB (Dai et al., 2023), and Donut (Kim et al., 2021).
Data. We evaluated the aforementioned models on the widely used tabular understanding benchmark: WikiTable- Questions (WTQ) (Pasupat & Liang, 2015). In particular, we divided the test set of the entire WTQ into four categories based on the nature of the tables, including understanding individual elements (IE), interpreting rows or columns (RC), comprehending compositional conditions (CC), and per- forming basic calculations or reasoning (CR). For conve-
Furthermore, we found that even the strongest model, GPT- 4o, performs poorly on CC tasks. The performance of open- source models is even worse, achieving less than half the accuracy of closed-source models. However, despite being compositional condition tasks, the answers to these ques-

---

Compositional Condition Question Answering in Tabular Understanding
leading to incorrect final answers. As shown in the exam- ple above, even though the model initially considered the condition "most close to 0", it failed to factor the condition of "negative" when answering the question, resulting in an incorrect response. The results in Figure 2 show that open- source models like LLaVA and DocOwl frequently overlook conditions. Even GP overlooks certain conditions at a high rate. More analysis can be found in Appendix D.1
rate
4 LLaVA-78
1.0
2 LLaVA-13B
0.8
%. mPlug Owl
DocOwl
0.6
0.4
0.2
In summary, we find that the poor performance of MLLMs in tabular understanding can be attributed to two main as- pects. First, the vision encoder has limited capabilities, failing to recognize the row and column information accu- rately. This is likely due to the visual encoder splitting the same sample or attribute into multiple patches, and relying solely on position embeddings may not capture this infor- mation effectively. Second, there is gap in how MLLMs understand both the question and the image, which causes a disconnection between the two and leads to the neglect of some conditions mentioned in the question.
0.0
OE
MC
Figure 2. further analysis of the compositional condition task. OE represents the OCR error rate, MC refers to the rate of missing conditions in answers, and IA denotes the incorrect alignment rate.
tions can be directly obtained from the table images, without requiring much additional knowledge. For human experts, these questions are also simple and straightforward. This raises the question: why do current large models struggle with even fundamental compositional condition tasks?
4. Compositional Condition Tabular Understanding
3.3. Why are MLLMs Limited in TQA?
To further investigate why MLLMs perform poorly on table understanding tasks, we analyze their structure, focusing on the visual encoder and LLM components. When it comes to tabular understanding, two key aspects arose: whether the table image can be accurately recognized and whether the LLMs can effectively comprehend the image. According to the results above, we primarily analyzed the performance of LLa VA-7B, VLa7B mPlug-Owl.
Motivated by our findings, this section presents our method, which incorporates row and column patches along with text- enhanced conditional tokens. We then outline the training process for our approach.
4.1. Contextual Information Extraction in Table Images
Traditional MILMs, such as LLaVA, Donut, Tabpedia, typically use the ViT architecture as the vision encoder. Specifically, for 2D images, VIT reshape the image X,, R R'xwxC into a sequence of flattened 2D patches X." R- where (11, W) is the resolution of the original image, C is the number of channels, (P, P) is the resolution of each image patch, and N HW/P pa is the resulting number of patches, which also serves as the effective input sequence length for the Transformer.
The adaptation of the visual encoder table images is crucial. We analyzed the capabilities of the vision encoder in MLLMs. We evaluated the instances of text recognition errors in the results of these MLLMs and found that LLaVA exhibited a significant number of text recognition mistakes. Even GPT-40 showed some cases of fuzzy text recogni- tion errors. For example, they often misinterpret "1007" as "1001" or "1009". Additionally, while analyzing these results, also observed errors in misalignment in rows or columns. For instance in Figure 1, the Oil Springs's change rate is -8.0, but recognized as Point Edward's change rate 0. by GPT-4o. When comparing the results of mPlug-Ow and DocOwl, we find that, with fine-tuning on document data, DocOwl significantly reduces rates of missing conditions and incorrect alignments. This highlights the importance of familiarizing the visual encoder with table images.
However, when dealing with table images, using the same patch division can result in the splitting of certain infor- mation column attributes across different patches, dis- rupting the contextual relationships within the table's rows and columns. For example, in the case in Figure 1, the village "Point Edward" and its corresponding change rate are divided into separate patches, severing their relation- ship within the same row. This causes incorrect matching during reading, where the change rate of "Oil Springs" is mistakenly paired with "Point Edward" instead. Therefore, ensuring that the model properly understands the content within the rows and columns becomes crucial.
Fully understanding each condition in the table is essen- tial for successfully handling compositional condition tasks. To further analyze why the model provides incorrect answers, we asked the model to answer step by step in a ofain-of-thought format. We found that in conditional con- dition scenarios, the model may overlook certain conditions,
To address this issue, we augment the original patches by
5

---

Compositional Condition Question Answering in Tabular Understanding
Casselman with a growth rate of -.2%%
Large Language Model
Pre-truising
LLM
Projection
Vision Encoder
Projection1
Projection2
Projection3
negathoo
Vision Facoder
Instraction Tining
Cross Attention
LLM
Projection
Word Embeilding
Which village has a negative growth rate that is the most close to 0?
Vision Encoder
Figure 3. The overall architecture of COCOTAB. It primarily augments the table image with row and column patches and incorporates conditional tokens to enhance the relationship between each condition in the question and the image. The training details within each stage are presented on the right.
adding additional row and column patches. The goal is to better preserve the relationships within the sample or attributes through these row and column patches, thereby reducing the problem of misaligned rows in the vision en- coder. Specifically, in addition to the original patches X2 with resolution (P, P). we introduce row patches X. with resolution (P, W) and column patches X with resolu- tion PH As result, the total number of patches is N = HW/P² + H/P W/P. Therefore, the final image tokens can be represented as follows:
computed to capture relationships within the sequence. AI- though the image patch tokens are projected into the same space as text tokens, this method appears inadequate for truly effective cross-modas understanding, leading to the missing of certain conditions in tabular understanding.
Therefore, fully leverage the information from both the image and the question, we attend to the image according to the question. First, we extract visual features using the visual encoder and then project them into language em- bedding tokens H. E NCC, where N, is the vision token length and C' is the embedding dimension. Similarly, we obtain the question embedding H. N. from the text input via a word embedding table. Subsequently, to guide the transformation of visual features towards the question, we employ cross-modal attention to obtain H'
Z. Zeow, Zoo1 = g Yow
Col
(3)
After obtaining the tokens for all image patches, we learn projection from each kind of patch token type text, result- ing in H, (Hę, Hoow H By partitioning the table into row and column patches, we enhance the integration of sample-level and attribute-level information within individ- ual patches, which preserves intra-sample and intra-atrribute relationships, ensuring their structural integrity. Further- more, the use separate projections enables the learning of distinct mappings, effectively mitigating information loss.
H.H. VC
H Attentionn (ociiiinnnQ, Q, V) = -Softmax
H
(4)
where Q equals the matrix Hg. K and V are equivalent to H,, H E NC, This process allows the visual features to be dynamically adjusted based on the input questions.
4.2. Attend Tables with Questions
After obtaining the question-guides image tokens, we feed them into the LLM alongside the text tokens. Specifically, by inputting H., H', and H, into the language model f-)... the model generates the response X, fo (H.., H Hg). This approach explicitly combines the information from both the question and the image, rather than leaving the LLM to learn the relationships between the image and text tokens independently. Doing so more effectively captures the conditions specified in the question.
In current MLLMs. such LLaVA, the visual encoder g built on the Vision Transformer (ViT). When process- ing image data, the model divides the image into patches and employs multiple layers of multi-head self-attention to capture the relationships between these patches. Simi- larly, in the language component, like the Transformer block in LLaMA, the architecture typically combines multi-hea head attention and MLP layers. For text, only self-attention is
6

---

Compositional Condition Question Answering in Tabular Understanding
4.3. Training Stages
Our goal is enhance different aspects of the tabular un- derstanding capabilities of MLLMs. The whole training procedure is divided into two distinct stages.
GPT-
Stage 1: Patch Learning and Modality Alignment. In this stage, we keep most of the parameters in the vision encoder and the LLM weights frozen, focusing on optimiz- ing the projectors to align the visual features H, with the pre-trained LLM word embedding. We unfrozen the row and column embedding capture the representation of the samples and attributes. Row tokens and column tokens are aligned to LLM word embedding with their own projection to capture corresponding information. To enhance the diver- sity of images, we extract a portion of data from LAION and CC12M and construct the caption data, along with some tabular datasets, including Pub1M (Smock al.. 2022), WTQ (Pasupat & Liang, 2015), NAT-QA I, PlotQA (Wang et al., 2024b), OCR-VQA 2 (Mishra et al., 2019).
QUAL Wb/
MMTO
Q::A1
Q2.41
3
06:44
Figure 4. The calibration process for constructing the MMTU. The calibration process is mainly divided into three stages: GPT Gen- eration, MLLMs Answer, and Manual Calibration.
tion answering. (2) Uncertainty of Table Images. Except ComTQA, other benchmarks do not provide table images. Since the method for converting data into table format can vary, this leads to potential biases in the evaluation results. (3) Lack of Systematic Evaluation. All existing bench- marks group similar QA tasks together without systemati- cally evaluating specific capabilities, such understanding individual cells, interpreting specific rows or columns, han- dling compositional conditions, and assessing reasoning and calculation abilities.
Instruction Tuning for Tabular Understand- ing, In the first stage, we learned the representations of the patches and their mapping to the text embeddings. In stage 2, we freeze the vision encoder weights while continuing to train the projection and LLM. Our primary goal in this stage is to refine the representations of rows and columns, facilitating the LLMs' adaptation to these representations. Specifically, we focus on learning the row and column patch tokens and the image tokens that the questions attend.
Stage 2:
To address these challenges, we selected four tasks: un- derstanding individual elements (IE), interpreting rows and columns (RC), comprehending compositional conditions (CC), and performing calculations or reasoning (CR). For the MMTU benchmark, we curated tables from WTQ (Pa- supat & Liang, 2015), TabFact (Chen et al., 2019), and NAT-QA, creating four QA task types across over ten do- mains, yielding 8921 QA pairs. To ensure quality, GPT-4 generated questions, LLMs and human experts validated an- swers, retaining consistent pairs and resolving discrepancies, as shown in Figure 4. More details are in Appendix C.
To tackle the challenge of limited data in TQA, we adopt a semi-automatic approach to acquire image- QA data. We begin by randomly sampling tables from existing datasets, selecting subset, and converting them into images. Next, we use GPT-4 to generate random questions about these ta- bles and provide corresponding answers. After obtaining the initial QA data, we validate it using Qwen-VL-Max, retain- ing only the datasets that pass the validation. Recognizing that this step may introduce noise and potential errors, we apply a manual calibration process further fine-tune and clean the data, thereby obtaining high-quality TQA data.
5. Experiments
In this section, we first outline the experimental framework, providing details on the specifie implementation, evaluation benchmarks, and MLLMs used for comparative assessment. Subsequently, we use tabular understanding benchmarks conduct a comprehensive comparison of CoCOTAB with state-of- taee-offtheeeeee of-the-art methods. Finally, this section summarizes the ablation study and visualizations for the tabular under- standing case, highlighting COCOTAB'S exceptional ability in handling compositional condition tasks.
4.4. MMTU: A Massive Multimodal Tabular Understanding Benchmark
There several existing tabular understanding bench- marks (e.g., WikiTTalleQuestions,, TabFact, FinaQA, and ComTQA) for MILMs, but they have some limitations: (1) Narrow Domain. FinaQA focuses primarily on simple calculations within the financial domain, TabFact assesses the truthfulness of content, and WTQ addresses basic ques-
5.1. Experimental Setup
'nttpe://nugginggaace http:///uugggggaaceccoaoaaaaaaaa staghacco/Acciivttabbb
Implementation Details: In this study, we configure Co- COTAB with the pre-trained Siglip-Vit ViT (Zhai et al., 2023) as the vision encoder and Qwen2- Instruct (Yang et al., 2024)
inttps//huuginggaacecco wwt.:eee..ooaaaaaeeeeeeoe howard-nou/CCR-100
7

---

Compositional Condition Question Answering in Tabular Understanding
Table Accuracy performance comparison on existing benchmarks. MMTU is our proposed benchmark, with four aspects of tabular understanding, including understanding individual elements (IE), rows or columns (RC), compositional conditions(CC), and calculations or reasoning (CR). Our COCOTAB achieves the best performance.
MMTU
WTQ
TabFact
ComTQA
IE
RC
CC
CR
LLaVA-1.6-7BB (Liu et al., 2024a)
0.50
0.32
0.12
0.06
0.23
0.27
0.27
LLaVA-I.6-138 (Liu al., 2024a)
0.59
0.38
0.13
0.08
0.24
0.51
0.29
Monkey (Li et al., 2024)
0.39
0.24
0.28
0.06
0.23
0.51
0.19
TextMonkey (Liu et al., 2024c)
0.62
0.36
0.28
0.06
0.28
0.37
0.25
mPlug-Owl (Ye et al., 2023a)
0.11
0.08
0.15
0.06
0.10
0.50
0.07
Docowl (Ye et al., 2023b)
0.65
0.45
0.26
0.07
0.33
0.61
0.31
shareGPT4V (Chen et al., 2025)
0.17
0.09
0.16
0.04
0.13
0.52
0.10
VisCPM (Hu et al., 2023)
0.04
0.03
0.27
0.04
0.09
0.36
0.04
InstructBLIP (Dai et al., 2023)
0.06
0.04
0.08
0.04
0.09
0.51
0.04
Donut (Kim al., 2021)
0.62
0.14
0.03
0.02
0.10
0.02
0.24
COCOTAB
0.68
0.50
0.43
0.38
0.45
0.74
0.34
as the backbone for LLM. The initial learning rates for the two stages are set as 2e-4 and 2e-6, respectively, with the batch size of 64 and 32. The learning rate for the vision
Dascline
Patches
CT
+0 GoGoTale
encoder is set as 5e-7. The entire training process is about
Arcurary
06-
5 days on the four A800 GPUs setup. Additionally, BF16
and TFF2 precision formats are employed to balance speed and accuracy throughout the training process meticulously. As shown in Figure 3, we set three projections for visual patches, row patches, and column patches respectively.
PC
Evaluation Benchmark: Our evaluation is divided into two parts: one assesses the capabilities of MI.LMs in various aspects of table understanding, while the other evaluates their overall performance. The first evaluation is conducted on our proposed benchmark, MMTU, where we break down table understanding into four distinct components for assess- ment. For the overall performance, the evaluation covers wide broad range of tabular understanding tasks, such as WTQ (Pasupat & Liang, 2015), TabFact (Chen et al., 2019), and ComTQA (Zhao et al., 2024).
Figure 5. Ablation study of components in CoCOTAB MMTU.
perform poorly. Compared to LLaVA-7B, despite the use of a larger language model in LLaVA- 13B, there is no sig- nificant improvement in performance on the CC and CR tasks. This validates our point that the primary limitation of the model's performance lies in the shortcomings of the visual encoder. In contrast, our COCOTAB achieves the best performance across all TQA tasks. Furthermore, it also delivers optimal performance on other existing benchmarks. In the CC and CR tasks, our performance far surpasses that of the original LLaVA, approaching even close to the performance of the closed-source model Qwen2- in Ap- pendix D.2. The experiment demonstrates the effectiveness of our approach to open- open-source MLLMs.
For comprehensive comparisons, we select leading open-source models in MLLMs,, includ- ing LLaVA (Liu al., 2024a), Monkey (Li et al., 2024), TextMonkey (Liu al., 2024c), mPlug-Owl (Ye et al., 2023a), DocOwl (Ye et al., 2023b), ShareGPT-4V (Chen et al., 2025) InstructBLIP--ienne Vicuna7B (Dai et al., 2023), Vis- CPM (Hu et al., 2023), and Donut (Kim et al., 2021).
Comparison Models:
Ablation Study. To validate the importance of each com- ponent in our approach, we evaluated different models on the same dataset. "Baseline" refers to the original LLaVA model in our data, "w/ CT" indicates the addition of con- ditional tokens top of the baseline, and "w/ Patches" represents the incorporation of row and column patches into the baseline. CoCOTAB is our method, which combines both components. As shown in Figure 5, each component of our method contributes to an improvement over the baseline, and our approach achieves optimal performance. This not
5.2. Results
Main Reuslts. In this section, we present table show- casing the table understanding capabilities of most current models. As shown in Table 2, nearly all open-source models perform modestly on table understanding tasks. Especially in compositional condition tasks, all open-source models
8

---

Compositional Condition Question Answering in Tabular Understanding
overall
conference
standing
overall A: The overal rocord for the 2011 2011-12 season 29-7.
season?
aathony grant
neaa first
2008-499
24-10
14-4
round cbi
O: Which season ended with an NCAA Final Four appearance??
2009--10
shaka
27.9
11-7
chrespion
2010--11
shaka smar
28-12
12-6
4th
final four
many seisons ovenall record more
A: Two seasons had an overall record with more than 27 wins: 'n'n- 2010 11 with record of
naua thire
28- 28-12v- 1-12 a record of
shalat
mund
now thins
is
conference wins between 2008-09 2011-12 scasons?
2012-113
shalen smar
27-9
12-4
2nd
A: The difference conference wins between the 2008-09 scasoo (14 wins) and the 2011-12
round
scason (15 wins) is win. This can be found row 2008-09 and 2011-12 and column conference.
Figure 6. A case of COCOTAB. We present an example illustrating four types of questions on a single table image: understanding individual elements (IE), rows columns (RC), compositional conditions (CC), and calculations reasoning (CR), Our method demonstrates strong capability in effectively addressing all these question types.
only validates the efffcctiveness of row and column patches for table data but also confirms that adding attention be- tween the question and the image enhances the model's ability to understand.
information and aligning the model's 's focus with the specific conditions outlined in the questions. We also incorporate an example in Figure 6, showcasing our model's capability to address diverse types of TQA tasks effectively. More details can found Appendix D.5.
6. Conclusion
In this paper, we introduce COCOTAB, new MLLM for TQA that effectively addresses compositional condition tasks. Our approach adapts to the features of table images by incorporating row and column patches in addition to the original patches. To help the model better understand each condition in the question, we use conditional tokens en- hance the interaction between the question and the image. Experimental results demonstrate that our method outper- forms nearly all current open-source models. Through our experiments, we observe that current MLLMs still have limitations in handling compositional condition tasks and calculating or reasoning tasks. We hope this paper sparks further reflection on MILMs, as even for foundational tasks, there is still substantial room for improvement.
10-2
PACA
10
8y4
img
P&CADA
Figure Visulization COCOTAB. can be observed that, after incorporating our row-column tokens and conditional tokens. the model places greater focus on relevant regions, leading to more accurate answer generation.
A Acknovledgements
This work is partially supported by National Key R&D Program of China (202YYFE0202800), NSFC (62376118, 62476123), Key Program of Jiangsu Science Foundation (BK20243012), CCF-Tencent Rhino-Bird Open Research Fund (RAGR20240101), Collaborative Innovation Center of Novel Software Technology and Industrialization.
5.3. Visulization.
To validate the effectiveness of our approach, we visual- ize all tokens by generating attention matrices during the QA process. The results, depicted Figure 7. reveal sev- eral key insights. While attention to image tokens becomes sparser, certain critical tokens continue to receive significant focus. Notably, the row-column tokens and conditional to- kens we introduced emerge as pivotal in generating accurate responses, as they are consistently among the most heavily attended tokens. This visualization underscores the effec- tiveness of our row and column patches in extracting critical
Impact Statement
This paper presents work whose goal is to advance the field of Machine Learning, especially Tabular Understanding. Our MMTU benchmark is designed to provide standard- ized framework for converting structured tables of arbitrary formats into images. At the same time, it categorizes cur-

---

Compositional Condition Question Answering in Tabular Understanding
rent question-answering tasks into four types: understanding individual elements (EE), rows columns (RC), composi- tional conditions (CC), and calculations or reasoning (CR), There are many potential societal consequences in our work related to our model and benchmark, none of which we feel must be specifically highlighted here.
Chen, W., Wang, H., Chen, J., Zhang, Y., Wang, H., Li, S., Zhou, X., and Wang, W. Y. Tabfact: A large-scale dataset for table-based fact verification. arXiv preprint arXiv:1909.00 Tet.rt999000222 02440,1666000 2019.
Chen, W., Chang, M.-W., Schlinger, E., Wang, W., and Cohen, W. W. Open question answering over tables and text. arXiv preprint arXiv:2010. 2020.
References
Da, C., Wang, P., and Yao, C. Multi-granularity prediction with learnable fusion for scene text recognition. arXiv 2023.
Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., al. Gpt-4 technical report. arXiv preprint arX1v:2303.08774, 2023.
preprint arXiv:2307.132:
Dai, W., Li, J., Li, D., Tiong, A., Zhao, J., Wang, W., Li, B., Fung, P., and Hoi, S. Instructblip: Towards general- purpose lisoun-aagguag models with instruction tuning. arxiv 2023. arXiv preprint arXiv:2305.0650 2, 2023.
Alayrac, J.-B., Donahue, J., Luc, P., Miech, A., Barr, I,, Hasson, Y., Lenc, K., Mensch, A., Millican, K., Reynolds, M., et al. Flamingo: visual language model for few- shot learning. Advances in neural information processing systems, 35:23716 22323666-3336,, 2022.
Dang. L. H., Le, T. M., Le, V., Phuong, T. M., and Tran, T. Sadl: An effective in in-context learning method for com- positional visual qa. arXiv preprint arXiv:440.001983 2024.
Appalaraju, S., Jasani, B., Kota, B. U., Xie, Y., and Man- matha, R. Docformer: End-to- transformer for doc- ument understanding. Proceedings of the IEEE/CVF international conference on computer vision, pp. 993- 1003, 2021.
Deng, N., Sun, Z., He, R., Sikka, A., Chen, Y., Ma, L.,, Zhang, Y., and Mihalcea, R. Tables as images? ex- ploring the strengths and limitations of llms on multi- modal representations of tabular data, arxiv preprint 202 2024.
Bai, J., Bai, S., Yang. S., Wang. S., Tan, S., Wang. P., Lin, J., Zhou, C., and Zhou, J. Qwen-vl: A frontier large vision- language model with versatile abilities. arXiv preprint arXiv:2308. /2966, 2023.
Deng, X., Sun, H., Lees, A., Wu, Y., and Yu, C. Turl: Table understanding through representation learning. ACM Special Interest Group on Management of Data Record, 51(1):33 2022
Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., al. Language models are few-shot learners. Advances in neural information processing systems, 33: 1877-1901, 2020.
Engin, 7. and Treleaven, P. Algorithmic government: Au- tomating public services and supporting civil servants in using data science technologies. The Computer Journal, 62(3):448 4600 2019.
Cai, H.-R. and Ye, H.-J Understanding the limits of deep tabular methods with temporal shift. arXiv preprint arXiv:2502.20066 2025.
Feng, H., Wang, Z., Tang, J., Lu, J., Zhou, W., Li, H., and Huang, C. Unidoc: A universal large multimodal model for simultaneous text detection, recognition, spotting and understanding. arXiv preprint arXiv:2308. 11592, 2023.
Chang, Y., Wang, X., Wang, J., WU, Y., Yang, L., Zhu, K., Chen, H., Yi, X., Wang, C., Wang, Y., et al. A survey on evaluation of large language models. ACM transactions on intelligent systems and technology, 15(3):1-45 2024.
Gilani, A., Qasim, S. R., Malik, L., and Shafait, F, Table detection using deep learning. In 2017 14th IAPR interna- tional conference on document analysis and recognition, volume 1, pp. 771-776. IEEE, 2017.
Changpinyo, S., Sharma, P.. Ding. N., and Soricut, Con- ceptual 12m: Pushing web-scale image-text pre-training to recognize long-tail visual concepts. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 3558-35666 2021.
Gu, Z., Meng, C., Wang, K., Lan, J., Wang, W., Gu, M., and Zhang, L. Xylayoutlm: Towards layout-aware multi- modal networks for visually-rich document understand- ing. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 4583-4592, 2022.
Chen, L., Li, J., Dong, X., Zhang, P., He, C., Wang, J., Zhao, F., and Lin, D. Sharegpt4v: Improving large multi-modal models with better captions. In European Conference on Computer Vision, pp. 370-387. Springer, 2025.
Han, K., Wang, Y., Chen, H., Chen, X., Guo, J., Liu, Z., Tang, Y., Xiao, A., Xu, C., Xu, Y., et al. A survey on
10

---

Compositional Condition Question Answering in Tabular Understanding
vision transformer. IEEE transactions on pallern analysis and machine intelligence, 45(1):87-110) 2022.
on Machine Learning (ICML 2000), pp. 1207-1216, Stan- ford, CA, 2000. Morgan Kaufmann.
Herzig, J., Nowak, P. K. Müller, T., Piccinno, F,, and Eisen- schlos, J. M. Tapas: Weakly supervised table parsing via pre-training. arXiv preprint arxivv2004..233444 2020.
Lei, J., Yu, L., Bansal, M., and Berg, T. L. Tvqa: Localized, compositional video question answering. arXiv preprint arXiv:1809.01960 2018.
Hollmann, N., Müller, S., Purucker, L.,, Krishnakumar, A., Körfer, M., Hoo, S. B., Schirmeesterr, R. T., and Hutter, F. Accurate predictions on small data with a tabular foundation model. Nature, 637(8045):319-326, 2025.
Ji, Li, D., Savarese, S., and Hoi, S. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In International conference on machine learning, pp. 19730-19742. PMLR, 2023.
Hu, A., Chen, S., and Jin, Q. Question-controlled text- aware image captioning. In Proceedings of the 29th ACM International Conference on Multimedia, pp. 3097-3105, 2021a.
Li, M., Cui, L., Huang, S., Wei, F., Zhou, M., and Li, Z. Tablebank: Table benchmark for image-bases table detec- tion and recognition. In Proceedings of the Twelfth Lan- guage Resources and Evaluation Conference, pp. 1918- 1925, 2020.
Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., and Chen, W. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2 A.Niv:2006.998 106.09685, 2021b.
Li, Z., Yang, B., Liu, Q., Ma, Z., Zhang, S., Yang, J., Sun, Y., Liu, Y., and Bai, X. Monkey: Image resolution and text label are important things for large multi-modal models. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition, pp. 26763-26773, 2024.
Hu, J., Yao, Y., Wang, C., Wang, S., Pan, Y., Chen, Q., Yu, T., Wu, H., Zhao, Y., Zhang, H., al. Large multilin- gual models pivot zero-shot multimodal learning across languages. arXiv preprint arXiv:2308. 12038, 2023.
Lian, W., Zhang, C., Zhang, H., Jia, B., and Liu, B. Rule- master+: Llm-based automated rule generation frame- work for intrusion detection systems. Chinese Journal of Electronics, 2025.
Huang, Y., Lv, T., Cui, L., Lu, Y., and Wei, F. Layoutimv: Pre-training for document ai with unified text and image masking. In Proceedings of the 30th ACM International Conference Multimedia, pp. 4083 1091409912220 2022.
Liu, H., C., Li, Y. B., Zhang, Y., Shen, S., and Lee, J. Llava-next: Improved reasoning, ocr, and world knowledge, 2024a.
Jiang, J.-P., Ye, H.-J., Wang, L.,, Yang, Y., Jiang, Y., and Zhan, D.-C. Tabular insights, visual impacts: transferring expertise from tables to images. In Forty-first Interna- lional Conference on Machine Learning, 2024.
Liu, H., Li, C., Wu, Q. and Lee, Y. Visual instruction tun- ing. Advances neural information processing systems, 36, 2024b.
Jiang, J.-P., Liu, S.-Y., Cai, H.-R., Zhou, Q., and Ye, H.-J Representation learning for tabular data: A comprehen- sive survey. arXiv preprint arXiv:2504. 200.2999.1619,, 2025.
Liu, Q., Chen, B., Guo, J., Ziyadi, M., Lin, Z., Chen, W., and Lou, J.-G. Tapex: Table pre-training via learning a neural sqi executor. arXiv preprint arXiv:2107.000 107.07653, 2021.
Jin, N., Siebert, J., Li, D., and Chen, Q- A survey table question answering: recent advances. In China Confer- ence on Knowledge Graph and Semantic Computing, pp. 174-186. Springer, 2022.
Liu, S.-Y. and Ye, H.-J Tabpfn unleashed: A scalable and effective solution to tabular classification problems, arXiv preprini arXiv: 2025.
Kim, G., Hong, T., Yim, M., Park, J.. Yim, J., Hwang, W., Yun, S., Han, D., and Park, S. Donut: Document understanding transformer without ocr. arXiv preprint arXiv:2111.15664 7(15):2, 2021.
Liu, Y., Yang, B., Liu, Q., Li, Z., Ma, Z., Zhang, S., and Bai, X. Textmonkey: An ocr-free large multi- modal model for understanding document, arXiv preprint 2024c.
Konopka, K., Turski, M., and Graliński, F. Arxiv tables: Document understanding challenge linking texts and ta- bles. In International Conference on Document Analysis and Recognition, pp. 94-107. Springer, 2023.
Meng, L., Gong, X., and Chen, Y. Bad-fm: Backdoor at- tacks against factorization-machine based neural network for tabular data prediction. Chinese Journal of Electron- ics, 0377 2024.
Langley, P. Crafting papers on machine learning. In Langley, (ed.), Proceedings of the 17th International Conference

---

Compositional Condition Question Answering in Tabular Understanding
Mishra, A., Shekhar, S., Singh, A. K., and Chakraborty, A. Ocr-vqa: Visual question answering by reading text in images. In 2019 international conference on document analysis and recognition, pp. 947-952. IEEE, 2019.
Smock, B., Pesala, R., and Abraham, R. Pubtables- Im: To- wards comprehensive table extraction from unstructured documents. In Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition, pp. 4634-4642 2022.
Nassar, A., Livathinos, N., Lysak, M., and Staar, P. Table- former: Table structure understanding with transformers. In Proceedings of the IEEE/CVF Conference Com- puter Vision and Pattern Recognition, pp. 4614-4623, 2022.
Sun, H.-L., Sun, Z., Peng, H., and Ye, H.-J. Mitigating visual forgetting via take-along visual conditioning for multi-modal long cot reasoning. In ACL, 2025a.
Sun, H.-L.., Zhou, D.-W., Li, Y., Lu, S., Yi, C., Chen, Q.G., Xu, Z., Luo, W., Zhang, K., Zhan, D.-C., et al. Parrot: Multilingual visual instruction tuning. In International conference machine learning, 2025b.
Oh, H.-J,, Sung, K.-Y., Jang, M.-G., and Myaeng, S. H. Compositional question answering: A divide and conquer approach. Information Processing & Management, 47(6): 808-824, 2011.
Sun, Q., Fang, Y., Wu, L., Wang, X., and Cao, Y. Eva- clip: Improved training techniques for clip scale. arXiv preprint arXiv:2303.15389 2023.
OpenAL. Gpt-4o: Hello gpt-4o. Technical report, Ope- nAI, 2024. URL https://openttt http:///ppena...omimneeeeeeee wtt.cccmm/ninnnn Pallo-gpt-4ol
Ouyang. L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730-27744: 2022.
Talmor, A., Yoran, O., Catav, A., Lahav, D., Wang, Y., Asai, A., Ilharco, G., Hajishirzi, H., and Berant, J. Multi- modalqa: Complex question answering over text, tables and images. arXiv preprint arXiv:2104.0660
2021.
Tang, Z., Yang, Z., Wang, G., Fang, Y., Liu, Y., Zhu, C., Zeng, M., Zhang, C., and Bansal, M. Unifying vision, text, and layout for universal document processing, In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 19254-19264, 2023.
Pasupat, and Liang, P. Compositional semantic parsing on semi-structured tables. arXiv preprint 1iXiv:558.00030 2015.
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748-8763, PMLR, 2021.
Team, G., Amil, R., Borgeaud, S., Alayrac, J.-B., Yu, Sori- cut, R., Schalkwyk, J., Dai, A. M.,, Hauth, A., Millican, K., et al. Gemini: family of highly capable multimodal models. arXiv preprint arXiv: 1805, 2023.
Salaheldin Kasem, M., Abdallah, A., Berendeyev, A., Elkady, E., Mahmoud, M., Abdalla, M.. Hamada, M., Vascon, S., Nurseitov, D., and Taj-Eddin, I. Deep learn- ing for table detection and structure recognition: A survey. ACM Computing Surveys, 56(12):1 2024.
Tianhao, F., Zehua, Y., Zhisheng, Y., Chenxiang, M., Yang, H., Yingwei, L., Xiaolin, W., and Zhenlin, W. A survey on the scheduling of di and I1m training jobs in gpu clusters. Chinese Journal of Electronics, 2025.
Wan, J., Song, S., Yu, W., Liu, Y., Cheng, W., Huang, F,, Bai, X., Yao, C., and Yang, Z. Omniparser: A unified framework for text spotting key information extraction and table recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11641-15653, 2024.
Schreiber, S., Agne, S., Wolf, I., Dengel, A., and Ahmed, S. Deepdesrt: Deep learning for detection and structure recognition of tables in document images. In 2017 14th IAPR international conference on document analysis and recognition, volume 1, pp. 1162-1167. IEEE, 2017.
Schuhmann, C., Beaumont, R., Vencu, R., Gordon, C., Wightman, R., Cherti, M., Coombes, T., Katta, A., Mullis, C., Wortsman, M., et al. Laion-5b: An open large-scale dataset for training next generation image-text models. Advances in Neural Information Processing Systems, 35: 25278 25278-25294, 2022.
Wang. P., Bai, S., Tan, S., Wang, S., Fan, 7.,, Bai, J., Chen, K., Liu, X., Wang, J., Ge, W., et al. Qwen2-vl: vl: Enhancing vision- language model's perception of the world at any resolution. arXiv preprint arXiv:2409. /2191, 2024a.
Wang, Z., Xia, M., He, L., Chen, H., Liu, Y., Zhu, R., Liang, K., Wu, X., Liu, H., Malladi, S., et al. Charxiv: Charting gaps in realistic chart understanding in multimodal llms. arXiv preprint arXiv:2406. 1852/, 2024b.
Shigarov, A. Table understanding: Problem overview. Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery, 1382) 2023.
12

---

Compositional Condition Question Answering in Tabular Understanding
Wei, H., Liu, C.,, Chen, J., Wang, J., Kong, L., Xu, Y., Ge,
Zavitsanos, E., Mavroeidis, D., Spyropoulou, E., Fergadi- otis, M., and Paliouras, G. Entrant: A large financial dataset for table understanding. Scientific Data, 11(1):
Z., Zhao, I.., Sun, J., Peng. Y., et al. General ocr theory: Towards ocr-2.0 via unified end-to-end model. 2024.
876, 2024.
Wei, J., Wang, X., Schuurmans, D.., Bosma, M., Xia, F., Chi, E., Le, Q. V., Zhou, D., et al. Chain-of-thougge prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824-248377, 2022.
Zhai, X., Mustafa, B., Kolesnikov, A., and Beyer, L. Sig- moid loss for language image pre-training. In Proceed- ings of the IEEE/CVF International Conference Com- puter Vision, pp. 11975-11988 2023.
Zhang, Y.-.., Lu, S., Li, Y., Ma, Y., Chen, Q., Xu, Z., Luo, W. Zhang, K., Zhan, D.-C., and Ye. H.-J Wings: Learning multimodal Ilms without text-only forgetting. Advances in Neural Information Processing Systems, 37: 3 31828-31853, 2024.
Xu, Y., Li, M., Cui, L., Huang, S., Wei, F., and Zhou, M. Layoutlm: Pre-training of text and layout for document image understanding. In Proceedings of the 26th ACM SIGKDD international conference on knowledge discov- ery & data mining, pp. 1192-1200, 2020.
Zhang, Y.-K., Zhan, D.-C., and Ye, H.-J Capability in- struction tuming. In Proceedings the AAAI Conference on Artificial Intelligence, volume 39, pp. -9956-25966, 2025.
Yang, A., Yang, B., Zhang, B., Hui. B., Zheng, B., Yu, B., Li, C., Liu, D., Huang, F,, Wei, H., al. Qwen2. 5 technical report. arXiv preprint arXiv:2412.15115, 2024.
Yanid, D., Davenport, A., Carmichael, X., and Thompson, N. From computation to adjudication: Evaluating large language model judges on mathematical reasoning and precision calculation. 2024.
Zhao, W., Feng, H., Liu, Q.. Tang, J., Wei, S., Wu, Liao, L., Ye, Y., Liu, H., Zhou, W., et al. Tabpedia: Towards comprehensive visual table understanding with concept synergy. arXiv preprint arXiv:2406. 01 326, 2024.
Ye, H.-J., Liu, S.-Y., Cai, H.-R., Zhou, Q-L.,, and Zhan, D.- C. A closer look at deep learning on tabular data. arXiv preprint 2r0: 00956, 2024.
Zhu,
D., Chen, J., Shen, X., Li. X., and Elhoseiny, M. Minigpt-4: Enhancing vision-language understanding with advanced large language models. arXiv preprint arXiv:2304 10592, 2023.
Ye. H.-J., Liu, S.-Y., and Chao, W.-L. A closer look at tabpfn v2: Strength, limitation, and extension. arXiv preprint arXiv:2502.17361 2025a.
Ye, H.-J., Yin, H.-H., Zhan, D.-C., and Chao, W.-L. Re- visiting nearest neighbor for tabular data: deep tabular baseline two decades later. In International Conference on Learning Representations, 2025b.
Ye, J., Hu, A., Xu, H., Ye, Q. Yan, M., Dan, Y., Zhao, C., Xu, G., Li, C., Tian, J., al. mplug-docowl: Modu- larized multimodal large language model for document understanding. arXiv preprint arXiv:2307.0244 2023a.
Ye, J., Hu, A., Xu, H., Ye, Q. Yan, M., Dan, Y., Zhao, C., Xu, G., Li, C., Tian, J., et al. mplug-docowl: Modu- larized multimodal large language model for document understanding. arXiv preprint arXiv:2307.02499, 2023b.
Yin, S., Fu, C., Zhao, S., Li, K., Sun, X., Xu, T., and Chen, E. A survey on multimodal large language models. arXiv preprint arXiv:2306. 13549, 2023.
Yuan, Z., Yuan, H., Li, C., Dong, G., Lu, K., Tan, C., Zhou, C., and Zhou, J. Scaling relationship on learning math- ematical reasoning with large language models. arXiv preprint Niv:2308.00/800 2023.
13

---

Compositional Condition Question Answering in Tabular Understanding
A. Training Datasets
In this section, we provide comprehensive overview the datasets employed our study. During the first stage, we focus on two primary objectives: (1) effectively learning row and column patches, and (2) establishing preliminary mapping from visual tokens to text embeddings. To achieve robust learning of row and column patches in tables, we trained our model using diverse collection of table-based QA datasets. Additionally, further enhance the training of the projection module. WTQ--enerate means we use the WTQ images and generate QA pairs with the help of GPT-4o. Moreover, the inclusion of conditional tokens during this stage allows the model to more effectively concentrate on visual tokens that are relevant to the posed questions, thereby improving its overall performance.
integrated the LLaVA-pretrain dataset to
In the second stage, our primary objectives are to further refine the projection module and fine-tune the MILM on table-based QA datasets. To achieve this, we leverage range of TQA datasets, including WTQ. NAT-QA, and others. By fixing the visual tokens, enable more effective optimization of the projection module, while unfreezing the LLM allows it to further adapt and learn from the QA datasets. Additionally, to enhance the model's versatility and performance across diverse tasks, we supplement the training with caption-based tasks, such OCR-VQA and ILLVA--fnettue. The datasets utilized in both stages are detailed in Table 3.
Table 3. Details on the CoCoTAB's training data, derived from publicly available datasets and some TQA datasets.
Training Stage
Datasets
Samples
Total
LLa LLaVA--.5-preerral (Liu al., 2024b)
50K
Laion-Caption' (Schuhmann et al., 2022)
50K
CC12M-Caption" (Changpinyo et al., 2021)
50K
PUB-IM (Smock et al., 2022)
500K
WTQ (Pasupat & Liang, 2015)
14K
Stage
1.4M
WTQ-Genrate* (Pasupat & Liang, 2015)
142K
TabFact* (Chen et al., 2019)
105K
NAT-QA"
270K
Plot-QA (Wang et al., 2024b)
100K
OCR-VQA" (Mishra et al., 2019)
168K
I.L.AA-..-finetum (Liu al., 2024b)
100K
PUB-IM (Smock et al., 2022)
82K
wTQ" (Pasupat & Liang, 2015)
14K
Stage
1M
WTQ--Genrate (Pasupat & Liang, 2015)
200K
TabFact* (Chen et al., 2019)
105K
NAT-QA* Plot Plot-QA* (Wang et al., 2024b)
174K
100K
OCR-V CCR-VQA" (Mishra al., 2019)
200K
B. Training Details
As shown Table we provide the training hyperparameters for COCOTAB. Throughout stages of training, we pre-train for one epoch and fine-tune for two epochs, with a batch size of 64 for the first stage and 32 for the second stage. We maintain an image resolution of 384x384 for all two stages and enable the gradient checkpoint mode for each training stage.
C. MMTU: A Massive Multimodal Tabular Understanding Benchmark
In this section, we begin by addressing current benchmarks' shortcomings, and then by outlining the features that an optimal tabular understanding benchmark should have. Additionally, we propose and build a new benchmark along with its associated evaluation strategy.
14

---

Compositional Condition Question Answering in Tabular Understanding
Table 4. Training hyperparamcters.
Config
Stage
Stage
MLP expert network
Linear layers with SiLU
Deepspeed
Zero3
Zero3
Image resolution
384x384
Image encoder
iglip--o4000mppatchhh 14-384
Feature select layer
-2
Image projector
2 Linear layers with GeLU
Epoch
1
2
Optimizer
AdamW
Learning rate
2c-4 5c-7
20-6
Learning rate Vision
Learning rate scheduler
Cosine 0.0
Weight decay
Text max length
8096 16
2048
Batch size per GPU
8
GPU
4 x A800-80G
Precision
6f16
Gradient checkpoint
True
C.1. Limitations of Existing Benchmarks
There several existing tabular understanding benchmarks (e.g., WikiTableeuestionn,, TabFact, FinaQA, and ComTQA) for MLLMs, but they have some limitations: (1) Narrow Domain. FinaQA focuses primarily simple calculations within the financial domain, TabFact assesses the truthfulness of content, and WTQ addresses basic question answering. (2) Uncertainty of Table Images. Except ComTQA, other benchmarks do not provide table images. Since the method for converting data into table format can vary, this leads to potential biases in the evaluation results. (3) Lack of Systematic
Evaluation. All existing benchmarks group similar QA tasks together without systematically evaluating specific capabilities, such as understanding individual cells, interpreting specific rows or columns, handling compositional conditions, and assessing reasoning and caleulation abilities.
C.2. Characteristics of an Effective Tabular Understanding Benchmark
To more suitably evaluate the tabular understanding capabilities of MLLMs, an ideal benchmark should exhibit the following characteristics:
(1) Multiple Domains of Content. An excellent benchmark should cover a wide range of application domains to minimize bias that may arise in specific areas. By including data from multiple domains, we ensure that the model's performance is not influenced by any single domain. This not only enhances the benchmark's generalizability but also allows for a better evaluation of the model's cross-domain adaptability, ensuring that effective models are more widely applicable in real-world scenarios.
(2) Unified Table-to-Image Conversion. To ensure faimess in the benchmark, a standardized rule should be established for converting data in formats such as tables, CSV, HTML, and MD into images. This unified standard eliminates potential differences that could arise from using varying formats, thus preventing discrepancies in the conversion process from affecting the model's 's evaluation. By employing consistent conversion method, all models being evaluated face the same challenges when processing data, thereby ensuring the evaluation results are more reliable and comparable.
(3) Multidimensional Evaluation of Table Understanding. Evaluating table understanding should not be limited to recognizing individual data cells but should encompass more complex tasks, For instance, the model needs to understand not only the content of individual cells but also the relationships between cells, rows, and columns. Additionally, the evaluation should assess the model's is ability to understand compositional conditions, reasoning capabilities, and computational skills. This multidimensional assessment will comprehensively measure the strength of a table understanding model, ensuring that it remains efficient and accurate when handling complex data.
(4) Accurate and Reasonable Evaluation. When designing a benchmark, it is important to avoid including tables with excessive row-column differences that could lead to extreme cases. For example, tables with large row-column disparities
15

---

Compositional Condition Question Answering in Tabular Understanding
QUAL Gbbbb
MMTU
2
QUAL
Q2:A2
Weting
Figure 8. The calibration process for constructing the MMTU benchmark. The calibration process is mainly divided into two stages: GPT-4o Generation, MLLMs Answer, and Manual Calibration.
may have overly complex structures that hinder the model's performance, and should therefore excluded from the benchmark's test scope. Additionally, the data and task design in the benchmark should be as clear and unambiguous as possible to ensure the reliability of the evaluation results. Most importantly, the dataset in the benchmark should not contain any errors, as any inaccurate data could lead to misleading evaluation results, ultimately distorting the true reflection of the model's 's performance.
Fee: Coneentie
SerVis Sorvaatiiee
conference
standing
xmbhumy
nuan fins round
bativtook
2008-009
24-10
1444
gratt
obi
2009-11
11-7
Pull
champion
conde
neaa lina forr
categories
2010-11
8,921 samylees
1:0
2011-12
sooks smat
29-7
153
De Bepprtecs
round
2012-13
shalen smart
27.9
1244
round
Q: Whtt was Score-Hinal! A:395 666
Q:What overall 2011-12 season?
A:29-7
WhWhich apparatus Olympic Games highest Rank-Qualifyinng AcAll-Around
Q:Which
NCAA appearmne??
Games
A:2010-11
Q.How many 27 wins?
record than
Qellow many apparatus had Rank- Qualifying less than at the Would Championshipe?
Society
A:TWO
10+ domains
Q-What diffirrence conference wins 2008-09 and 2-11-12 sessons?
Q-Howw much higher is the Score-Qullifiig of Floor exercise (Semi- final) ompured Uneven burs (Qullfiicatio) the Champsouships?
Flim
Music
TEMINABE
Figure Detailed information and some cases in our MMTU benchmark.
C.3. Construction of the MMTU benchmark
We select four tasks for inclusion: understanding individual elements (ie), interpreting rows or columns (rc), comprehending compositional conditions (cc), and performing basic calculations or reasoning (cr). For the massive multimodal tabular understanding benchmark, denoted as MMTU, we select and clean the suitable tables from WTQ, FinaQA and Arxiv papers. Based on these tables, we constructed four types of aneestion-answerigg tasks according to different understanding objectives, resulting in four question categories, six domains, and approximately 10,000 question-answe pairs.
To reduce noise and errors that may arise during the data construction process, we implemented several strategies to improve translation quality, as shown in Figure 8. First, we used GPT-4 to generate questions for each image based on our predefined question types. Next, we enhanced the data quality through a peer evaluation process involving both large language models (LLMs) and human experts. Specifically, we leveraged three of the most reputable MLLMs in the industry: GPT-4, Qwen2- VL-Max, and Intern VLB to generate corresponding answers. Using consistency criteria, we retained
16

---

Compositional Condition Question Answering in Tabular Understanding
Table 5. Accuracy performance comparison on existing benchmarks.
MMTU
WTQ
TabFact
ComTQA
IE
RC
CC
CR
LLaVA-1.6-7B
0.50
0.32
0.12
0.06
0.23
0.27
0.27
LLaVA-1.6-133
0.59
0.38
0.13
0.08
0.24
0.51
0.29
Monkey
0.39
0.24
0.28
0.06
0.23
0.51
0.19
TextMonkey
0.62
0.36
0.28
0.06
0.28
0.37
0.25
mPlug-Owe Owl
0.11
0.08
0.15
0.06
0.10
0.50
0.07
Docowl
0.65
0.45
0.26
0.07
0.33
0.61
0.31
shareGPT4V
0.17
0.09
0.16
0.04
0.13
0.52
0.10
VisCPM
0.04
0.03
0.27
0.04
0.09
0.36
0.04
InstructBLIP
0.06
0.04
0.08
0.04
0.09
0.51
0.04
Donut
0.62
0.14
0.03
0.02
0.10
0.02
0.24
CoCOTAB
0.68
0.50
0.43
0.38
0.45
0.74
0.34
Qwen2-VL
0.93
0.71
0.38
0.38
0.51
0.72
0.52
GPT-4o
0.95
0.88
0.61
0.85
0.67
0.73
0.62
QA pairs with matching answers. For pairs with inconsistent answers, we consulted multiple human experts to resolve discrepancies, resulting in the final MMTU benchmark.
This approach significantly enhanced the quality of the data, ensuring it is more representative of real-world scenarios and diverse across various question types. We list the detailed information and some cases in Figure 9, As result, our benchmark provides more comprehensive and accurate assessment of tabular understanding. We believe this improvement contributes positively to the overall robustness of our research findings.
D. More Experiments
D.1. More Analysis
In analysis, also identified significant challenges in the computational and reasoning capabilities of MLLMs. However, this paper does not focus on these issues for two main reasons. First, computational and reasoning limitations have long been a persistent challenge for large language models (Wei et al., 2022; Yuan et al., 2023; Yanid et al., 2024), primarily due to inherent architectural deficiencies. Second, our study prioritizes improving aspects of MLLMs that directly impact table comprehension, such as the limitations in visual encoding and the misalignment between visual and textual spaces. As result, we do not directly address computational and reasoning challenges in this work.
D.2. Comparison with GPT-40
Furthermore, we incorporate closed-source methods in our benchmarks, including GPT-4o (OpenAI, 2024), and Qwen2-VL VL- Max (Wang al., 2024a) to show the SOTA remarkable performance.
From the experimental results, significant performance gap between open-source and closed-source models is evident. This discrepancy be attributed to the following primary factors: Model Size: The parameter scale of models such as GPT and Qwen far exceeds that of our model, as their architectures are designed with substantially larger parameter counts. Training Data: Our model was trained on a relatively limited dataset, while these models likely benefited from extensive training on significantly larger and more diverse datasets, enabling them to achieve superior generalization and performance."
D.3. Comparison with Structured Methods
Our method primarily focuses on question answering over visual tables, as in most cases, we do not have direct access to the structured information of the tables. Instead, we feed the table image directly into the model and train a more effective end-to-end model. This approach reduces the information loss between rows and columns that can occur during the intermediate step of converting the image into structured text, and also avoids potential errors introduced during such
17

---

Compositional Condition Question Answering in Tabular Understanding
Table 6. Comparison with structured methods.
IE
RC
CC
CR
TAPEX (Liu et al., 2021)
0.73
0.79
0.33
0.19
TaPas (Herzig al., 2020)
0.59
0.52
0.04
0.19
CoCOTAB
0.68
0.50
0.43
0.38
Table 7. Inference time and memory cost of different methods.
Model
Time(s)
Memory(G)
InstructBlip
0.29
30.98
1.a
0.45
17.21
DocOWL
0,36
19.50
TextMonkey
1.68
23.96
Donut
0.42
0.478
VL2.
1.27
43.28
TabPedia
0.21
19.02
CoCoTab
0.20
20.47
conversions.
Here, we also conducted experiments using structured table parsing- based QA methods based on the data presented in Section 3.2. The experimental results shown Table 6.
As seen from the results, our method achieves comparable performance to existing approaches IE and RC tasks. However, for the more complex CC and CR tasks, COCOTAB significantly outperforms the structured table parsing-based QA methods. This is because once the table is converted into structured text, it becomes challenging for pure language models to preserve spatial relationships between rows and columns, which leads to performance degradation on tasks that require complex reasoning. such as CC and CR. contrast, our method directly processes the table image and, with the help of row and column patches, effectively captures the spatial layout and structure, enabling more accurate question answering. This further validates the effectiveness of our approach.
D.4. Computation Cost
The original method divides the table image into 27x27 patches (729 total). COCOTAB adds 27 row patches (one per row, covering full width) and 27 column patches (one per column, covering full height), resulting in only 54 additional patches (7% increase over the original 729). This modest increase in token count (from 729 to 783) incurs minimal computational overhead during training and inference.
As shown in our ablation study, adding row/column patches alone improves accuracy on CC tasks over the baseline. This demonstrates that the cost increase highly justified by the performance gains, particularly for critical tasks like compositional reasoning.
While paper focused accuracy and structural alignment improvements, conducted additional experiments to quantify the computational overhead of COCOTAB. Key results and analyses are in Table 7.
We have measured the average inference time and memory consumption per sample. As shown in the results, our method does not introduce significant overhead. On the contrary, compared to LLaVA, our approach provides more concise and precise answers, leading to improved inference speed. CoCOTAB adds only 7% in token count but outperforms in all baselines.
D.5. More Visulizations
Visulization. To validate the effectiveness of our approach, we visualize all tokens by generating attention matrices during the QA process. The results, depicted in Figure 10, reveal several key insights. the lower layers of the model, attention
18

---

Compositional Condition Question Answering in Tabular Understanding
Layer
Layer
Layer 28
10
QA|PACA
10-3
P&CADA
Figure 10. Visulization of attention matrix from lower layers to deeper layers.
distributed relatively evenly across different token types. However, in the deeper layers, while attention to image tokens becomes sparser, certain critical tokens continue to receive significant focus. Notably, the row-column tokens and conditional tokens we introduced emerge as pivotal in generating accurate responses, as they are consistently among the most heavily attended tokens. This visualization underscores the effectiveness of our row and column patches in extracting critical information and aligning the model's focus with the specific conditions outlined the questions. We also incorporate several illustrative examples in Figure 11, showcasing our model's capability address diverse types of TQA tasks effectively.
Q: was
A: The Teant's Score-Final was 395 6666
3.00
Q: Which appuratus the Olympic Games the highest Rumk- Ruml-Qualifying? A: All-Around, This can he found Rank-Qualifying column.
Curges
11.00
1179
Q: How many appararus Rank-Qullifyngg of less than 5 the World Championships? A: the World Championships Paris, three apparntss had Rank-Qualiffing less than 3/biin1. Unevan bars (Qunlficction)) muil--uuliftyingg Lunk-Qualifying: 3'a3, Floor exercise (Qualification) Rank- Lank--Qalifying::
V7
Balance heam (Qualification)
Rechin
AMI
Q: How much higher the Sccre-Qualifying Floer exercise (Semi-final) ompared to Uneven bus (Qulificatiio) in World Champicnships? Score-Qualifyyin Floor exercise (Semi-final) not provided.
Her cande
ovcrall
conference
standing
pusleeczeo nean finst round
Q: the overall record 2011-12 season?
anthony
A: The overall recond for the 201 1-12 scason is 29.7.
grazz
Q: Which senson ended with an NCAA Final Four appearance? 2010-11
2009-11
shaka smart
27.9
11-7
5th
chumpice
2010-11
Mola sueet
28-12
12-6
4h
neu
Q: How many scasons had everall recurd with more than 27 wins?
Two
record
2011-12
15:3
2nd
ocia thir
28 12-n 2011 with record of 29.7
round
2012-113
shak smart
27.9
12-4
2nd
round
A: The difference in conference wins between the 2008-09 09 scuson (14 wins) and the 2011-12 wins) This found
difference conference between
2011-12 1-21--12-0 scasonc?
and 2011-11 2011-12
Figure 11. Some cases of COCOTAB.
E. Limitations and Future Works
In this paper, we aim to address the problem of tabular understanding. We propose four types of issues table understanding. namely understanding individual elements (IE), interpreting rows and columns (RC), comprehending compositional
19

---

Compositional Condition Question Answering in Tabular Understanding
conditions (CC), and performing basic calculations or reasoning (CR). In work, primarily focus addressing the CC problem, it involves deeper understanding of the intrinsic row-column structure of tables. While propose four types of problems, we only partially tackle the CR problem. Through analysis, we find that a better understanding of the table's ' intrinsic structure can enhance the CR capabilities of MLLMs. However, the CR ability primarily places higher demands on the reasoning and computational power of L.Mss, rather than on table understanding. Therefore, we were unable to fully address this issue in the current work, and it will be part of our future research.
From another perspective, there is still significant gap capabilitiess between open-source and closed-soures models. As we analyzed earlier, this gap primarily stems from two factors: the difference in model size and data volume. Closed-source models tend to use larger parameter counts and more data, which ultimately contribute their high performance. However, in our work, we have used relatively fewer parameters and less data, yet we have achieved better results with open-source models. In the future, we will further explore the gap between open-source and closed-soure models in table understanding, investigating more fundamental issues, with the aim narrowing this gap and even surpassing closed-source models.
Finally, we hope that this paper will spark deeper reflection on the fundamental capabilities of MILMs, encouraging the exploration of basic yet important issues. We aim contribute more meaningful work the field of MLLMs.
20


Reviews:
[{"review_id": "ERiwhADFY7", "reviewer": "Reviewer_9AZj", "summary": "This paper investigates the table understanding task under compositional condition questions using multimodal large language models (MLLMs). The authors point out that current MLLMs face two major challenges: the inability of the vision encoder to accurately recognize table row contents and the tendency of the model to overlook conditions specified in the question. To address these issues, the paper proposes a new method, COCOTAB, which enhances row/column patch information in the vision encoder and introduces conditional tokens between visual patches and queries to improve the model’s ability to understand tabular data. Additionally, the paper constructs the MMTU benchmark dataset to comprehensively evaluate the capabilities of MLLMs in table understanding tasks.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "The proposed method is reasonable and well-aligned with the problem. The authors augment ViT-based vision encoders with structural patch enhancements (row and column tokens) and apply cross-modal attention using conditional tokens to better ground queries to relevant table regions. The evaluation framework is clear and systematic, using a new benchmark (MMTU) with four distinct task types (IE, RC, CC, CR) alongside existing benchmarks such as WTQ, TabFact, and ComTQA.\n\nHowever, no measurements of model complexity or inference latency are reported. This omission makes it difficult to assess the practical deployment potential of the proposed system.", "theoretical_claims": "There are no formal theoretical proofs in the paper.", "experimental_designs_or_analyses": "The authors present a comparative evaluation across multiple state-of-the-art open-source and proprietary MLLMs, conduct a detailed error analysis (OCR, alignment, and condition misses), and perform ablation studies isolating each component of their method.", "relation_to_broader_scientific_literature": "The paper builds effectively upon recent advances in MLLMs for table and document understanding, referencing core works like LLaVA, BLIP, Donut, DocOwl, and more. It identifies a gap in MLLM performance on CC-style questions and introduces a novel approach to fill this gap.", "essential_references_not_discussed": "No", "other_strengths_and_weaknesses": "Strengths\n（1）The paper identifies common errors in MLLMs when handling tabular data, such as misalignment of the vision encoder with rows/columns (Figure 1: Westport’s negative growth rate being misrecognized as Oil Springs) and the model’s tendency to overlook conditions (e.g., GPT-4o exhibiting a high error rate on CC tasks). The motivation is reasonable. （2）The results indicate that the proposed method achieves a decent improvement.\n\nWeaknesses\n（1）The core improvements of the paper (row/column patches + conditional tokens) mainly focus on enhancing the vision encoder, raising concerns about the originality of the approach. （2）The paper primarily compares MLLMs but does not include comparisons with structured table parsing-based QA methods. （3）Table 2 only reports accuracy but does not provide inference time data. If COCOTAB requires significantly more inference time to achieve better performance, its practical application value may be limited.", "other_comments_or_suggestions": "No", "questions_for_authors": "1.Efficiency Consideration: Your method adds both row/column patches and conditional tokens. Have you evaluated the runtime or memory overhead this introduces during inference? Please include such results if available.\n2.Comparison with Structured Methods: Have you considered comparing your method against strong structured table QA models (e.g., TAPEX, TaPas)? Even if they operate on different modalities, such a comparison would contextualize your model’s relative advantages and limitations.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Phe15CSMs7", "reviewer": "Reviewer_R1mg", "summary": "COCOTAB is a novel method for improving table question answering, particularly for queries with compositional conditions. The paper identifies two key challenges in existing models: the disruption of table structure due to patch-based vision encoders and the models' tendency to miss important query conditions. To overcome these, COCOTAB introduces dedicated row and column patches to better preserve structural relationships and uses conditional tokens to align query conditions with visual features. Additionally, the paper presents the MMTU benchmark, a comprehensive dataset for evaluating various aspects of table understanding. Experimental results show that COCOTAB significantly outperforms prior approaches, especially on complex compositional tasks.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your recognition of our setting and method and your valuable comments. We will answer the questions below, and we hope this clears up your concerns.\n\n**Q1**: While the paper emphasizes the vision encoder’s patch-based approach as a major contributor to misalignment, have you conducted experiments to isolate the impact of the LLM’s reasoning and alignment capabilities separately?\n\n**A1**:  Thank you for raising this critical question. We fully acknowledge the challenge of disentangling the LLM’s reasoning capabilities from alignment issues in multimodal systems. To address this, we designed ablation studies (Section 5.2 and Figure 5) to **isolate the impact of our proposed structural enhancements** (row/column patches and conditional tokens). These performance gains with our architecture were achieved without modifying the LLM, demonstrating that our structural enhancements directly address vision-text alignment limitations, independent of the LLM’s reasoning capabilities.\n\n We agree that stronger LLMs can further improve performance. However, our method focuses on optimizing the visual encoder and cross-modal interaction, which are **orthogonal to the choice of LLM**. This ensures compatibility with any downstream LLM.\n \nIn summary, our ablation studies confirm that the performance improvements stem from better structural alignment in the visual encoder and cross-modal attention, rather than LLM-specific reasoning enhancements. Thank you for your valuable question. We promise that we will make this clear in the final version.\n\n**Q2**: Did you have some qualitative examples showing the failure cases of COCOTAB?\n\n**A2**: Thank you for this question. While COCOTAB improves performance on CC tasks, we acknowledge that failure cases persist, particularly in scenarios involving **OCR errors or complex reasoning**. Here are two representative examples from our analysis:\n \nIn our response in CC task,  the model correctly identified relevant rows but erred in misrecognized scores (e.g., “93” misread as “98” in one cell). As shown in Appendix E, the OCR ability of our model may be strengthened through other strategies, such as more datasets and more vision encoders or tokens.\n\nFrom another perspective, when it comes to CR tasks, the model erred in computing the difference between 63.49 - 38.77 (= 24.72) as 37.22. In this paper, we concentrate on CC tasks. For such calculation and reasoning tasks, other methods such as Group Relative Policy Optimization (GRPO) in DeepSeek will help improve performance. \n \nThanks for your valuable suggestion. We will discuss these failure cases in Appendix and plan to address them via OCR error correction and enhance the reasoning ability in future work.", "claims_and_evidence": "Most claims are supported by convincing experimental evidence. For instance, the paper clearly demonstrates through quantitative comparisons and ablation studies that existing MLLMs struggle with compositional conditions, and that augmenting the vision encoder with dedicated row and column patches—along with introducing conditional tokens—significantly improves performance on benchmarks like MMTU.\n\nHowever, one potential concern is the claim that the vision encoder’s patch-based approach is solely responsible for the shortcomings. While evidence supports its major role, other factors (such as the LLM's reasoning and alignment capabilities) might also contribute, but these aren’t fully disentangled in the experiments. Overall, though, the primary claims are well supported.", "methods_and_evaluation_criteria": "The proposed methods and evaluation criteria are well-aligned with the challenges of table question answering. The authors introduce row and column patches to preserve the structural relationships within tables and conditional tokens to ensure that query conditions are accurately attended. This dual approach directly targets the shortcomings of patch-based vision encoders and misaligned query interpretations.\n\nOn the evaluation side, breaking down table understanding into four aspects—individual elements (IE), rows/columns (RC), compositional conditions (CC), and calculations/reasoning (CR)—along with the introduction of the MMTU benchmark, offers a comprehensive framework.", "theoretical_claims": "The paper does not present formal proofs for its theoretical claims. Instead, it provides mathematical formulations (e.g., Equations 1–4) that describe the model's architecture and the transformation of visual tokens into language embeddings. These expressions are consistent with standard practices in multimodal models and transformer architectures. Since the focus is on designing and empirically validating a new method rather than on deriving novel theoretical guarantees, no issues were found in the provided formulations.", "experimental_designs_or_analyses": "The experimental design is sound overall. The paper evaluates models on table question answering by dividing tasks into four aspects—individual elements (IE), rows/columns (RC), compositional conditions (CC), and calculations/reasoning (CR)—and uses both the WikiTableQuestions (WTQ) dataset and the new MMTU benchmark. The authors also conduct detailed ablation studies (e.g., Figure 5) and error analyses (e.g., Figure 2) to pinpoint issues like misalignment and missing conditions.Yes. I reviewed the supplementary material.", "relation_to_broader_scientific_literature": "The paper builds on prior work in multimodal and document understanding by addressing shortcomings identified in existing MLLMs. Earlier studies (e.g., Donut, mPLUG-Owl, LLaVA) showed that patch-based vision encoders and general alignment methods work well for basic table extraction and OCR tasks, but they struggle with the inherent structure of tables when faced with compositional condition questions. COCOTAB extends these ideas by introducing row and column patches to preserve the table's spatial relationships and by incorporating conditional tokens to more effectively align the query with the table content.", "essential_references_not_discussed": "I didn't notice any missing key references.", "other_strengths_and_weaknesses": "Strengths and weaknesses were discussed in other sections.", "other_comments_or_suggestions": "Overall, the paper is well-written and clearly presents its contributions.", "questions_for_authors": "- While the paper emphasizes the vision encoder’s patch-based approach as a major contributor to misalignment, have you conducted experiments to isolate the impact of the LLM’s reasoning and alignment capabilities separately?\n- Did you have some qualitative examples showing the failure cases of COCOTAB?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xqLdFf7SET", "reviewer": "Reviewer_eFpb", "summary": "The authors identified two issues that the current models lack in the context of multimodal table understanding: 1) the visual encoder’s patch-based processing which splits rows and columns can lead to misalignment, and 2) a failure to fully integrate the conditions specified in the query with the visual features.\n\nTo tackle these issues, the authors propose CoCoTab: 1) augment the vision encoder with row and column patches, and 2) add conditional tokens to explicitly link query conditions to the corresponding image regions.\n\nFinally, the authors propose the Massive Multimodal Tabular Understanding (MMTU) benchmark to assess the full capabilities of MLLMs in tabular understanding, and show that CoCoTab performs best.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your recognition of our setting and method and your valuable comments. We will answer the questions below, and we hope this clears up your concerns.\n\n**Q1**: A well-structured table might not further transfer to a more generic type of vision input.\n\n**A1**: Thank you for this insightful observation. We fully agree that extending tabular understanding to unstructured or free-form visual inputs (e.g., handwritten blackboard tables) is an important and challenging direction. Our work focuses on structured tables for two key reasons:\n\n**Foundation for Real-World Applications**: Existing benchmarks (e.g., WTQ, TabFact) and practical use cases predominantly involve structured tables. Even in these \"well-structured\" settings, current MLLMs struggle with basic compositional reasoning. Addressing these fundamental limitations is a necessary step before extending to more complex, unstructured scenarios.\n\n**Generalization via Training Diversity**: While our primary evaluation focuses on structured tables, CoCoTab’s training incorporates diverse visual QA datasets (e.g., OCR-VQA, PlotQA) that include semi-structured inputs such as book covers, posters, and diagrams. These datasets help the model develop robustness to irregular layouts, partial occlusions, and noisy text—capabilities that can facilitate generalization to blackboard-style tables.\n\nWe acknowledge that fully adapting to free-form tables requires additional innovations, such as dynamic patch resizing to accommodate arbitrary layouts. We will incorporate a discussion of these limitations and potential future directions in the final version. \n\n**Q2**: More patches and the cost.\n\n**A2**: Thank you for raising this practical concern. We acknowledge that introducing additional patches incurs computational overhead. However, our design carefully balances cost and performance:\n\nThe original method divides the table image into 27×27 patches (729 total). COCOTAB adds 27 row patches (one per row, covering full width) and 27 column patches (one per column, covering full height), resulting in only 54 additional patches (7% increase over the original 729). This **modest increase in token count** (from 729 to 783) incurs minimal computational overhead during training and inference.\n\nAs shown in our ablation study, **adding row/column patches alone improves accuracy** on CC tasks over the baseline. This demonstrates that the cost increase is highly justified by the performance gains, particularly for critical tasks like compositional reasoning. \n\nIn summary, COCOTAB achieves **performance gain with only a marginal cost increase**, making it a practical and efficient solution. **Please refer to A1 to Reviewer qd5K for more details.**\n\n**Q3**: The baselines are slim.\n\n**A3**: Thank you for your feedback. Our work focuses on improving tabular understanding in MLLMs, especially for compositional QA. To ensure fair comparisons, we chose state-of-the-art document-understanding MLLMs that integrate advanced techniques like data augmentation and positional encoding. For example, Donut: Uses synthetic data for pertaining. DocOwl: Leverages high-resolution document images for positional awareness. TextMonkey: Utilizes shifted window attention for spatial understanding.\n\nThese baselines represent the strongest open-source MLLMs in this area and inherently employ the techniques you mentioned. Yet, as shown in Table 2, CoCoTab outperforms them on all tasks, highlighting that existing approaches still **struggle with tabular structure alignment and conditional reasoning**. **More experiments are in A2 to Reviewer qd5K.**\n\n**Q4**: The evaluation and comparison don't seem fair. \n\n**A4**: We appreciate the importance of fair comparisons and would like to clarify our experimental design:\n\n**Strict Data Separation**: While WTQ was included in our training data for Stage 1, we rigorously ensured that the test splits of WTQ, TabFact, and other benchmarks were **never exposed during training**. Many compared models are pre-trained or fine-tuned **on the same data** (e.g., DocOwl uses DocVQA, TabFact and WTQ). Our work follows a similar protocol but focuses on structural alignment.\n\n**MMTU**: The proposed MMTU was designed to evaluate TQA from different aspects. It includes tables from WTQ but **reformulates questions and introduces new compositional conditions**, ensuring no overlap with training data. Previous work [1] shows that existing benchmarks lack variation in their questions, resulting in a bad performance with perturbations. CoCoTab's superiority on MMTU and ablation study further validates its generalizability.\n\nIn summary, our experiments adhere to fair evaluation standards, and the improvements are attributable to our method’s architectural innovations rather than data bias. We appreciate your attention and we will make it clear in the final version.\n\n[1] CharXiv: Charting Gaps in Realistic Chart Understanding in Multimodal LLMs. NeurIPS, 2024.", "claims_and_evidence": "The authors posit that current MLLMs underperform on compositional condition questions due to patch misalignment and the overlooking of query conditions. This is backed by initial analysis by splitting TQA into 4 aspects: IE, RC, CC, and CR and show that CC is particularly bad for most models. The main evidence other than the effectiveness of the proposed method (will discuss more in the later sections). In all, the claim/hypothesis is clear and straightforward and easy to test.", "methods_and_evaluation_criteria": "Strength:\n- The proposed method is simple and shown to be effective. \n\nWeakness:\n- The proposed method can be seen as a form of data augmentation. Despite the simplicity and effectiveness, this type of data augmentation might have the following limitations: 1) assumes a particular structure of the data (well structured table) and might not further transfer to more generic type of vision input (e.g., an image of tables drawn on a blackboard), 2) by adding more patches and training on them it increases the cost. \n- The baselines comparisons are slim. How does this method compare with other data augmentation methods or position-aware encoding methods.\n- The evaluation and comparison doesn't seem totally fair. The proposed method is trained on a couple of datasets (sec 4.3), some are of the same distribution w.r.t. the final evaluation (e.g, WTQ), but the other models in comparison are not further trained.", "theoretical_claims": "The authors do not propose theoretical claims in this paper.", "experimental_designs_or_analyses": "See Methods And Evaluation Criteria.", "relation_to_broader_scientific_literature": "The authors positioned the work in the MLLM and tabular understanding literature. But I find the discussion a bit lacking around the data augmentation aspect of the proposed method. I would love to see more comparison on this front.", "essential_references_not_discussed": "See Relation To Broader Scientific Literature.", "other_strengths_and_weaknesses": "The paper is clear and easy to follow. My concern is mostly around the baseline comparisons and the evaluation protocol (i.e., other models are not being trained, thus the comparison is not exactly fair).", "other_comments_or_suggestions": "N/A", "questions_for_authors": "See previous sections.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "bQW26TGa3C", "reviewer": "Reviewer_qd5K", "summary": "This paper mainly aims at the problem of poor tabular comprehension ability of MLLM, and proposes a new method for extracting visual context information. By adding row and column-based image patch segmentation, and using the cross-attention method to fuse visual and textual features, the MLLM can capture the contextual relationships at the row and column levels in the table and the connections between images and text, thereby improving its performance on tabular Q&A tasks. At the same time, this paper also proposes a multimodal tabular understanding benchmark MMTU, including four tabular understanding tasks across ten different domains, to better evaluate the tabular understanding ability of MLLM.\n\n\n## update after rebuttal\nI still maintain my original decision because I think there are some fatal problems with this paper that have not been resolved.\n1. The introduction of additional row and column patches does bring many extra visual tokens. This is contrary to the current development direction of MLLM. For high-resolution images, this method has obvious limitations. Moreover, there are already many redundant visual tokens in the current MLLM, and this method further increases the redundancy.\n2. The authors configure COCOTAB with the pre-trained Siglip-ViT as the vision encoder and Qwen2-Instruct as the backbone for LLM. However, in the ablation study, they use the vanilla LLaVA-1.6 as the baseline to verify the validity of each component. This operation is puzzling. The authors did not address this issue in their rebuttal. Why not use the same baseline?\n3. The experiment content is too small, only including a table and a figure. More adequate experimental design and analysis are needed.\n4. Compared with some recent MLLMs, the proposed model does not have a significant performance advantage.\nIn summary, I think the current version of this paper is not well prepared and does not meet the standards of ICML.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your recognition of our setting and method and your valuable comments. We will answer the questions below, and we hope this clears up your concerns.\n\n**Q1**: additional computational overhead\n\n**A1**:  We appreciate the need to rigorously evaluate computational efficiency. While our paper focused on accuracy and structural alignment improvements, we conducted additional experiments to quantify the computational overhead of CoCoTab. Here are the key results and analyses:\n| Model         | Time (s) | Memory (G) |\n|--------------|---------|-----------|\n| **InstructBlip** | 0.294 | 30.98|\n| **LLaVA1.6**     | 0.456 | 17.21|\n| **DocOwl**       | 0.3695| 19.50|\n| **TextMonkey**   | 1.684| 23.96|\n| **Donut**        | 0.425 | 0.478|\n| **internvl**     | 1.277 | 43.28|\n| **TabPedia**     | 0.201 | 19.02|\n| **CoCoTab**      | 0.209 | 20.47|\n\nWe have measured the **average inference time** and memory consumption per sample. As shown in the results, our method does not introduce significant overhead. On the contrary, compared to LLaVA, our approach provides more **concise and precise answers**, leading to improved inference speed. As noted in response A2 to Reviewer eFpb, COCOTAB adds only 7% in token count but outperforms in all baselines. \n\n**Q2**: More baselines and results.\n\n**A2**: Thank you for your critical feedback. We appreciate the opportunity to clarify and strengthen our experimental comparisons. Below, we address each concern systematically:\n\nOur work focuses on document and tabular understanding, a subfield of MLLM that demands specialized structural alignment capabilities. To ensure fair and state-of-the-art comparisons, we selected the strongest open-source baselines published in 2024, including LLava1.6(baseline), TextMonkey, DocOwl2.These models represent the most advanced techniques in document/table understanding as of mid-2024. \n\nFor completeness, we also compared it with InternVL2.5-8B and TabPedia (NeurIPS 2024).\n| Model| IE  | RC  | CC  | CR  |\n|---------|----|----|----|----|\n| InternVL| 0.68 | 0.48 | 0.25 | 0.6  |\n| TabPedia| 0.71 | 0.47 | 0.34 | 0.27 |\n| CoCoTab| 0.68 | 0.5  | 0.43 | 0.38 |\n\nAlthough our method may not outperform InternVL and TabPedia in certain aspects, it significantly **surpasses both in CC** performance. Moreover, our training data (1.8M) is considerably smaller than that of TabPedia (~4M) and certainly much less than InternVL. Despite this, our approach still achieves competitive performance. We believe these results conclusively demonstrate CoCoTab's advancements in tabular understanding and we promise we will add these comparisons in our final version.\n\n**Q3**: The improvement on weak baseline LLaVA.\n\n**A3**:  Thank you for raising this critical point. We appreciate the need to validate our method’s contributions against stronger baselines and would like to clarify our experimental design:\n\n Our ablation studies (Section 5.2) intentionally used the original LLaVA **architecture (Vision encoder + MLP + LLM)** to isolate the impact of our proposed components (row/column patches and conditional tokens) under strictly controlled conditions. By keeping the LLM, training data, and hyperparameters identical, we ensure that performance differences directly reflect the effectiveness of our structural enhancements. This approach is standard in ablation studies to **disentangle contributions from architectural changes** versus other factors. Our ablation study shows that the performance gain validates the importance of our proposed components.\n\nFrom another perspective, our ablation baseline (LLaVA + our training data) already **achieves higher accuracy** than vanilla LLaVA-1.6, demonstrating that our curated training data itself improves performance. This validates the quality of our data pipeline even before adding structural components.\n\n**Q4**: How the image patches with the size of HP and PW are input into the image encoder.\n\n**A4**: We appreciate the opportunity to clarify our methodology for handling multi-scale patches.\n\nIn our CoCoTab, we generate row-wise (HP) and column-wise (PW) patch embeddings in parallel with the standard PP patches using lightweight convolutional operations. Here are the key steps:\n\nThe input image is resized to 384×384 (as in SigLIP) and split into 27×27 patches of size 14×14 (P=14) with a 2D convolutions with kernal size PP, following standard ViT processing. Similarly, we apply **2D convolutions with kernel sizes H×P (for rows) and P×W (for columns)** to the resized image. These operations are computationally lightweight and performed before the ViT’s patch embedding layer. This design allows the ViT to implicitly reason about row/column structures without violating its architectural constraints. We will revise Section 4.1 to detail these steps.\n\nThank you again for highlighting this ambiguity. We hope this clarification helps strengthen the details of our proposed CoCoTab.", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "This paper proposes a new benchmark for MLLM tabular understanding, which includes richer domains and problem types, which is meaningful for the evaluation of MLLM tabular understanding ability.", "theoretical_claims": "Yes, the proposed claims are correct.", "experimental_designs_or_analyses": "1. The proposed method increases the number of input image patches and the input context length of the LLM in the image encoder stage, which will introduce additional computational overhead in model inference, but the proposed method is not compared with other methods in terms of model inference speed and computational overhead in the experimental part.\n2. The methods compared in the experimental section are outdated.\n3. The overall results of the method proposed by the authors are relatively low in the current community.", "relation_to_broader_scientific_literature": "None", "essential_references_not_discussed": "The author needs to add more papers on multimodal large models published in the second half of 2024.", "other_strengths_and_weaknesses": "My main concerns are about the issues mentioned above regarding experiments.", "other_comments_or_suggestions": "The author used the original LLaVA as a baseline and added two components on top to prove their effectiveness. However, the performance of the original LLaVA is low and is far behind some of the latest MLLMs. The improvement on such a weak baseline does not necessarily prove that these components can bring new inspiration or help to the community. The author should verify it on a stronger baseline.", "questions_for_authors": "In addition to splitting the input image with the size of H*W into patches of P*P, the input image is also slice according to the dimensions of H*P and P*W. The image encoder used in the article is ViT, which can only accept the input patch size with a fixed size of P*P, and the article does not mention how the image patches with the size of H*P and P*W are input into the image encoder.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper proposes CoCoTab, a novel method addressing limitations of Multimodal Large Language Models (MLLMs) in handling compositional condition questions in tabular understanding tasks. Reviewers highlighted clear strengths, including well-motivated enhancements to the vision encoder via additional row and column patches and the introduction of conditional tokens for improved condition-query alignment. The proposed Massive Multimodal Tabular Understanding (MMTU) benchmark further enriches the evaluation landscape by rigorously assessing compositional reasoning abilities across diverse tabular tasks. Despite initial concerns raised regarding model efficiency, inference overhead, and comparisons to structured parsing methods, the authors' rebuttal provided thorough clarifications and supplemental experimental data demonstrating reasonable computational overhead and superior performance on compositional reasoning tasks relative to state-of-the-art baselines. Given these clarifications and empirical results, the paper constitutes a meaningful advancement in multimodal tabular QA. Based on the final rating from reviewers, I recommend a weak acceptance (low priority: accept if there is room in the program) for this submission.
Decision: Accept (poster)