TabPedia: Towards Comprehensive Visual Table Understanding with Concept Synergy
Authors: Weichao Zhao
Hao Feng
Qi Liu
Jingqun Tang
Binghong Wu
Lei Liao
Shu Wei
Yongjie Ye
Hao Liu
Wengang Zhou
Houqiang Li
Can Huang
Venue: NeurIPS 2024 poster (2024)
Keywords: Multimodal Large Language Model
Visual Table Understanding
Multimodal Learning
Primary area: machine_vision

Abstract: Tables contain factual and quantitative data accompanied by various structures and contents that pose challenges for machine comprehension. Previous methods generally design task-specific architectures and objectives for individual tasks, resulting in modal isolation and intricate workflows. In this paper, we present a novel large vision-language model, TabPedia, equipped with a concept synergy mechanism. In this mechanism, all the involved diverse visual table understanding (VTU) tasks and multi-source visual embeddings are abstracted as concepts. This unified framework allows TabPedia to seamlessly integrate VTU tasks, such as table detection, table structure recognition, table querying, and table question answering, by leveraging the capabilities of large language models (LLMs). Moreover, the concept synergy mechanism enables table perception-related and comprehension-related tasks to work in harmony, as they can effectively leverage the needed clues from the corresponding source perception embeddings. Furthermore, to better evaluate the VTU task in real-world scenarios, we establish a new and comprehensive table VQA benchmark, ComTQA, featuring approximately 9,000 QA pairs. Extensive quantitative and qualitative experiments on both table perception and comprehension tasks, conducted across various public benchmarks, validate the effectiveness of our TabPedia. The superior performance further confirms the feasibility of using LLMs for understanding visual tables when all concepts work in synergy. The benchmark ComTQA has been open-sourced at https://huggingface.co/datasets/ByteDance/ComTQA. The source code and model also have been released at https://github.com/zhaowc-ustc/TabPedia.

TabPedia: Towards Comprehensive Visual Table Understanding with Concept Synergy
Weichao Zhao 1.2, Hao Feng l.+
Qi Liu Jingqun Tang? Shu Wei
Binghong Wu? Can Huang?
Lei Liao?
Yongjie Ye? Hao L Wengang Zhou' Houqiang Li' University of Science and Technology of China, ByteDance (saruka, haof) @mail.ustc edu.en, (zhwg, lihq) @ustc.edu.en (liuqi.nero, haoliu.0128, can.huang' @hytedance.com
Abstract
Tables contain factual and quantitative data accompanied by various structures and contents that pose challenges for machine comprehension. Previous methods generally design task-specific architectures and objectives for individual tasks, resulting in modal isolation and intricate workflows. In this paper, we present a novel large vision-language model, TabPedia, equipped with concept synergy mechanism. In this mechanism, all the involved diverse visual table understand- ing (VTU) tasks and multi-source visual embeddings are abstracted as concepts. This unified framework allows TabPedia to seamlessly integrate VTU tasks, such table detection, table structure recognition, table querying, and table ques- tion answering, by leveraging the capabilities of large language models (LLMs). Moreover, the concept synergy mechanism enables table perception-relatt rellate and cumprehension-relatee tasks to work in harmony, as they can effectively leverage the needed clues from the corresponding source perception embeddings. Fur- thermore, to better evaluate the VTU task in real-world scenarios, we establish a new and comprehensive table VQA benchmark, ComTQA, featuring approx- imately 9,000 QA pairs. Extensive quantitative and qualitative experiments on both table perception and comprehension tasks, conducted across various public benchmarks, validate the effectiveness of our TabPedia. The superior performance further confirms the feasibility of using LLMs for understanding visual tables when all concepts work in synergy. The benchmark ComTQA has been apen-sourced at https://uuginngfacecomm http://uuginggicccoooo/ttt////// www.e//dtssseeeeeeeeeeeo//oooooo also have been released at https//gghhabbco///awwwut/tt/// Padeea
The source code and model
Introduction
With the rapid advancement of digital technology, numerous paper documents must be converted into electronic formats for efficient storage and utilization. Tables, as indispensable components of documents, play a vital role in summarizing facts and quantitative data 2 The compact yet informative nature of tables makes them advantageous for various applications, thereby attracting widespread research attention toward Visual Table Understanding (VTU). VTU generally encom- passes four subtasks: Table Detection (TD), which locates tables within document images; Table Structure Recognition (TSR), which parses the structure of tables in table-centric images; Table Querying (TQ), which recognizes the structure of table from an entire image at given location, task that remains underexplored in the previous works; and Table Question Answering (TQA), which
Equal contribution. Interns at ByteDance. Project lead. X Corresponding authors: Wengang Zhou and Hao Liu.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Table Query
different task
TD ba Dtipe:
Stracture Parser
Instruction
Table Detector
table locationns
TSR The stractund
Table Structure Parscr
morkup code / table anlic incations
TabPedia
Table QA En-Decader
TQ settettttnn
specific response
TQA Dt Siise bads
region tel, whe codt yol
waghtin
Input
Previous task-spucific pipelines
(b) proposed TabPudiia
table detection (TD)
table structure recognition (TSR)
table query (TQ)
mall question answering (TQQ))
Figure I:
Comparison with previous task-specific pipelines for visual table understanding. In contrast
to design different architectures for various table tasks, our TabPedia effectively performs these tasks
in unified framework through delicately leveraging the understanding capability of LLMs.
answers questions based on table contents. These tasks pose challenges from various perspectives due to the need for representations at different visual-semantic granularities and hierarchies.
Given the success achieved, many pioneering works have mainly centered on the specific subtask with various task-specific architectures, as shown in Fig. (a). For visual table perception tasks such as TD and TSR, one of most adopted approaches is the detection manner Bg In contrast, generative vision-language models 10 are often employed to generate answers conditioned on the semantic content tables for TQA task. Specifically, Vision Transformers (ViT) 14 pretrained on CLIP
15 or EVA-CLIP (16). Transforree [17], and similar models serve vision encoders, while [18] [9] or decoder-only frameworks [1] 22). Besides, recent fast-growing Large Vision Language Models (LVLMs) [T] B4 have shown their powerful capabillities to perceive and understand visual clues by integrating instruction following of Large Language Models (LLMs) 39). Despite impressive progress, the status quo begs for question: "Can we leverage the advantages of LVLMs to solve all the VTU tasks once and for all?"
language models operate in either
A straightforward solution would be to train the LVLM directly using all the VTU data. However, aside from the diverse table structure and the various relations of table contents, it remains a nontrivial issue due to two cruxes of table parsing and understanding: (i) discrepancy between the representation formats (two-dimensional structure VS. -ie-immnsioonal sequence); (ii) required image resolutions. Although some works 44 represent table structure in markup formats like HTML, XML, Markdown, or LATEX. However, they neglect spatial coordinates for cells and only encode logical relationships implicitly. The generated code contains extensive formatted information from different markup languages, increasing output length and potentially causing parsing issues with illegal grammars.
To attack above issues, we this paper propose novel LVLM tailored for comprehensive VTU, TabPedia, to effectively solve all VTU tasks in unified framework, as shown in Fig. (b). More concretely, we employ dual vision encoders, namely ViT-L. [15) and Swin-B [43) to encode the global and fine-graine' local information in the low- and high-resolution formats of the input image respectively, acquiring multi-suurce visual embeddings. Here, all the involved VTU tasks and multi- source visual embeddings are abstracted concepts and concept synergy mechanism is implemented by introducing the mediative tokens to the L.LM in our model. Thanks to this mechanism, all the concepts in TabPedia on both table perception and comprehension tasks across various public benchmarks confirm the effectiveness of our proposed TabPedia. To further investigate the potential of our model in more challenging and realistic scenarios, we establish a new and comprehensive table VQA benchmark, ComTQA, featuring round 1,500 images and 9, 9,000 QA pairs.
work in synergy flexibly. Quantitative and qualitative experimental results
Our contributions are summarized as follows,
We propose a novel large vision-language model, TabPedia, to integrate various VTU tasks into unified framework, including TD, TSR, TQ and TQA. Specifically, TabPedia fully leverages the comprehensive capabilities of LLMS to fertilize complex table understanding.
We design a concept synergy mechanism to harmonize both table perception and compre- hension tasks. Through introducing the meditative tokens into our framework, TabPedia

---

adaptively enables useful information in multi-source visual embeddings and task instruc- tions, generating accurate and plausible responses.
Extensive quantitative and qualitative experiments validate the effectiveness of our proposed TabPedia across various tasks and benchmarks. To further exploit the potential our model in more complex scenarios, we build a new table VQA benchmark, ComTQA, involving multiple answers, mathematical calculation and logical reasoning, etc.
2 Related Work
2.1
Table Recognition
Table recognition is generally divided into table detection, table structure recognition and table content recognition In our work, table content recognition is beyond our scope.
For TD task, the earliest approaches are rule-based methods for locating tables inside documents 46.. With the rapid advances in deep learning. numerous CNN-based methods show impressive performance. Most of these methods directly adopt top-down object detection frameworks solve this problem [5| 52 For instance, Sun et al. [52) adopt Faster R-CNN
[52] detect table boxes and the corresponding corner boxes simultaneously, and then adjust table boundaries according to the detected corners. Some other methods model each document image as graph and formulate TD as graph labeling problem [53 55). In addition, TATR [9] first applies the transformer manssdreer-baeedd detector, DETR [56], to improve the detection accuracy without special customization.
For TSR task. one of the most common modeling approaches still to regard as some form of object detection 5 9 57-60). Among them, DeepDeSRT [4] and TableNet [61] are both representative works exploring semantic segmentation to obtain table cell boundaries. TATR ( first proposes to utilize DETR for this task. TSRFormer (58) introduces a cross-attention module into the DETR framework to improve the localization accuracy of row/column separators. Some other methods attempt parse table structure via modeling relationship among different table elements (62H64). As the most relevant to our approach, markup generation-base methods directly generate markup (HTML LaTeX) sequences from raw table images 41 665.. EDD 65) introduces cell decoder and a structures decoder to generate HTML codes. OmniParser 41 further integrates three task specisc decoders to enhance the table structure representation.
While the previous methods have achieved promising results on table perceptive tasks, they are still limited in table intricate content understanding. In our work, we jointly exploit table perception and comprehension tasks in a unified framework, concurrently enriching visual table understanding.
2.2 Large Vision-Language Models
with visual comprehension capability. The main- stream approaches attempt to connect visual encoders and LLMs with intermediate modules such as simple Projectors (30], Qooreer (25) Perceiver Resamplers (33, achieving visual language under- standing through pre-training alignment and instruction fine-tuning. For text-rich document scene, several works [30 [3
LVLMs aim to equip LLMs [29 36 38 39
72
72,74) propose to enhance the LVLMMs capabilities in understanding textual elements (text-centric VQA, OCR, text spotting, etc.). Among them, TextMonkey [12] employs shifted window attention and token resampler module to improve the training process. DocOwl-1.5 40 collects comprehensive dataset DocStruct4M to support unified structure learning.
Despite achieving extraordinary progress on visual understanding, existing LVLMs still face chal- lenges in two wwo-dimensional table parsing and understanding. In this paper, we propose unified framework concurrently achieve table perception and comprehension with the support of LLMs.
2.3 Additional Tokens
In the trend of Transformer-based approaches, extending the input sequence with special tokens is popularized for various intentions, such as extracting task-specific information [14, (6), providing
extra information [75 76 or improving model performance 80). For instance, VIT [4 utilizes [CLS] token for classification. Similarly, DETR (56) proposes object queries for detection. ATR [76] adopts tape tokens to obtain useful information from a memory bank. In addition, the Memory Transformer [77] presents simple approach to improve translation performance by attaching trainable memory tokens after the token sequence. Darcet et al., [79] further attempt to add extra tokens in VIT- based frameworks, e.g., CLIP [15] and DINOv2 [81), thus improving visual tasks. In our work,

---

Pre- Trainign
Hendering
Here
element
Vicuna-7B BB
1110,0.43,,55030000
Projections
Large Language Model (Vieuna-7B)
Vision Encoders
Mediative Tollns
Fime-Tuning
Tokenization
Projection
Projection
Vicuna-7B
"Display the table blement positions in this picture."
Low-Resolution Vision Encoder
Righ-Resolution
Projections
Vision Encoder
Vision Encoders
The illustration of our proposed TabPedia. Given the input image, TabPedia feeds into both vision encoders attached projections to extract different granular features. Then, the visual tokens are combined with instruction-deriven tokens, and fed into the LLM. The LLM leverages its powerful understanding ability to generate plausible response.
Figure 2:
inherit this spirit and design meditative tokens to enhance TabPedia's perceptive and comprehensive capability for visual tables.
Method
As shown in Fig 2 we present an overview of TabPedia. The overall training pipeline consists of two phases. Concretely, the pre-training stage aims to align the visual features to the large language model, and the fine-tuning stage focuses on visual table-aware understanding. elaborate on the architecture of TabPedia, followed by the exposition of its two training phases.
the following, we
3.1 Model Architecture
the high-resolution image is critical to ensuring that the LLMs could grasp rich visual information. Following Donut [3] we adopt Swin-B B [17 to encode the high nigh-resolution format of input image. Given the input RGB image I, first resize it to pre-defined high-resolution scale of H x W, denoted In. By default, both Н and W are set to 2,560 and 1.920, respectively. Notably, we maintain the aspect ratio during the resizing process prevent distortion of table contents and structures. Then, the resized image 4, fed into the vanilla Swin Transformer initialized from 43 to obtain a feature map V,, downsampled by factor of 1/32, each token with 1,024 dimension.
Nrigh-Resolution Vision Encoder. As proved by previous methods 43 82 83
Now-Resolution Vision Encoder. To keep the overall layout information, the raw image is also resized to low-resolution one denoted as 4. We choose the pre-trained CLIP visual encoder ViT- L/14 [55 to encode the low-resolution image with 224 224, which has been pre-trained on 400 million image-text pairs sourced from the open-world data, thereby embedding extensive world knowledge into its pretrained weights. To preserve its generalization ability, we keep frozen during the whole training procedure. The output sequence Vi is composed of 256 tokens, each with 1024 dimension.
Projections. The projections are designed to align visual tokens with the input token dimension of the subsequent large language model [66]. For the high-resolution feature map V,, due to the limitation of input text length, we employ 2D convolutional layer with kernel size of 3 and a stride of 2, and then flatten it into H. W inspired from the paradigm of advanced LVLMs 29 30), we adopt a linear layer to project visual
64 64 tokens, denoted as Vg,- For the low-resolution visual features Vi,
tokens, denoted as Vi.
Concept Synergy. Given the massive visual tokens and the embedding of textual instruction Q, we utilize Vicuna-7B (66) as LLM to generate its response. Taking into account the discrepancy of table perception and comprehension tasks, we introduce meditative tokens M to implement the concept synergy for the I.I.M, which adaptively enable different region of visual tokens and understand the intentions of specific task question. Finally, construct the whole input sequence as X

---

Table I: Summary of training data statistics in the fine-tuning stage.
Table 2: Different task types and their instruc
tion examples.
Datasel
Subset
Task
Num
Task
Example
M-Dta P--
TD
460%
TD
"Give me the areas where table clement's
PubTabIM
TSR,TQA
759k
locations in this picture."
PuhTab Syn
TQ
381k
TSR
"Parse the structural information of the cropped table in this picture."
FinThbbNet
TSR,TQA
78k
PubTabNet
TSR
434k
TQ
"Parse the table structure within the region [0.095, 0,673, 0.869, 0.851] in this picture."
WTQ
TQA
1k
TQA
"What was the lowest stock price in the fourth quarter 2010?"
TabFact
TQA
9k
IQ, <IMG_S> ; Vi SIMG_SEP" Vn <IMG_E> M), where [:] means the concatenation operation. <IMG_S> s>____s_e__ <IMG_E> and <IMG_SEP) are learnable special tokens, that denote the start and end of visual tokens well as the separation of different resolution tokens, respectively.
Objective. Since TabPedia trained to predict the next tokens like other LLMs, optimized by maximizing the likelihood of prediction loss at training time.
3.2 Pre-training
To enable the capable of vision encoders capture text-rich information from high-resolution images and aligning embedding space with the large language model [66], we first perform extensive text- aware pre-training. As shown in Fig. 2 jointly optimize the high-resolution visual encoder with both projectors, while freezing the large language model and low-resolution vision encoder. Specifically, followed by (10), our pre-training procedure involves a variety of perception tasks. i.e., text detection 184), recognition [85]. spotting [86]. long-text reading [43] and image captioning [87). The first four tasks focuses on the various document images, while the last one targets natural scene images. These comprehensive tasks endow the vision encoders of TabPedia to effectively perceive textual and visual information from both document and natural scene images. More detailed pre-training settings about dataset and experiment could be referred to [10].
3.3 Table-aware Fine-tuning
Through pre-training, TabPedia could well understand text and structure of diverse document images but cannot follow instructions to perform different table understanding tasks. In order to enhance the model capability of instruction following, we first construct a large-scale dataset for visual table understanding. We will elaborate on the dataset construction in the Sec. 4 Based on this dataset, we introduce four table-related tasks, i.e., TD 9 TSR 9 65 TQ and TQA g
gg g9 to
simultaneously cultivate the perception and comprehension capabilities. In this stage, we further unfreeze the LLM and fine-tune the entire framework except the low-resolution vision encoder.
4 Dataset Construction
In this section, we aim to introduce the collected instruction following dataset. The entire data derived from five public datasets, including PubTablM (. FinTabNet [5), PubTabNet [65].
WikiTablleQuestions (WTQ) (88 and TabFact [89). Among them, PubTabl [9] contains two subsets, ie.,, PubTab M-Detection PubTabiM-Der and I--tructue (PubTabl M-Str).. Moreover, since the table images in PubTab 1 are cropped from PubTabl M-Det, we transform the annotations of the table structure in Pub LubTabIM-Su into the original images and synthesize a new subset PubTabl M--Snn which could be utilized for TQ task. The statistical data are summarized in Tab. I To ensure the instruction diversity, we generate multiple instructions for each task using GPT3.5 [1 In Tab. 2 we display one exemplar about user's question for each table task. We will provide a detailed exposition of them in the following.
Table Detection (TD). As fundamental task, TD task targets to detect all table locations in document image. Previous methods K3 mainly utilize DETR (56) or variants of R-CNN [90 to predict numerous overlapping bboxes, that inevitably needs complex post-processing. such as non-maximization suppression (NMS), to generate final results. In contrast, we employ L.M to directly generate the locations of instance tables in the format of "[x1, yl, x2, y2]", where xl. yl, x2,
5

---

y2 represent the normalized coordinates of the top-left and bottom-right of the corresponding bbox. Moreover, to facilitate detection results for multiple tables, we split multiple table positions with the special symbol "'n" in the output response. We adopt PubTabIMMDDe [ to perform TD task, where images are collected from PDF documents with different scale and rotation types of tables.
Table Structure Recognition (TSR). The TSR targets to parse table structure in terms of rows, columns and cells. HTML and Markdown codes are mainly two kinds of text sequences used to represent a table. HTML could represent all kinds of tables, with or without cells spanning multiple rows and grids, but they contain massive markup grammars i.e. "<div></div>" and "<tá></td>> resulting in excessively lengthy output responses. Compared with HTML, Markdown represents a table more succinetly, but it cannot represent cells spanning multiple rows or columns. By weighing the simplicity of the output and the completeness of the table parsing, we propose canonical table structure representation based on the detection format. Inspired by [9, we jointly adopt five object classes to model TSR, including table column, table row, table column header, table projected row header and table spanning cell. To better understanding. we display a representative sample in Appendix B Taking into account the serialized output of the LLM, we represent the table structure with series of "[object] [xl, yl, x2, y2]", which are also separated by "In". Notably, we standardize the order of the output objects to ensure uniqueness of the table parsing results.
We select the PubTablM-Si 9. FinTTbbNet [5] and PubTabNet [65 to support the TSR task. where tables are collected from scientific and financial articles. These datasets contain pairs of table images and HTML annotations. We convert HTML codes into our designed annotation format using the pre-processing tool offered by (9.
Table Querying (TQ). Different from recognizing table structure from the cropped table-centric images in TSR task, the TQ task directly parses the table from the original document image based on the given table location. This task is more challenging due the degradation of the table's 's resolution and the interference of other document contents around it. Moreover, this task could potentially be combined with TD task to enable automatic parsing of all table structure information in original images. Therefore, we introduce this task to fully unlock the comprehension capabilities of large language models for visual table understanding. For the annotation of table parsing, we adopt the same format as TSR. Since there is no readily available dataset, we synthesize a large amount of available data based on the annotations from P [9, namely M--Tab
Table Question Answering (TQA). TQA aims to provide precise answers through table understand- ing and reasoning. For both public TQA datasets, i.e., WTQ (88) and TabFact (89). the table images are collected from wikipedia tables with pairs of content-related question and answer. Thus, we could directly apply these available data to support this task. However, the images of current TQA data are rendered from text-based tables with variations in background color and font size, resulting in poor generalization in real-world tables. In addition, the TQA data volume lags far behind other tasks. To alleviate these obstacles, we generate numerous TQA data with partial images in Fin TabNet [5] and PubTablN MM [9] by employing the powerful
understanding capabilities of Gemini We provide more detailed descriptions of the procedure in the Appendix A.
Pro 193]
To better evaluate TQA performance of various models on real-world table images, we build complex TQA dataset (ComTQA) based on test set of Fin TabNet [3] and PubTab IM [9]. Compared to WTQ and TabFact, ComTQA has more challenging questions, such as multiple answers, mathematical calculations, and logical reasoning. In total, we annotate ~99 high-quality QA pairs from ~ 11.5k images by expert annotation. More statistics about ComTQA could be found in the Appendix A.2
5 Experiment
5.1 Implementation Details
Parameter Settings. For the hyper-parament parameters in model design, the number of meditative tokens is set to 256. The max length of text sequence is set to 4000 satisfy task requirements. To implement TabPedia, we adopt a cosine schedule with one-cycle learning rate strategy [94.. In the pre-training phase, the learning rate warms up the first 2% of the training process and then decreases from the peak rate (1e-3) with batch sizes of 64. In the fine-tuning phase, we set the peak learning rate as 5e-6 with batch sizes of 16. We employ the AdamW optimizer [95] both phases. All experiments are implemented by PoTorc (96] and trained on 16x A100 GPUs.

---

Table 3: Comparison with the existing best table detec- tion model TATR [9] NMS denotes Suppression.
Table 4: Comparison with end-to- end TSR methods on two datasets. represents the results reported by 41.
IoU@0.75
Method
Backhone
NMS
Precision
Recall
F1
Method
Input Size
Neb
S-TR0S
FinTabNNet
S-TRD8
TATR 4
Faster R-CNN
DETR
92.7
98.8
98.1
86.6
98.4
89.5
Donur
EDD
6S
1,280
512
89.90
25.28
90.60
30.66
Chnni
1,024
90.45
TabPedia
LVLM
X
98.5
98.4
98.4
TahPodia
2,560
95.41
95.11
Table 5: Quantitative results on two subsets of Pub-
Table 6: Comparison with existing LVLMs on TQA task. "x" denotes the results obtained through the open- source checkpoint or API of the closed- source model. ComTQA our re- leased new benchmark. The second best methods are underlined.
Tab M [2]. including PubTabIM-Str and PubTablM- Syn.
(a) Comparison with the task-specific model, TATR O TSR
task. "Cropped" denotes utilizing cropped table-centric images.
Method
Hackhooee
Image
NMS
Sub
GriTSmm
GelTScos Ge'VSu 8-1108
TATR 0
Paeur RACN
DETR
Coppod
Croppad
98.46
16.16
$5.88
97.81
97.11
72.11
97.65
Method
Input Size
WTQ
TabPact
ComITQA
Acc
TobPedia (TSR)
LNLM
Chopped
96.52
95.4
95.66
TeuMonkey
Monkey
896
49.8
(b) Quantitative results both TQ and TD+TQ tasks.
Cogagen 100
1,120
30.2"
51.7"
DezOsi 1.5 20
1,344
39.8
30.4
13.5"
PubTa
GPT4V [0]
645
45.5"
69.3"
27.2"
Method
Image
NMS
Task
Gesiai Pro 93
32.3'
67.9"
GriTS
GriTS GriTS
S-TEDS
Xoompaser2
511
TabPedia
Raw
TD-TQ
94.54
94.63
96.23
94.95
93.25
93. 38
TabPedia
2,300
47.8
71.3
53.5
Datasets. In order to comprehensively evaluate the capability of TabPedia, we employ multiple benchmarks for each task. For performance assessment, we set the temperature parameter as 0.2 in both quantitative and qualitative evaluations. For TD task, Pub'Tab1 M-Det [9] contains 57,125 images for testing. For TSR task, FinTabNet [5), PubTabNet (65) and PubTabl 1 M--Str (9 are adopted for evaluation with 9,289, 9,115 and 93,834 testing samples, respectively. For TQ task, the synthetic dataset PubTab M- [9 also provides 47,186 samples for testing. For TQA task, WTQ (88), TabFact [89 and our annotated ComTQA contain 4,343, 12,722 and 9,070 QA pairs, respectively.
Evaluation Metries. For TD task, we report the results with object detection metrics, including precision, recall and fl-score with loU@0.75. For both TSR and TQ tasks, we utilize Structure Tree-EdidDistance-biie Similarity (S-TEDS) (65), which evaluates table similarity of structural aspects in HTML format. The metric represents the HTML table as a tree, and the TEDS score computed through the tree-edit distance between the ground truth and predicted trees. In order to convert the results of TabPedia into HTML format, we employ the post-pros pest-processing algorithm provided by (9. Moreover, we report the recently proposed GriTS metrics (97 for PubTabl to align its original metric. Different from S-TEDS, GriTS represents tables matrices, better capturing the two-dimensional structure and the orders of cells in a table. Further, GriTS enables TSR to be assessed from multiple perspectives, with GriTSTop measuring cell topology recognition, GriTS measuring cell content recognition, and GriTS measuring cell location recognition. For TQA task, we adopt the accuracy metric where the response generated by the model is judged correct if it contains the string present in the ground truth (98).
5.2 Quantitative Results
We conduct quantitative evaluations of current state-of-the-art methods for specific tasks in perception and comprehension, comparing them to our proposed TabPedia.
Evaluation on TD. In Tab. 3 we compare TabPedia with the previous state-of-the-an method, TATR 9 TATR performs the table detection with two classic visual detection backbones, i.e, DETR (56) and Faster R-CNN (91). Compared with them, TabPedia outperforms Faster R-CNN with a notable margin and achieves competitive performance with DETR. Notably, since TabPedia directly generates the independent locations of instance tables without densely overlapped bboxes,
7

---

there are no extra post- pro--rreessing operations involved, i.e., Non-Maximum Suppression (NMS). This advantage could enable TabPedia to perform more complex table understanding, such as parsing all tables by combining TD and TQ tasks.
Evaluation TSR. Tab. reports the performance of TSR task compared end-to-end TSR models on PubTabNet and Fin TabNet datasets. Specifically, the OCR-free model Donut [43] is fine-tuned for TSR with the official default training configuration. Although OmniParser [4 integrates multiple visually-situater text parsing tasks into a unified framework, it adopts three isolated decoders to perform different tasks. Compared with OmniParser, TabPedia consistently surpasses it with 4. 4.96% and 3.56% S- TEDD on both datasets, respectively. In Tab. 5a TATR as the task-specific method, shows high performance with the DETR architecture. Our proposed TabPedia, a generic model for tasks involving both perception and comprehension, still achieves comparable performance without the need for complex post-processing. These results highlight the exceptional capability of TabPedia.
Evaluation on TQ. As a new and unexplored task, the TQ task aims to parse table structures with the specific location directly from the raw image without additional cropping. In the first row of Tab. (5b) we provide a strong baseline with 96.04% and 95.07% on GriTIS and S-TEDS, respectively, which nearly reaches the same performance as parsing from the cropped images under the interference of the document content around the table. Furthermore, we integrate both TD and TQ tasks in the form of multi-round dialogue, which endows TabPedia to directly parse all existing tables in a document image. We report the final result in the second row of Tab. [5b] These impressive results demonstrate that TabPedia has the potential to enable more holistic table understanding.
Evaluation on TQA. Due to the complex structure of tables and the dense text, the understanding of the table contents remains challenging issue. To thoroughly evaluate the performance of the understanding of table content and structure, we adopt two public benchmarks, i.e., WTQ [88] and TabFact (89), and our collected dataset ComTQA, as shown Tab. 6 On the WTQ and TabFact, TabPedia achieves promising performance among the open and close sources LVLMs. In contrast to existing benchmarks, TomTQA contains real-world table images with more complex questions. It is observed that current LVLMs show poor performance due to the incomplete understanding of real-world table structures. Compared with them. TabPedia achieves the optimal result with a notable margin, which demonstrates the effectiveness of jointly learning perception and comprehension tasks.
5.3 Qualitative Results
We further conduct qualitative evaluation on TabPedia's perception and comprehension capabilities. Firstly, we show the perception capability of TabPedia with solely TD and TSR tasks, as illustrated in the first row of Fig. 3 TabPedia accurately generates reliable and formatted results, which rendered to the original image for better observation. Secondly, TabPedia performs a complex task to directly parsing all table structure information in a document image by integrating instructions of TD and TQ tasks within a multi-round dialogue. As shown in the second row of Fig. 3 the example indicates that TabPedia is capable of exploring more holistic visual table understanding. In the last row, we display the table comprehensive capability of TabPedia. It is observed that the response not only contains concise and reliable answer, but also provides the specific contents in the table to support its answer. Especially, TabPedia even acquires certain math calculation ability to capture the connections among table contents, as shown in the bottom right example in Fig. 3 These results demonstrate Tabpedia's 's powerful multimodal comprehension capabilities. We also display more visualization results in the Appendix
5.4 Ablation Studies
In this section, we conduct ablation studies to validate the effectiveness of core settings and compo- nents in TabPedia. All experiments are conducted on three datasets across three tasks: PubTablM-
Det [9 FinTabNet [5] and WTQ [8)
Necessity of Meditative Tokens. In Tab. we conduct the experiment to investigate the impact of adding meditative tokens in TabPedia. It observed that adding meditative tokens significantly improves TabPedia's capabilities of table perception and comprehension.
What Information Matters for Meditative Tokens? We sample 100 test cases for each task and report the averaged numeric importance of high- and low-resolution vision tokens when they are attended by the meditative tokens for different tasks in the Tab. g Specifically, for the various VTU
8

---

The ublc soocturd asocention patuec
Hoc of the locaiion teble eloment pease: FL.85./F2..055...4444 (1.000 009900011105511
(0.566.154..292 198.0110 spoiing 15960.199..900 now hender (0004/220
colum (1877)) (54)59910.444)))
Rendeiing
stacued infommtioon
ocuatal pietar
Elppaay dimeat postions
the crepodd tablo dea
patere.
Here of (5015.0057 685 (ea. 3,300300000000 21122.6411144)3
colore UBS de 9.31% pudee)
(3.8640.0977) 91..0,0097..000000 91.2
colare 22730.697.00 (.351,0657
cokarn (0.301 100404444440000
Douriee specific anactore of latte in the righor (U ooT. 81667) miiiciioo
colarn (0.5500055000000 (57. 0.677
Daseribe the spccifi inusture rable in the rigion J1513, 0.445) of pioure.
arightil ploteee
The ubble arucura indersation is the oegion (aiae a8s 6677) the picnes lesol following prgecied beaks
The rale stramred information region (0.86,,0.9n, 0.4..888.5.2..6577 pictarc is loted to fillwiigg prspesad mos hoaker
Digtaa cloneni postionss piippim
Acconding lo the ulie, the chonge wrightd evasise prize foon weightad Si 2006
Whe now the change is wrightsd svrrage corerie pike from 2005 2006
decresad $36.45.
aed bzease compuoz of the song on fhe altura con todix No?
Decmase ohtht thooaadh Reesos: The weighted average exercise pioee lecrond
X
435:48
Via Rasen, From Be isage, see tuat Joel Houskss componeed unee the albure Con ladis which are thin Euili Comooll, Thnie Inseiie and Yoo Tool
far
fstam Bloes the
DooOwd
Figure 3: Qualitative results of TabPedia diverse tasks. The first row shows its perception capability on both TD and TSR tasks. The second row further exhibits TabPedia's 's powerful ability by employing multiple instructions of different tasks. The bottom row showcases TabPedia's accurate responses based on intricate contents in visual tables. Zoom in for best view.
tasks, we calculate the averaged attention scores (across all layers and attention heads) from the LLM decoder, which indicates the extent to which the meditative tokens focus on either high- or low-resolution visual tokens. For the TSR and TQ tasks, the meditative tokens pay significantly more attention to the high-resolution visual encoder tokens. We attribute this the fact that both tasks require more fine- fine-graines grained visual information to be "deliberated" in order to construct the dense table structure. In contrast, for the TD and TQA tasks, the two visual encoders contribute almost equally to the information attended to the meditative tokens, validating the importance of both vision encoders for these tasks.
Contributions of Different Tokens. In the Tab. 7
we calculate the averaged scores of the TabPedia- generated answers with respect meditative tokens, high-resolution visual tokens, and low-resolution sual tokens across all the attention maps from the L.M,, respectively. One can observe that the meditative to- kens contribute the most information to the generation of satisfactory answers, which demonstrates that the proposed meditative tokens are indispensable and ef- fective. We also provide a detailed analysis of the attention map of meditative tokens in Fig. D4 of Appendix. D
Table 7: Contributions of different tokens.
Meditative tokens
High-res visual tokens
Low-res visual tokens
Task
TD
0.65
0.16
0.19
TSR
0.64
0.12
0.24
TQ 0.71
0.11
0.19
TQA
0.56
0.18
0.25
Impact Dual Vision Encoders. As shown in Table we explore the impact of different vision encoders that capture global and local information from input images at various resolutions. The high- resolution encoder extracts intricate details from text-rich images, outperforming the low-resolution encoder, which struggles with nuanced visual representations in complex document images. Different

---

Table 9: Contribution of different visual
Table 8: Impact of meditative tokens in TabPedia.
kens from dual vision encoders.
meditative token
Pub
FinTabNet
WTQ
Task
High-res visual tokens Low-res visual tokens
Precision
S-TEDS
Acc
0.51
X
93.5
92.17
43.2
TSR
0.71
0.29
98.5
95.11
47.8
TQ
0.73
0.27
TQA
0.51
0.49
Table 10: Impact of different training strate- gies on low-resolution vision encoder.
Table 11: Impact of dual vision encoders.
High-Res Res Encoder
Low-Res Engoder
PubTabIM-Det
FinTabNet
WTQ
Low-Res
PubTabIM-Det
FinTabNet
WTQ
Precision
S-TEDS
Acc
Encoder
Precision
S-TEDS
Acc
96.5
93.6
44.9
frozen
95.11
47.8
V
86.2
81.3
24.7
unfrozen
95.11
V
98.5
95.11
47.8
tasks may require distinct visual cues, so dual vision encoders offer flexibility. For instance, TQA tasks need detailed table information, while TSR tasks depend on global layout. The low-resolution encoder provides comprehensive layout insights, complementing the high-resolution encoder's 's limited receptive field. Our results demonstrate that combining both encoders enhances the extraction of structural and content-relater details from tables, improving perception and comprehension tasks.
Frozen vs. Unfrozen Low-Resolution Vision Encoder. We further investigate different training strategies in terms of the low-resolution vision encoder. shown in Tab. 10 observed that no significant performance improvement but with longer training time consumption by unfreezing it, which in line with the conclusion in the pioneering work [003. Besides, we suppose the encoder frozen serve regularization, facilitating the extraction of layout information and alleviating potential overfitting problems, as well as more stable training. To strike the trade-off between computational consumption and performance, thus freeze the low-resolution vision encoder during training.
6 Limitation
In this section, we discuss the limitations of our TabPedia. Firstly, since we represent the table structure with regular rectangular boxes, TabPedia currently not capable of accurately parsing structural information for twisted distorted tables. Secondly, all images in TQA datasets, including
WTQ (8]) TabFact [89] and ComTQA are dominated by tables. Therefore, TabPedia still lacks the capability to directly answer the table question with original document image. In addition, compared to parallel decoding algorithms such as DETR (56) and Faster R-CNN [91), it consumes longer decoding time. Meantime, certain algorithmic designs such KV cache, flash attention, and hardware improvements can effectively improve inference efficiency. We believe that with the iterative development of large model technology, the inference efficiency of TabPedia can be significantly improved.
7 Conclusion
In this paper, propose novel large vision- model unify diverse visual table under- standing tasks, namely TabPedia. Specifically, we present a concept synergy mechanism to seamlessly integrate diverse tasks and multi-source visual tokens embedded from dual vision encoders as con- cepts. This mechanism is implemented by introducing the meditative tokens into the L.M.. Then, we fully leverage the capability of LLMs effectively understand these concepts and generate accurate and plausible responses. Extensive quantitative and qualitative experiments across various public benchmarks validate the effectiveness of our TabPedia. To further investigate the potential of TabPedia, we establish a challenging table VQA dataset, ComTQA, featuring round 9,000 QA pairs.
10

---

References
[1] Liangcai Gao, Yilun Huang, Herve Dejean, Jean-Luc Meunier, Qinqin Yan, Yu Fang, Florian Kleber, and Eva Lang. ledar 2019 competition on table detection and recognition (ctdar). In International Conference on Document Analysis and Recognition, 2019.
[2] Max Gobel, Tamir Hassan, Ermelinda Oro, and Giorgio Orsi. Icdar 2013 table competition. In International Conference on Document Analysis and Recognition, 2013.
[3] Devashish Prasad, Ayan Gadpal, Kshitij Kapadni, Manish Visave, and Kavita Sultanpure. Cascadetabnet: An approach for end to end table detection and structure recognition from image-based documents. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshop, pages 372-573, 2020.
[4] Sebastian Schreiber, Stefan Agne, Ivo Wolf, Andreas Dengel, and Sheraz Ahmed. Deepdesrt: Deep learning for detection and structure recognition of tables in document images. In international conference on document analysis and recognition, pages 1162-1167, 2017.
[5] Xinyi Zheng, Douglas Burdick, Lucian Popa, Xu Zhong, and Nancy Xin Ru Wang. Global table extractor (gte): framework for joint table identification and cell structure recognition using visual context. In Proceedings of the IEEE Winter Conference on Applications of Computer Vision, pages 697-706, 2021.
[6] Shoaib Ahmed Siddiqui, Muhammad Imran Malik, Stefan Agne, Andreas Dengel, and Sheraz Ahmed. DeCNT: Deep deformable enn for table detection. IEEE access, 6.74151-74161, 2018.
[7] Duc-Dung Nguyen. TableSegNet: a fully convolutional network for table detection and seg- mentation in document images. International Journal Document Analysis and Recognition, 25(1):1-14, 2022.
[8] Daqian Zhang. Ruibin Mao, Runting Guo, Yang Jiang, and Jing Zhu. YOLO-Table: disclosure document table detection with involution. International Journal on Document Analysis and Recognition, 26(1):1-14, 2023.
[9] Brandon Smock, Rohith Pesala, and Robin Abraham. PubTables- IM: Towards comprehensive table extraction from unstructured documents. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pages 4634 4642, 2022.
[10] Hao Feng, Qi Liu, Hao Liu, Wengang Zhou, Houqiang Li, and Can Huang. Docpedia: Unleashing the power of large multimodal model in the frequency domain for versatile document understanding. arXiv, 2023.
[1I] Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. Qwen technical report. arXiv, 2023.
[12] Yuliang Liu, Biao Yang, Qiang Liu, Zhang Li, Zhiyin Ma. Shuo Zhang, and Xiang Bai. Textmonkey: An ocr-free large multimodal model for understanding document. arXiv, 2024.
[13] Jiabo Ye, Anwen Hu, Haiyang Xu, Qinghao Ye, Ming Yan, Yuhao Dan, Chenlin Zhao, Guohai Xu, Chenliang Li, Junfeng Tian, et moLUG-DocOwl: Modularized multimodal large language model for document understanding. arXiv, 2023.
[14] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, al. An image is worth 16x16 words: Transformers for image recognition at scale. In Proceedings of the International Conference Learning Representations, 2020.
[15] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. International Conference on Machine Learning, pages 8748-8763 8763, 2021.

---

[16] Yuxin Fang, Wen Wang, Binhui Xie, Quan Sun, Ledell Wu, Xinggang Wang, Tiejun Huang, Xinlong Wang, and Yue Cao. Eva: Exploring the limits of masked visual representation learning at scale. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, pages
2023.
[17] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin Transformer: Hierarchical vision transformer using shifted windows. In Pro- ceedings of the IEEE International Conference on Computer Vision, pages 10012-10022, 2021.
[18] Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke 7ettlemoyer. BART: Denoising sequence-to-sequene pre-training for natural language generation, translation, and comprehension. In Annual Meeting of the Association for Computational Linguistics, pages 7871-7880, 2020.
[19] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1-67, 2020.
[20] Alec Radford, Karthik Narasimhan, Tim Salimans, lya Sutskever, et al. Improving language understanding by generative pre-training. 2018.
[21] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot leamers. In Proceedings of the Advances in neural information processing systems, pages 1877-1901, 2020.
[22] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi. Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv, 2023.
[23] Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson. Karel Lene, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: visual language model for few-shot learning. In Proceedings of the Advances in neural information processing systems, pages 23716 17716632766 2022.
[24] Xi Chen, Xiao Wang, Soravit Changpinyo, AJ Piergiovanni, Piotr Padlewski, Daniel Salz, Sebastian Goodman, Adam Gryener, Basil Mustafa, Lucas Beyer, et al. Pali: A jointly-scaled multilingual language-image model. arXiv, 2022.
[25] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-ima pre-training with frozen image encoders and large language models. arXiv, 2023.
[26] Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li. Pascale Fung, and Steven Hoi. InstructBLIP: Towards general-purpose vision-language models with instruction tuning. arXiv, 2023.
[27] Shaohan Huang, Li Dong, Wenhui Wang. Yaru Hao, Saksham Singhal, Shuming Ma, Tengchao Lv, Lei Cui, Owais Khan Mohammed, Qiang Liu, et al. Language is not all you need: Aligning perception with language models. arXiv, 2023.
[28] Zhiliang Peng, Wenhui Wang, Li Dong, Yaru Hao, Shaohan Huang, Shuming Ma, and Furu Wei. Kosmos-2: Grounding multimodal large language models the world. arXiv, 2023.
[29] Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. MiniGPT-4: Enhancing vision-language understanding with advanced large language models. arXiv, 2023.
[30] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36, 2024.
[31] Qinghao Ye, Haiyang Xu, Guohai Xu, Jiabo Ye, Ming Yan, Yiyang Zhou, Junyang Wang, Anwen Hu, Pengcheng Shi, Yaya Shi, et al. mPLUG-OWI: Modularization empowers large language models with multimodality. arXiv, 2023.
12

---

[32] Keqin Chen, Zhao Zhang, Weili Zeng, Richong Zhang, Feng Zhu, and Rui Zhao. Shikra: Unleashing multimodal 11m's ii's referential dialogue magic. arXiv, 2023.
[33] Bo Li, Yuanhan Zhang. Liangyu Chen, Jinghao Wang, Jingkang Yang, and Ziwei Liu. Otter: A multi-modal model with in-context instruction tuning. arXiv, 2023.
[34] Zhengyuan Yang, Linjie Li, Kevin Lin, Jianfeng Wang, Chung-Ching Lin, Zicheng Liu, and Lijuan Wang. The dawn of Imms: Preliminary explorations with gpt-4v (ision). arXiv, 9(1):1, 2023.
[35] OpenAL. Gpt-4 technical report, 2023.
[36]
Tom Brown, Benjamin Mann, Nick Ryder, Melanic Subbiah, Jared D Kaplan, Prafulla Dhari- wal, Arvind Neclakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. In Proceedings of the Advances in neural information processing systems, 2020.
[37] Rohan Anil, Andrew M Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, et al. Palm technical report. arXiv, 2023.
[38] Peng Gao, Jiaming Han, Renrui Zhang, Ziyi Lin, Shijic Geng, Aojun Zhou, Wei Zhang, Pan Lu, Conghui He, Xiangyu Yue, et al. Llama-adapter v2: Parameter-efficient visual instruction model. arXiv, 2023.
[39] Qwen. Introducing qwen-7b: Open foundation and buman-aligned models (of the state-of-the- arts), 2023. URL httph: //op::://itte tithub..t...... com Wwwen////wenenn
[40] Anwen Hu, Haiyang Xu, Jiabo Ye, Ming Yan, Liang Zhang, Bo Zhang, Chen Li, Ji Zhang, Qin Jin, Fei Huang, et al. mPLUG-DocOnn 1.5: Unified structure learning for ocr-free document understanding. arXiv, 2024.
[41] Jianqiang Wan, Sibo Song, Wenwen Yu, Yuliang Liu, Wenqing Cheng, Fei Huang, Xiang Bai, Cong Yao, and Zhibo Yang. OmniParser: A unified framework for text spotting, key information extraction and table recognition. arXiv, 2024.
[42] Sueng Peng. Seongmin Lec, Xiaojing Wang, Rajarajeswari Balasubramaniyan, and Duen Horng Chau. UniTable: Towards a unified framework for table structure recognition via self-supervised pretraining. arXiv, 2024.
[43] Geewook Kim, Teakgyu Hong. Moonbin Yim, GcongYeon Nam, Jinyoung Park, Jinyeong Yim, Wonseok Hwang, Sangdoo Yun, Dongyoon Han, and Seunghyun Park. OCR-free document understanding transformer. In Proceedings of the European Conference on Computer Vision, pages 1978-51,47 2022.
[44] Thomas Kieninger and Andreas Dengel. The t-rees table recognition and analysis system. In International Association on Pattern Recognition, pages 255-270, 1999.
[45] Basilios Gatos, Dimitrios Danatsas, Ioannis Pratikakis, and Stavros Perantonis. Automatic table detection in document images. International Conference on Advances Pattern Recognition, pages 609-618, 2005.
[46] Gaurav Harit and Anukriti Bansal. Table detection in document images using header and trailer patterns. In Proceedings of the Eighth Indian Conference on Computer Vision, Graphics and Image Processing, pages 1-8, 2012.
[47] Nguyen D Vo, Khanh Nguyen, Tam V Nguyen, and Khang Nguyen. Ensemble of deep object detectors for page object detection. In Proceedings of the International Conference on Ubiquitous Information Management and Communication, pages 1-6, 2018.
[48] Azka Gilani, Shah Rukh Qasim, Imran Malik, and Faisal Shafait. Table detection using deep learning. In international conference on document analysis and recognition, volume 1, pages 771-776 2017.
13

---

[49] Yilun Huang, Qinqin Yan, Yibo Li, Yifan Chen, Xiong Wang, Liangcai Gao, and Zhi Tang. A yolo-based table detection method. In International Conference on Document Analysis and Recognition, pages 813-818, 2019.
[50] Devashish Prasad, Ayan Gadpal, Kshitij Kapadni, Manish Visave, and Kavita Sultanpure. CabNe:eTabNet: An approach for end to end table detection and structure recognition from image-based documents. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshop, pages 572-573, 2020.
[51] Madhay Agarwal, Ajoy Mondal, and CV Jawahar. Cdec-net: Composite deformable cascade network for table detection in document images. In international conference on pattern recognition, pages 1491-9498, 2021.
[52] Ningning Sun, Yuanping Zhu, and Xiaoming Hu. Faster r-cnn based table detection combining corner locating. In international conference on document analysis and recognition, pages 1314-1319, 2019.
[53] Pau Riba, Anjan Dutta, Lutz Goldmann, Alicia Fornés, Oriol Ramos, and Josep Lladós. Table detection in invoice documents by graph neural networks. In International Conference on Document Analysis and Recognition, pages 122-127, 2019.
[54] Martin Holeček, Antonín Hoskovec, Petr Baudiš, and Pavel Klinger. Table understanding in structured documents. International Conference on Document Analysis and Recognition Workshops, pages 158-164,22 2019.
[55] Minghao Li, Yiheng Xu, Lei Cui, Shaohan Huang, Furu Wei, Zhoujun Li, and Ming Zhou. Docbank: A benchmark dataset for document layout analysis. In Annual Meeting of the Association for Computational Linguistics, pages 949-960, 2020.
[56] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In Proceedings of the European Conference on Computer Vision, pages 213-229, 2020.
[57] Zengyuan Guo, Yuechen Yu, Pengyuan Lv, Chengquan Zhang, Haojie Li, Zhihui Wang, Kun Yao, Jingtuo Liu, and Jingdong Wang. Trust: an accurate and end-to-end table structure recognizer using splitting-based transformers. arXiv, 2022.
[58] Weihong Lin, Zheng Sun, Chixiang Ma, Mingze Li, Jiawei Wang, Lei Sun, and Qiang Huo. TSRFormer: Table structure recognition with transformers. In Proceedings of the 30th ACM International Conference on Multimedia, pages 6473-6482 2022.
[59] Hao Liu, Xin Li, Mingming Gong, Bing Liu, Yunfei Wu, Deqiang Jiang, Yinsong Liu, and Xing Sun. Grab what you need: Rethinking complex table structure recognition with flexible components deliberation. In Proceedings of the AAAI Conference on Artificial Intelligence. pages 3600-3611, 2024.
[60] Jingqun Tang, Wenming Qian, Luchuan Song, Xiena Dong, Lan Li, and Xiang Bai. Optimal boxes: boosting end-to-end scene text recognition by adjusting annotated bounding boxes via reinforcement learning. In Proceedings of the European Conference on Computer Vision, pages 233 228,2248111 2022.
[61] Shubham Singh Paliwal, D Vishwanath, Rohit Rahul, Monika Sharma, and Lovekesh Vig. TableNet: Deep learning model for end-to-end table detection and tabular data extraction from scanned document images. In International Conference on Document Analysis and Recognition, pages 128-133 133, 2019.
[62] Hao Liu, Xin Li, Bing Liu. Deqiang Jiang, Yinsong Liu, Bo Ren, and Rongrong Ji. Show, read and reason. In Proceedings of the ACM International Conference on Multimedia, 2021.
[63] Zewen Chi, Heyan Huang, Heng-Da Xu, Houjin Yu, Wanxuan Yin, and Xian-Ling Mao. Complicated table structure recognition. arXiv, 2019.
14

---

[64] Hao Liu, Xin Li, Bing Liu, Deqiang Jiang, Yinsong Liu, and Bo Ren. Neural collaborative graph machines for table structure recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4533-4542, 2022.
[65] Xu Zhong, Elaheh ShafieiBavani, and Antonio Jimeno Yepes. Image-based table recognition: data, model, and evaluation. In Proceedings of the European Conference on Computer Vision, pages 564-580, 2020.
[66] Wii-Lin Chiang, Zhuoban Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng. Siyuan Zhuang. Yonghao Zhuang, Joseph E Gonzalez, et al. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality. See https://vicun. Imsys. org (accessed 14 April 2023), 2(3); 2023.
[67] Zhen Zhao, Jingqun Tang, Binghong Wu, Chunhui Lin, Shu Wei, Hao Liu, Xin Tan, Zhizhong Zhang, Can Huang, and Yuan Xie. Harmonizing visual text comprehension and generation. arXiv, 2024.
[68] Jingqun Tang, Qi Liu, Yongjie Ye, Jinghui Lu, Shu Wei, Chunhui Lin, Wanqing Li, Mohamad Fitri Faiz Bin Mahmood, Hao Feng, Zhen Zhao, et al. MTVQA: Benchmarking multilingual text-centric visual question answering. arXiv, 2024.
[69] Jingqun Tang, Chunhui Lin, Zhen Zhao, Shu Wei, Binghong Wu, Qi Liu, Hao Feng, Yang Li, Siqi Wang, Lei Liao, et TextSquare: Scaling up text-centric visual instruction tuning. arXiv, 2024.
[70] Jinghui Lu, Haiyang Yu, Yanjie Wang, Yongjie Ye, Jingqun Tang, Ziwei Yang. Binghong Wu, Qi Liu, Hao Feng, Han Wang, et al. A bounding box is worth one token: Interleaving layout and text in a large language model for document understanding. arXiv, 2024.
[71] Zhen Zhao, Jingqun Tang, Chunhui Lin, Binghong Wu, Can Huang. Hao Liu, Xin Tan, Zhizhong Zhang, and Yuan Xie. Multi-modal in-context learning makes an ego-evolving scene text recognizer. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 15567-15376, 2024.
[72] Hao Feng, Zijian Wang, Jingqun Tang. Jinghui Lu, Wengang Zhou, Houqiang Li, and Can Huang. Unidoc: A universal large multimodal model for simultaneous text detection, recogni- tion, spotting and understanding. arXiv, 2023.
[73] Haoran Wei, Lingyu Kong, Jinyue Chen, Liang Zhao, Zheng Ge, Jinrong Yang, Jianjian Sun, Chunrui Han, and Xiangyu Zhang. Vary: Scaling up the vision vocabulary for large vision-language models. arXiv, 2023.
[74] Masato Fujitake. LayoutLLM: Large language model instruction tuning for visually rich document understanding. arXiv, 2024.
[75] Tao Ge, Hu Jing, Lei Wang, Xun Wang, Si-Qing Chen, and Furu Wei. In-context autoencoder for context compression in a large language model. In Proceedings of the International Conference on Learning Representations, 2023.
[76] Fuzhao Xue, Valerii Likhosherstov, Anurag Arnab, Neil Houlsby, Mostafa Dehghani, and Yang You. Adaptive computation with elastic input sequence. In International Conference on Machine Learning, pages 38971-38988 2023.
[77] Mikhail S Burtsev, Yuri Kuratov, Anton Peganov, and Grigory V Sapunov. Memory transformer. arXiv, 2020.
[78] Aydar Bulatov, Yury Kuratov, and Mikhail Burtsev. Recurrent memory transformer. Proceed- ings of the Advances neural information processing systems, 35:1 35:11079-11091, 2022.
[79] Timothée Darcet, Maxime Oquab, Julien Mairal, and Piotr Bojanowski. Vision transformers need registers. In Proceedings of the International Conference on Learning Representations, 2023.
15

---

[80] Sachin Goyal, Ziwei Ji, Ankit Singh Rawat, Aditya Krishna Menon, Sanjiv Kumar, and Vaishnavh Nagarajan. Think before you speak: Training language models with pause tokens. arXiv, 2023.
[81] Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy V Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel HAZIZA, Francisco Massa, Alaaeldin El-Nouby, et al. DINOv2: Learning robust visual features without supervision. Transactions Machine Learning Research, 2023.
[82] Kenton Lee, Mandar Joshi, Iulia Raluca Ture, Hexiang Hu, Fangyu Liu, Julian Martin Eisen- schlos, Urvashi Khandelwal, Peter Shaw, Ming-We Wei Chang, and Kristina Toutanova. Pix2struct: Screenshot parsing pretraining for visual language understanding. In International Confer- ence on Machine Learning, pages 18992-18912, 2023.
[83] Jiabo Ye, Anwen Hu, Haiyang Xu, Qinghao Ye, Ming Yan, Guohai Xu, Chenliang Li, Junfeng Tian, Qi Qian, Ji Zhang, al. UReader: Universal ocr-free visually-situated language understanding with multimodal large language model. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 2841-2858. 2023.
[84] Minghui Liao, Zhaoyi Wan, Cong Yao, Kai Chen, and Xiang Bai. Real-time scene text detection with differentiable binarization. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 11474-11481, 2020.
[85] Kai Wang, Boris Babenko, and Serge Belongie. End-to-end scene text recognition. In Proceedings of the IEEE International Conference on Computer Vision, pages 1457-1464, 2011.
[86] Xuebo Liu, Ding Liang, Shi Yan, Dagui Chen, Yu Qiao, and Junjie Yan. Fots: Fast oriented text spotting with a unified network. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 5676-5685, 2018
[87] MD Zakir Hossain, Ferdous Sohel, Mohd Fairuz Shiratuddin, and Hamid Laga. A compre- hensive survey of deep learning for image captioning. ACM Computing Surveys, 51(6):1-36 2019.
[88] Panupong Pasupat and Percy Liang. Compositional semantic parsing on semi-structured tables. In Annual Meeting of the Association for Computational Linguistics, pages 1470-1480 2015.
[89] Wenhu Chen, Hongmin Wang, Jianshu Chen, Yunkai Zhang, Hong Wang, Shiyang Li, Xiyou Zhou, and William Yang Wang. TabFact: A large-scale dataset for table-based fact verification. In Proceedings of the International Conference on Learning Representations, 2019.
[90] Ross Girshick. Fast R-CNN. In Proceedings of the IEEE International Conference on Computer Vision, pages 1440-1440 2015.
[91] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster R-CNN: Towards real- time object detection with region proposal networks. Proceedings of the Advances in neural information processing systems, 28, 2015.
[92] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask R-CNN. In Proceedings of the IEEE International Conference on Computer Vision, pages 2961-2966 2017.
[93] Machel Reid, Nikolay Savinov, Denis Teplyashin, Dmitry Lepikhin, Timothy I. Lillicrap, Jean- baptiste Alayrac. Radu Soricut, Angeliki Lazaridou, Orhan Firat, Julian Schrittwieser, et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv, 2024.
[94] Leslie N Smith and Nicholay Topin. Super-convergence: Very fast training of neural networks using large learning rates. In Artificial intelligence and machine learning for multi-domain operations applications, pages 369-386. 2019.
[95] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In Proceedings of the International Conference on Learning Representations, 2018.
16

---

[96] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et PyTorch: An impera- tive style, nigh-performance deep learning library. Proceedings of the Advances in neural information processing systems, 32, 2019.
[97] Brandon Smock, Rohith Pesala, and Robin Abraham. GriTS: Grid table similarity metric for table structure recognition. In International Conference Document Analysis and Recognition, pages 535-549, 2023.
[98] Yuliang Liu, Zhang Li, Hongliang Li, Wenwen Yu, Mingxin Huang, Dezhi Peng, Mingyu Liu, Mingrui Chen, Chunyuan Li, Lianwen Jin, et al. On the hidden mystery of ocr in large multimodal models. arXiv, 2023.
[99] Zhang Li, Biao Yang. Qiang Liu, Zhiyin Ma, Shuo Zhang, Jingxu Yang, Yabo Sun, Yuliang Liu, and Xiang Bai. Monkey: Image resolution and text label are important things for large multi-modal models. arXiv, 2023.
[100] Wenyi Hong, Weihan Wang, Qingsong Lv, Jiazheng Xu, Wenmeng Yu, Junhui Ji, Yan Wang, Zihan Wang, Yuxiao Dong, Ming Ding, et al. Cogagent: visual language model for gui agents, arXiv, 2023.
[101]
GPT-4V(ision) system card. 2023.
[102] Xiaoyi Dong, Pan Zhang, Yuhang Zang, Yuhang Cao, Bin Wang, Linke Ouyang, Xilin Wei, Songyang Zhang, Haodong Duan, Maosong Cao, et al. free-form text-image composition and comprehension vision-language large model. arXiv preprint 240:v22401.16420, 2024.
Mastering
[103] Xiaohu Huang, Hao Zhou, Kun Yao, and Kai Han. Froster: Frozen clip is strong teacher for vocabuacyy action recognition. Proceedings of the International Conference on Learning Representations, 2024.
17

---

A More details about TQA datasets
A.1 QA Pairs Generation
We depict the procedure of collecting QA pairs with an example in Fig. Al, For input image, Gemini Pro [93] is prompted first recognize the table structure with OCR results in the image. then generate several question and answer pairs according to OCR results. In order to improve the reliability of the generated answers, we leverage various prompting techniques, i.e, Chain-of-Thought and few-shot prompting. According to the specific prompt, Gemini Pro will generate multiple QA pairs for each input image and return them in an agreed- upon format. After obtaining raw responses generated by Gemini Pro, we utilize the regularized matching algorithm and the special character filter in turn to extract available question and answer pairs.
First recognize
results image, then plasse genentte several distinet question and auwerr pain.
smseture
QUESTION
torm and
abligations is FANNSWER (85,543,003 WiSQUESTTN
the quesion answer pairs should sordfy-
What is the amounnt of short-ttre debe NANSERA
2. The questions and the answers should be much details as possible.
1. STRICTLY Y related is the abile stucture inchuling DCR roslls
Generate
58 58,676600000 win
Gemini
d'applicuble, Bouugh pattur..
should complenented "chais-of"
QUESTION 9 Which year has the losed smount of llong-em dehi and lease obigations"nANN WWRSS 201 has the least amount of long- tomm debt and leuse obligations, with 585, $40,000
questions should
least reasening cuestion.
5. fomnut should soietly Fdllow: QUESTIONI
XXXXERXXXXX QUESTION2: XXXX ANSWERXXXXX
Filter
2011
Q: What is the unount of long- tern deh and lasse obligations 2011? A:585, 54,000
Q: What is the trand of income from 2011? A: uend inserest incume from 2009 2011 increasing.
Q Which year has the least smount of term dehi and lavse obligstion?? 2011 the least smount term
$40.541
511,141
(1.144
(2.1.5)
(2:47
...
96.50
(2.997
(1.70)
1977
and lease obligation, with $85, 543 1000
900 041
Figure A1: The illustration of an example for generating QA pairs with the powerful LVLM, Gemini Pro 93). The prompt includes several key rules to ensure the response quality as much as possible.
A.2 ComTQA Benchmark
In Tab. Al we present the distribution of both data sources 15] 9 within the ComTQA dataset. Concretely, ComTQA comprises a total of 9,070 QA pairs across 1,591 images, averaging 5 questions per image. Different from existing TQA benchmarks [ا 89 ComTQA contains more complex table questions in real-world table images to assess the robustness of various models. As shown in Fig. A2 we showcase several representative examples, including multiple answers, mathematical calculation and logical inference, which are the question types lacking in previous benchmarks. To this end, we hope that ComTQA could fill this gap and serve as a reasonable benchmark for community development.
Fatss on cedeliaiaaionn The Mibrral
Table A1: Statistics of ComTQA benchmark.
PubTabiM
Fin
Total
#images
932
659
1,591
Q: Which beta value has the highest classification accuracy?
#QA pairs
6,232
2,838
9,070
A: 0.4 in 0.6 0.8 in 1.0
Avg. per image
6
5
(a) multiple answers
Take Miservad vollect Wilk
JTT
424
678
2520
6.20
Q: What is the sum of the calibration factors for the three models?
Q: Which model predicts the largest volume at time 72? A: Richards
A: 3,8018
(b) mutheraatical calculation
(c) logical inference
Figure A2: More visualization on ComTQA benchmark. We display several complex QA types, such as multiple answers, mathematical calculation and logical inference. Zoom in for best view.
18

---

B Annotation in TSR task
We illustrate the object classes utilized in TSR and TQ tasks shown in Fig. B3 A table generally is composed of five basic elements, i.c., column, row, spanning cell, column header and projected row header. "Row" denotes the rectangular boxes of each row's content in the table, while "Column" denotes the rectangular boxes of each column's content. The area where each row and each column intersect represents the table cell. Besides these both most common table elements, "Column header" refers to the area in the table that contains the data type content for each column, usually occupying multiple rows the top of the table. "Projected row header". as a special row, represents the area that contains single non-blank cell in row. "Spanning cell" refers to cell in table that spans multiple rows or columns. According to these definitions, these objects have implicit relationship and construct table's s hierarchical structure through physically overlapped rectangle boxes
CYPEAT MY
pattippantt
genotype
Papents
OR 195% CP
OR 95%
Patental
OR 196% CIR
Arican
Amenican
<0.430
Non:M1
88/75
Reterent
46/51
Reterent
20/24
Reterent
20:430
Non M1
75/67
(.9--.)))
21/18
(.9--.)))
54/51
1.0 (055-23)
100.430
42/46
-1.0:0.8-..11
25/33
1.2:00.-2221
7/13
0.810.2-1.8
-0.430
Any
59/54
(0.8-25))
17/14
1.7 (0.8-..))
42440
1.0 (055-23)
IOR 195% Oi
0.0.0...000
0.312211:.
0410513
Whitees
Non
174/133
Reterant
118/83
Reterent
58/50
00.349
(0.5-1.0)
38/43
(0.4-1.1)
84/105
(0.5-1.3)
<00.340
Any M1
0.8:105-11:
29/31
0.7:0.4-12.
11-10.5-..
20,349
Any
(0.4-1.4
11/40
(0.3-19))
(0.4-1.7)
105%
0810312
00111101
Row (add)
Column (odd)
Spanning cell Column
Projected row header
(even)
(even)
Figure B3: The illustration of an example table with dilated bounding box annotations for different object classes for modeling table structure recognition.
C Broader Impact
Our proposed model targets to unify multiple visual form comprehension tasks. This technology could help more people with visual impairments access tabular data through cooperating with improved screen readers and other assistive technologies. Moreover, automating table understanding technology could reduce the need for time-consuming manual data entry and correction, freeing up human resources for more complex and creative tasks. To be honest, this technology also brings some negative societal impacts. As more table data is extracted and processed with automatic visual table understanding, there is heightened risk of sensitive information being mishandled exposed. It is crucial to ensure robust data privacy measures.
D
More Qualitative Results
Results in in-the-wild cases. For better investigating the generalization of our proposed TabPedia, randomly select some document images from a document website and illustrate the generation results in Fig. [5 For perception and comprehension tasks, TabPedia generates accurate and reasonable responses in TD, TSR and TQA tasks, which sufficiently proves the robustness of our method for visual table understanding.
Attention map of meditative tokens. order to analyze the information extraction of meditative tokens for different tasks, visualized the attention maps of meditative tokens for input instructions
19

---

with different granularity of visual feature tokens, as shown in Fig. D4 For each task, we select the shallow and deep four-layer attention maps in the LLM for visualization, respectively. The y-axis represents the meditative tokens, while the x-axis represents the sequence of instruction tokens and different granular visual tokens. For perceptive tasks, meditative tokens are densely attentive to most of the input information in the shallow layers, while they showcase diverse attention regions in the deeper layers. This phenomenon illustrates that meditative tokens could adaptively capture task-related information with respect to diverse tasks. For the comprehension task (TQA), meditative tokens show a different attention pattern from perception tasks, which maintain sparse attention with input tokens in the shallow layers. These results validate that our proposed meditative tokens adaptively enable different regions of visual tokens and understand the intention specific task questions.
Map 24)
Highte
High
Layer
Layer 29
Layer
Layer 30
Layer
Layer 31
Layer
Layer 52
Tiille Structure Recogniticn Attention Map (Head 24)
Layer
Layer
Layer
Layer
Layer
Layer 4
Layer 32
Table Query Atlention (Heul
Len-dor
Low-loo
High-to
Layer
Layer
Layer
Layer 30
Layer
Layer
Layer
Layer
Question Answering
Map (Ilead
Low Ros
Low-Bu
Layer 1
Layer 29
Layer
Layer 30
Layer 3
Layer 31
Layer 4
Layer
Figure D4: Visualization of attention maps between meditative tokens and the sequence of instruction and visual tokens. "Q", "Low-Res" and "High-Res" denote the instruction tokens, global visual tokens and local visual tokens, respectively. Y-axis denotes the meditative tokens. Zoom for best view.
20

---

Visualization of TabPedia's responses. As shown in Tab. D2 we introducing meditative tokens can bring promising performance across VTU tasks. We compare in detail the differences in the generated results before and after the introduction of Meditative in different VTU tasks. It is observed that introducing meditative tokens mainly improves the quality of long-form responses. Also for the perception tasks including TD and TSR, introducing meditative tokens can alleviate the meaningless or repetitive word generation. For the comprehension task, TQA, introducing meditative tokens can generate more elaborated and reasonable response. As suggested, we showcase several samples for better understanding.
Table D2: Qualitative results of TabPedia's 's responses.
Image
Question
Tekens)
what is the las ste's sunsipaity?
"Pokorama""
"Pricenks in Reaon: The list site a Tachilaaaa, and is miniipaity Frlereka"
"Pukoraana 'nRrescn: it sloos the last row the tahv lest sike municcppllity Pulay
john leizes and janes phllip both drive which ope of car?
"Honda Prciude Chewoler"
"Manaa RXC-7 la Reasns: From the imrege we ron nee that bait
"Honda Prelude Chevoole Rovent From the imver. wn cas see dual betl. John Beiass
John Brigas and Junes Phillipps crow Mards
lames Phillip
Henda
No Prelude Chevoslet, while James Philig diove the No. 10 Honds Pretude Caroniet. So they boll drove same typo of cor. which Handa Pielude Chronlaa
Ourput the tible
"Here is list of all dae
"Here list of the sabk element
"Hose is litt the
dleneal lanadions
locaisune ckemam
banaiome clemees
photo
patueeta
picture in 140.0-422.0.00 JI A-0.0.17pe [OLONLL 207. 477,0-444.3662 (O.3 1OUD 156. MON R21]"
[0.005 A. (3,2713
1590..28. 229/a 199.5,..09992.......0.
10
DUSEROO MS 207..0 301
21

---

(a) In-the-wild cases on TD task
17111
211113
1.8
1A
07160
25/
1148
Midec
0.002 (hagg)
w.200.50 Teati/
Malioy (21
331
06.3
001
16.1
076
2.35
648
639
Bloe Rou3-0 211 022
2317
41.
43.1
085
43.7
069
083
Methed
Ardh
Data
That
wil
Tlad.
V2
Machod
Arch.
Dnt
Heal
CLIP
917-5-114
W .001
71-8
84.3
86.
15.0
PLP
VIT
RTT BoC
CLIP
WCZ 4004
86.5
86.3
08.3
15.0
WTT
$2.8
EWNG
QparCLIP
V12-38/14
LABON-0B
VIT-
LARON
VC2-0:234
LA30N-25
85.2
88.2
17.2
LARON-ID
KVA-DLIPP
VIT-624
43.8
09.8
89.3
17.4
861
19.3
17.5
(b) In-the-wild cases
TSR task
Tustaex
YoNl
1994
YUAL
Q: Answer the question and give reason. what's the point Flamingo-808 VQAv2? A: 56.3 Reason: Flamingo-808 is model in VQAv2. Its VQAv2 score is shown in the second column, which is 56.3.
OLIS (Naoa 120 2371
003
Qula (Varmm 1301
77.4
412
79.4
LLAWAL (Mluuu-770
55.1
23.5
Ceaa-12-Cha
213
Q: Answer the question and give reason. How many datasets utilized in the Conversation task?
Makk
Rranpiss
Jajaget
Captioning
3Rlk
Caramn [2]] TearCape [126]
VQ4x2 (54), OKVQA [101) A-OKVQA (122)
VQA
1.1M
laarOA 199), ADD (?1), DQA [64]
OCR-VQA [107] CharQQ (155] DeeVQA (20),
OCR
24K
ST-VA (12), EST-VQA (150]] Inf (106).
There are 5 datasets utilized in the Conversation task, which are LLaVA- 150K, SVIT, VisDial, LLaVA-Mix-665K and LRV-Instruction.
A: Reason:
LaaVA [182]
Guoudding
123K
Ra/C0C04ll [103, Tolla (140]
Toloka
Grousdal Cap.
264K
Rc/CoCo--ly [103, 170]
Canseratioon
14M
LL8K 1921, SVT
L.V-Inscraciio (90)] LLAVA, MBSK [91]
SVIT (183), Viaial [36]
(c) In-the-wild cases on TQA task
Figure D5: Qualitative results of TabPedia in-the-wild cases. TabPedia achieves impressive performance these unseen images, which validates its robustness and generalization. Zoom for best view.
22

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The main claims could be found in the abstract of the main paper.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitations of our work in Sec. 6
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
23

---

Justification:
Our paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification:
We elaborate the experimental details in Sec. 5
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
24

---

Answer: [No]
Justification: [TODO]
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cm
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ ccpppp.ccppulli//guuii wwwiii/gguie///eeeeeeoecc CoodeSubeiiieeee ssi wswonnolli onnolicy)))) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We have provide the training and test details in Sec.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: [TODO]
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
25

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We conduct all experiments in 16 NVIDIA A100 GPUs.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g.. preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethies
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: htapp::///euripssse httpp//s....cuuooaccaaasooooooo.
Answer: [Yes]
Justification: [TODO]
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss the broader impact in the Appendix C
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to
26

---

generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative socictal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: [TODO]
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: [TODO]
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, wwppppswpptseeee whworee.wthhhe....c.. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's is creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and the documentation provided alongside the assets?
27

---

Answer: [Yes]
Justification: [TODO]
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/coddemoodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: [TODO]
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: [TODO]
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
28


Reviews:
[{"review_id": "fBYe5FqEIV", "reviewer": "Reviewer_Xi9r", "summary": "This paper introduces TabPedia, a novel large vision-language model designed to address the challenges in visual table understanding (VTU) tasks. TabPedia incorporates a concept synergy mechanism that treats various VTU tasks and multi-source visual embeddings as concepts within a unified framework, allowing for the seamless integration of tasks such as table detection, structure recognition, querying, and question answering by utilizing the capabilities of large language models (LLMs). A new comprehensive table visual question answering benchmark called ComTQA is created, which includes approximately 9,000 question-answer pairs. Extensive experiments on both table perception and comprehension tasks across various public benchmarks demonstrate the effectiveness of TabPedia.", "questions": "See weaknesses.", "limitations": "Yes.", "rating": 3, "confidence": 4, "soundness": 3, "presentation": 1, "contribution": 2, "strengths": "This paper presents very detailed experiments on the VTU tasks and achieves good results.\n\nIt is meaningful to combine all VTU tasks into the same framework.", "weaknesses": "The novelty of TabPedia seems to be minimal, as existing VLM models are almost similarly structured.\n\nAlthough the authors present the Attention map of meditative tokens in the Appendix, I still don't understand the reason why meditative tokens work. \n\nDifferent vision encoders are introduced. But I don’t know how they help each other and what they extract. \n\nIn addition, there are many places where the author doesn't explain things clearly. For example, \n\nL177 why the low-resolution vision encoder is not trained?\n\nL207 “To better understanding, we display a representative sample in Appendix B.” After reading the appendix I still don't understand how the authors constructed the TSR data. By the way, is there any grammatical mistake in \"To better understanding\"?\n\nL248 “temperature parameter” is also not explained.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the detailed comments and acknowledgment of our contributions. We provide the responses as follows.\n\n*  **Q1:** The novelty of TabPedia seems to be minimal, as existing VLM models are almost similarly structured.  \n    **A1:** We would like to re-emphasize the main novelty, which has been confirmed by Reviewer #bkeN and #fN3K, lies in the unified framework integrating various VTU tasks with \"impressive\" performance achieved.  We do agree our TabPedia is built upon the canonical \"Vision Encoder + Projection + LLM\" paradigm, however, our main focus is to explore the synergistic effects of diverse VTU tasks and multi-source visual embeddings through the proposed concept synergy mechanism, of which the indispensability has been verified by the qualitative and quantitive experimental results given in the paper (see Tab.7 and Fig.D5). Actually, we have performed further explorative experiments by infusing the proposed concept synergy mechanism into another powerful VLM, QWEN VL, and surprisingly found the better performance. To put it another way, our concept synergy mechanism is a versatile one, which can be readily applied to most existing VLMs.\n\n* **Q2:** Although the authors present the Attention map of meditative tokens in the Appendix, I still don't understand the reason why meditative tokens work.  \n  **A2:** The most intuitive motivation behind our meditative tokens is from the success of \"additional tokens\" (see Sec. 2.3 of main text), where the input sequence is extended with these additional tokens popularized for various intentions, such as extracting task-specific information, providing extra information or improving model performance. Inspired by it, our proposed meditative tokens serve as the informative buffer to adaptively integrate different partial visual tokens and understand the intentions of specific task questions in visual table understanding. As illustrated in Figure D5, the meditative tokens can adaptively capture task-related visual features with respect to diverse tasks.\n\n* **Q3:** Different vision encoders are introduced. But I don’t know how they help each other and what they extract. Why the low-resolution vision encoder is not trained?  \n  **A3:** We equip our TabPedia with dual vision encoders to effectively extract visual information. For the low-resolution vision encoder, we utilize the one from the CLIP vision encoder (ViT-L), which has been pre-trained on 400 million image-text pairs sourced from the open-world data, thereby embedding extensive world knowledge into its pretrained weights. To preserve its generalization ability, we keep it frozen during the whole training procedure. By performing the comparative experiments (see the following table), we observe no significant performance improvement but with longer training time consumption by unfreezing it, which is in line with the conclusion in the pioneering work [1]. Besides, we suppose the encoder frozen can serve as a regularization, facilitating the extraction of layout information and alleviating potential overfitting problems, as well as more stable training. However, ViT-L is constrained by its limited ability to capture nuanced visual representations from high-resolution document images with intricate textual content and dense table structures contained. As various tasks may require different vision clues from either vision encoders, the dual vision encoders are expected work flexibly for various tasks (TQA often requires detailed table information while global layout matters for TSR task), which also triggers our meditative tokens. To strike the trade-off between computational consumption and performance, we thus freeze the low-resolution vision encoder during training. This explanation will be further elaborated upon in our revised manuscript.\n\n  | Exp Setting   | PubTab1M-Det  | FinTabNet | WTQ |\n  |-------------|----------------|----------------|----------------|\n  | low-res enc (frozen) + high-res enc (unfrozen) | **98.5** | 95.11 | **47.8** |\n  | low-res enc (unfrozen) + high-res enc (unfrozen) | 98.4 | **95.62** | 46.4 |\n\n  [1] Huang, Xiaohu, et al. \"Froster: Frozen clip is a strong teacher for open-vocabulary action recognition.\" In ICLR 2024.\n\n* **Q4:** “To better understanding, we display a representative sample in Appendix B.” After reading the appendix I still don't understand how the authors constructed the TSR data. By the way, is there any grammatical mistake in \"To better understanding\"?\n\n  **A4:** Please refer to the official comment. For all the typos, we will thoroughly revise them in the future version.\n\n* **Q5:** \"temperature parameter\" is also not explained.  \n  **A5:** To our best knowledge, \"temperature\" is a basic concept in Machine Learning field [1,2], which has emerged long before deep learning era. Considering as a submission of the conference in the field, we thus assume all the readers have this background. In the context of Language Model (LM), specifically Large Language Models (LLMs), the temperature parameter is used to control the randomness or uncertainty in the output generated by the model. A small temperature value sharpens the probability distribution. This means that the most likely tokens are given even higher probabilities, while less likely tokens receive lower probabilities. This leads to more deterministic and less diverse outputs. \n\n  [1] https://en.wikipedia.org/wiki/Boltzmann_distribution  \n  [2] Bishop C M, Nasrabadi N M. Pattern recognition and machine learning[M]. New York: springer, 2006.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "iscNn8bfkX", "reviewer": "Reviewer_bkeN", "summary": "This paper proposes TabPedia, a novel large-scale vision-language model designed for comprehensive visual table understanding (VTU) within a unified framework.\nIt addresses the challenge of modal isolation and task exclusivity by proposing a concept synergy mechanism that treats diverse VTU tasks and multi-source visual embeddings as interconnected concepts. \nThis framework integrates various table tasks with the powerful capabilities of large language models. \nExtensive experiments are conducted to validate the effectiveness of TabPedia.\nA new TQA dataset ComTQA is established for better evaluating the VTU task in real-world scenarios.", "questions": "Please refer to the Weaknesses.", "limitations": "Please refer to the Weaknesses.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- This paper first investigates the collaboration of table perception and comperhension tasks in a unified framework, and achieving impressive performances on diverse VTU tasks.\n- This paper proposes an efficient table detection strategy, without the demand of complex NMS algorithm to eliminate densely overlapped boxes. It directly predict the postions of all table instances conveniently, inspiring a new way to solve the detection-related tasks. \n- The new TQA benchmark, ComTQA, comprises high-quality QA pairs extracted from real-world table images. This benchmark addresses the limitations of previous benchmarks by including more complex question types that were previously absent, making it a more challenging and suitable benchmark for community development.", "weaknesses": "- Please add the references of all datasets in Tab.1.\n- In table 5, the task \"TD+TQ\" is unclear. Please clarify the setting of this task.\n- In ComTQA dataset, there exists some samples with multiple answers. Since this paper utilizes the accuracy metric, how to judge the multiple answers as correct or incorrect?\n- In addition, I have a minor consideration about the inference efficiency of the proposed TabPedia due to the mechanism of autoregressively output. Despite some of the unfairness of such comparisons, I suggest that it needs to be properly discussed in the Limitation section.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the detailed comments and acknowledgment of our contributions. We provide the responses as follows.\n\n*  **Q1:** Please add the references of all datasets in Tab.1.\n\n   **A1:** We will add the references of all datasets in Tab.1 in our revised manuscript following your suggestions.\n\n*  **Q2:** In table 5, the task \"TD+TQ\" is unclear. Please clarify the setting of this task.\n\n   **A2:** In the \"TD+TQ\" setting, the document images are first fed into TabPedia with the prompt of the TD task to detect all possible table positions. Next, the detected table positions are fed into TabPedia with the prompt of the TQ task to parse the specific table structure in turn. Since the table positions are generated from TabPedia rather than directly adopting ground-truth coordinates, this setting brings more challenges. Impressively, TabPedia still achieves plausible performance with slight accuracy degradation under this setting compared with those under the TQ setting.\n\n*  **Q3:** In the ComTQA dataset, there exists some samples with multiple answers. Since this paper utilizes the accuracy metric, how to judge the multiple answers as correct or incorrect?\n\n   **A3:** For multiple answers $y_1, y_2, ..., y_n$ in the single QA pair, we separate them with the special char '\\n' as the ground-truth answer. During testing, we judge if each answer $y_i$ exists in the response of TabPedia. The response is considered correct only if all the answers are present in the response; otherwise, it is considered incorrect.\n\n*  **Q4:** In addition, I have a minor consideration about the inference efficiency of the proposed TabPedia due to the mechanism of autoregressive output. Despite some of the unfairness of such comparisons, I suggest that it needs to be properly discussed in the Limitation section.\n\n   **A4:** Thanks for your constructive suggestions. We will add the following content to the Limitation section to discuss the inference efficiency of TabPedia. \n   \n   “TabPedia, as a multimodal large model, requires autoregressive answering. Compared to parallel decoding algorithms such as DETR [1] and Faster R-CNN[2], it consumes longer decoding time. Meantime, certain algorithmic designs such as KV cache, flash attention, and hardware improvements can effectively improve inference efficiency. We believe that with the iterative development of large model technology, the inference efficiency of TabPedia can be significantly improved.”\n\n   [1]: Carion, Nicolas, et al. \"End-to-end object detection with transformers.\" In ECCV 2020.\n      \n   [2]: Sun, Xudong, Pengcheng Wu, and Steven CH Hoi. \"Face detection using deep learning: An improved faster RCNN approach.\" In Neurocomputing 2018.\n\nIf our responses solve your concerns, we sincerely hope that you could raise your score. It's important to us.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Mri54UpJOR", "reviewer": "Reviewer_fN3K", "summary": "This paper introduces a novel large vision-language model, named TabPedia, aiming to perform comprehensive visual table understanding (VTU) in a unified framework. In order to tackle the dilemma of modal isolation and task exclusivity, it presents a concept synergy mechanism, in which diverse VTU tasks and multi-source visual embeddings are regarded as concepts. This framework integrates table detection (TD), table structure recognition (TSR), table querying (TQ) and table question answering (TQA) with the powerful capabilities of large language models (LLMs). Extensive experiments are conducted to validate the effectiveness of TabPedia. In addition, this paper establishes a new and comprehensive table VQA benchmark, ComTQA, featuring about 9,000 hihg-quality QA pairs.", "questions": "See the above Weaknesses.", "limitations": "Although there exist some dicussions about the broader impact and limitations of the proposed framework in the main paper, some deeper discussion is missing, such as whether the techniques presented in the paper can be extended in other areas? the potential challenges of visual table understanding in multilingual scenarios? More detailed dicussions could fullfill the comprehension of this paper.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "- This paper presents a unified approach that combines table perception and comprehension tasks with autoregressive language models for generating textual descriptions. Impressively, the design of TD, TSR and TQ tasks to adapt the serilization output format of LLMs also achieves comparable performances with previous task-specific methods.\n- The proposed concept synergy mechanism effectively enables the table perception-related and comprehension-related tasks in harmony, and achieves impressive performance in various visual table tasks.\n- The TQ task validates the powerful capabilities of LLMs for VTU. Preivous methods suffer from the redundant operation that first crops table-centric images from original documents, and then recognizes table structure. In contrast, TabPedia could directly parse table structure information from original documents with no significant degradation in performance. These results could inspire more researches to explore the LLMs' potential ability for broadly visual table understanding.\n- The qualitative visualizations of various VTU tasks, including TD, TSR, and TQA, on real-world images showcase the broad applicability of TabPedia. These visualizations highlight TabPedia's ability to generalize across different scenarios, suggesting its potential real-world applications.", "weaknesses": "- In object detection tasks, the outputs for different objects are unordered. However, large language models generally produce serialized outputs. Could you please explain in detail how TabPedia addresses the mismatch between the task and model architecture?\n- In line 187, the authors provide one exemplar about user's question for each table task. Could you give more detailed descriptions about instruction design and display complete instructions for each tasks?\n- In line 206-207, the structure of the table is represented with five object classes. These objects are indepedent of each other or have implicit relationship among them. Please explain this part for better understanding this descriptive format.\n- For ComTQA benchmark, please provide more detailed statistical information, such as the average question length, average answer length.\n\nThe Broader Impact section is not comprehensive enough, more detailed discussions are necessary.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the detailed comments and acknowledgment of our contributions. We provide the responses as follows.\n\n*  **Q1:** In object detection tasks, the outputs for different objects are unordered. However, large language models generally produce serialized outputs. Could you please explain in detail how TabPedia addresses the mismatch between the task and model architecture?\n\n   **A1:** When creating data annotations, for multiple unordered table boxes in a single image, we sort them in ascending order based on the coordinates of the top-left corner of each box. Specifically, we sort them first by the horizontal coordinate $x$, and then by the vertical coordinate $y$. This strategy ensures that the annotations for each image are unique and guides TabPedia to orderly understand image structure information and generate answers sequentially.\n\n*  **Q2:** In line 187, the authors provide one exemplar about the user's question for each table task. Could you give more detailed descriptions about instruction design and display complete instructions for each task?、\n\n   **A2:** Large language models inherently have powerful comprehension ability. Our instruction design follows two principles: 1) For different tasks, the instructions should be distinct to help the model better understand the differences between various visual tasks. 2) For a single task, the instructions should be diverse to enhance the model's ability to follow instructions. Based on both rules, we adopt GPT-3.5 to expand the manually designed instruction set. The complete instructions for different VTU tasks are shown in the uploaded pdf.\n  \n\n* **Q3:** In line 206-207, the structure of the table is represented with five object classes. These objects are indepedent of each other or have implicit relationship among them. Please explain this part for better understanding this descriptive format.\n\n  **A3:** Please refer to the official comment.\n\n* **Q4:** For the ComTQA benchmark, please provide more detailed statistical information, such as the average question length, average answer length.\n \n  **A4:** We calculate the average, maximum, and minimum lengths of the questions and answers in the ComTQA based on the number of characters. It is observed that the longest answer length reaches 1000 characters, attributable to the inclusion of all possible correct answers within the corresponding answer annotations, especially in cases where multiple answers are provided. More detailed statistical information could be found in the uploaded pdf.\n  \n    | Statistic   | Value          |\n    |-------------|----------------|\n    | Min question length | 17 |\n    | Max question length | 273 |\n    | Avg question length | 67 |\n    | Min answer length | 1 |\n    | Max answer length | 1000 |\n    | Avg answer length | 13 |\n\n\n* **Q5:** Although there exist some dicussions about the broader impact and limitations of the proposed framework in the main paper, some deeper discussion is missing, such as whether the techniques presented in the paper can be extended in other areas? the potential challenges of visual table understanding in multilingual scenarios? More detailed dicussions could fullfill the comprehension of this paper.\n\n  **A5:** Thanks for your constructive suggestions. We will add the following content to the Broader Impact section. \n  \n    \"In TabPedia, the collaboration of visual perceptive and comprehensive tasks via concept synergy mechanism shows impressive performance. For other document-related fields, this mechanism also could be applied or adapted to improve the reading capability of models. Exploring the transferability of our model to related tasks or domains could shed light on its versatility and applicability in various contexts. Furthermore, addressing the challenges of visual table understanding in multilingual scenarios is a crucial aspect that warrants more detailed discussion. Multilingual settings introduce complexities such as language variations, cultural differences, and diverse table formats that may impact the performance of our model. Investigating these challenges and proposing strategies to overcome them would enhance the robustness and generalizability of our approach.\"", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Thanks for the thorough reading and fruitful reviews, below we address the key concerns and suggestions case by case. For more clarity, we append the pdf containing Figures and Tables.

Meta-review: This paper got conflicting reviews, with two reviewers supporting acceptance and one supporting rejection. 

The point of contention appears to be around the novelty of the contribution itself and whether it merits publication at NeurIPS. The authors claim that their novelty lies in (1) the unified formulation of different table tasks, (2) the meditative tokens.

I am fine with (1). But I remain unconvinced by (2). The inclusion of the meditative tokens feels orthogonal to the contribution around table comprehension. It is also not explored sufficiently. One reviewer asked if the performance improvements from meditative tokens are simply because they add more parameters. This is a reasonable question and one that was not answered adequately. There is also the question of extra computation that the meditative tokens require; could the extra computation explain the performance increase?

Regardless, the performance increases are promising. The paper is being accepted but I would urge the authors to include the following experiments:

(1) Train your model with random meditative tokens frozen. If your model performs better, then we know that the meditative tokens do more than just provide extra computation. 
(2) I would also like to see experiments where the meditative tokens are used with single vision encoders, to understand their role further.
Decision: Accept (poster)