Tables as Texts or Images: Evaluating the Table Reasoning Ability of LLMs and MLLMs (2024)

Abstract: In this paper, we investigate the effectiveness of various LLMs in interpreting tabular data through different prompting strategies and data formats. Our analyses extend across six benchmarks for table-related tasks such as questionanswering and fact-checking. We introduce for the first time the assessment of LLMs’ performance on image-based table representations. Specifically, we compare five text-based and three image-based table representations, demonstrating the role of representation and prompting on LLM performance. Our study provides insights into the effective use of LLMs on table-related tasks.

heading: 1 Introduction
text: Recent years have witnessed an explosion of Large Language Models (LLMs), with impressive performance on various Natural Language Processing (NLP) tasks (Brown et al., 2020; Touvron et al., 2023; Team et al., 2023). Research to date has examined the performance of LLMs for various aspects and abilities (Bang et al., 2023b; Bubeck et al., 2023; Akter et al., 2023), but their effectiveness on structured data such as tables is less explored.
Unlike unstructured text, tables are systematically organized structures of a large amount of information. This characteristic makes tabular data serve as the foundations for numerous applications, including medical diagnostics, virtual personal assistants, customer relationship management (Hemphill et al., 1990; Dahl et al., 1994; Akhtar et al., 2022; Xie et al., 2022), etc.
The evaluation of LLMs on processing tabular data involves many challenges. First, there are many ways to represent the information in tables. If we represent the table in pure text, we may use
*Contributed equally to this work. Appendix A lists the detailed contributions.
naive linearization or insert brackets to better represent table structures. Meanwhile, emerging multimodal LLMs like GPT-4 (Achiam et al., 2023) and Gemini (Team et al., 2023) offer image-based approaches, where we can pass the table as images to the LLMs. In such cases, visual cues like color highlighting in tables can influence outcomes. Second, diverse prompting methods for text may also apply to tabular data, which can yield varied results (Wei et al., 2022). Furthermore, the tasks involving tabular data are diverse, including table fact-checking (Chen et al., 2019) and table question answering (Pasupat and Liang, 2015), and table-totext generation (Novikova et al., 2017), etc.
In this paper, we systematically evaluate model performance on tabular data for both textual LLMs and multi-modal LLMs. Specifically, we investigate several research questions, including the effectiveness of image-based representation of tabular data and how different text-based or imagebased prompt methods affect LLMs’ performance on table-related tasks. In addition, we provide analysis and hypothesis of LLMs’ behaviors. Our findings include:
• LLMs maintain decent performance when we use image-based table representations. Sometimes, image-based table representations can make LLMs perform better.
• There are nuances in the prompting design for table-related tasks, revealed by our comparisons of various prompting methods for text- and image-based table representations.
To the best of our knowledge, we are the first to study how LLMs perform with image-based table representations. We believe this paper draws new insights into optimizing table-based information processing.
407

heading: 2 Related Work
text: Table-Related Tasks. Tasks involving structured data have attracted interest in various tasks from diverse communities (Deng et al., 2020; Chen et al., 2021a, 2022; Deng et al., 2022), among which there is a huge focus on tabular data (Yin et al., 2020; Herzig et al., 2020).
Researchers have investigated various ways to encode tabular data. Hwang et al. (2019); Liu et al. (2021); Cong et al. (2024) linearize the table content. Others employ model-specific techniques such as adapting the attention mechanism to better align transformer-based models with the tabular data (Zhang et al., 2020; Yang et al., 2022) or designing hierarchical encoding to capture the table structure (Wang et al., 2021), further tuning LLMs on tabular data (Zha et al., 2023; Zhang et al., 2023), etc. In contrast, our work focuses on exploring various table representations and prompts LLMs directly.
Prompting LLMs. Researchers have prompted LLMs to evaluate LLMs’ performance on traditional NLP tasks (Bang et al., 2023a) as well as on various complex reasoning tasks (Jin et al., 2022; Wu et al., 2023; Zheng et al., 2024). On the contrary, to the best of our knowledge, few works have prompted these LLMs on tasks involving tabular data.
For closed-source LLMs, researchers adopt hard prompts to manually craft text prompts with discrete tokens (Qiao et al., 2022; Bahng et al., 2022; Liu et al., 2023). Wei et al. (2022) develop chainof-though prompting, Xu et al. (2023a) develop expert prompting. In our work, we include the comparison between vanilla, chain-of-thought, and expert prompting for LLMs on table-related tasks.
uments revealing this information. “ / ” indicates whether the LLM is open-source ( ) or closed-source (
). “+V?” indicates whether the visual input is allowed for the LLM. “Company” indicates which company the LLM is from. We conduct the majority of our experiments with Llama-2. GPT-3.5, GPT-4 and Geminipro. We prompt the January 2024 versions of the GPT-3.5 and GPT-4. We include additional experiments for GPT4o, Llama-3, and Gemma models in Appendix G.

heading: 3 Experiment Setups

heading: 3.1 Experimented LLMs
text: Table 1 describes the LLMs we use for our experiments. We use closed-source models such as GPT3.5 and GPT-4 (Brown et al., 2020; Ouyang et al., 2022), and Gemini (Team et al., 2023). We note that GPT-4 and Gemini are multimodal models, which can take tables as images. For open-source models, we use the chat models from Llama-2 (Touvron et al., 2023) families from the 7 billion to the 70 billion parameter version as they are claimed to perform on par with closed-source models like ChatGPT.*
*https://huggingface.co/meta-Llama/Llama-2-70b-chat

heading: 3.2 Prompting Strategies
text: We explore two ways to represent tables in the prompt, Text-Based and Image-Based.
Text-Based. Apart from the information contained in the cells of tables, the structure of the table maintains information such as what cell values are in the same row or column, and what cell values correspond to a particular column. Therefore, we explore various ways to incorporate such structure information into the text prompt.
• Vanilla-T lists column names followed by cell values in each row sequentially, an approach adopted in various prior works (Hwang et al., 2019; Liu et al., 2021).
• Row-Identifier adds an identifier as the prefix for each row to distinguish different rows in the linearized table sequence.
• Bracket encloses the column names and their values in brackets to distinguish each row.
• Column-JSON represents the table in JSON format, where column names are the keys that map to the list of cell values corresponding to that column.
• Row-JSON represents each row as a JSON object, within which the column names and their corresponding cell values are represented as keyvalue pairs.
Table 2 shows examples of these text-based table representations.
Image-Based. Alternatively, we can pass the table as an image to the recent multimodal LLMs such as GPT-4 and Gemini. In this way, LLMs would “view” the table in a similar way as how we human beings view the table. We explore various table-highlighting methods as different visual cues may influence the model outcomes as shown by Shtedritski et al. (2023) who study how highlighting can influence CLIP model (Radford et al., 2021)’s performance on vision and language tasks. We pass these images of the table to LLMs.
• Vanilla-V feeds the table image without any colors or highlighting to LLMs.
• Column-Color uses a single color for each table column. Therefore, the LLM may easily distinguish columns as cells in the same column are annotated by the same color, whereas different colors annotate cells from different columns.
• Row-Color uses a single color for each row in the table. The same color annotates cells in the same row, whereas different colors annotate cells in different rows.
Figure 2 show examples for these image-based table representations.
On top of different methods to represent tables, we test the vanilla prompting, chain-of-thought prompting (Wei et al., 2022), and expert prompting (Xu et al., 2023a) by adding “let’s pretend you are an expert in reading and understanding tables” to the prompt. Appendix E provides an example for each table representation and prompting method.

heading: 3.3 Datasets
text: We make use of six previously introduced datasets that cover different table sources such as Wikipedia and financial reports, examine model abilities such as information extraction and arithmetic reasoning,
and cover table-related tasks such as table question answering, table fact-checking, and table-to-text generation. Table 3 provides information for each dataset we use. Considering the limited access to LLMs’ APIs and the scale of the comparison, we randomly select 100 examples from the test set for each of these datasets to conduct our analysis. In total, we run the our experiments on 54,000 instances*. In addition, we have observed significant differences between LLMs’ performance and report the results from three significance tests in Appendix B.

heading: 3.4 Metrics
text: Following Pasupat and Liang (2015); Chen et al. (2019, 2020, 2021b), we compute accuracy scores on WikiTQ, TabFact, LogicNLG, FinQA.
We adopt the automatic ROUGE evaluation for table-to-text generation datasets ToTTo and E2E. In addition, the authors manually investigate the generation quality on the E2E dataset by whether the generation encapsulates the table information without any additional information that cannot be inferred from the table.
*5 text form table representations × 3 prompting strategies × 6 LLMs × 6 datasets × 100 examples / dataset + 3 image table representations × 3 × prompting strategies × 2 MLLMs × 6 datasets × 100 examples = 54,000 instances. This does not include the number of additional experiments we run on other models such as Gemma, Llama-3 and GPT-4o.

heading: 4 Research Questions
text: Using the setup described previously, we can now seek answers to several research questions concerning the use of LLMs for tabular data.
RQ1. Are image-based representations of tabular data effective? Test: We compare using the linearized table representation (Vanilla-T in text-based table representation) and the table image (Vanilla-V in imagebased table representation) as the input for both GPT-4 model and Geminipro. We use vanilla prompting in this comparison and plot Figure 3. We report the results for other prompting methods in Figures 9 and 10 in Appendix C.1.
TL;DR Answer: Yes.
Full Answer: Figure 3 shows that in most cases, LLMs perform comparably if we represent tables as images versus text. On datasets such as FinQA, passing image representation of tables to Geminipro and GPT-4 outperform passing text representations of the tables significantly. As FinQA focuses on financial question answering with long context and many numerical relations, we hypothesize that representing tables as images can help LLMs in complex reasoning. Since these multimodal LLMs have a strong capability over visual input (Yang et al., 2023), representing tables as images may reduce
the cognitive load for LLMs to parse and understand dense text. This is especially beneficial when the context involves long passages of text that may also contain numerous numerical relations. As shown in Figure 4, since the context is long (around 416 English words, approximately 556 tokens for GPT models) and involves various numerical relations, GPT-4 ignores the relevant clues in text when we pass text representation of the table.
In contrast, when we pass the table image, GPT-4 can effectively leverage information from both the text and visual modality for its reasoning process.
On WikiTQ and TabFact, both Geminipro and GPT-4 perform better with the text than the image representation of the table significantly. We notice that both datasets are sourced from Wikipedia and the texts from Wikipedia are commonly used to pre-train LLMs (Brown et al., 2020; Touvron et al., 2023). GPT-4 and Geminipro may have encountered these tables in their pre-training phase in the text format rather than the image format, leading to the performance disparity between text and image representation of tables for both Geminipro and GPT-4 on WikiTQ and TabFact.
RQ2. How do different text-based prompt methods affect LLMs’ performance on table-related tasks?
Test: We compare the five text-based table representations introduced in Section 3.2. On top of the five representations, we also compare how vanilla, chain-of-thought, and expert prompting affect the model performance. We conduct the comparison
using all six LLMs in Section 3.1 and average their accuracy scores across FinQA, LogicNLG, TabFact, and WikiTQ. Appendix C.2 reports LLMs’ performance on E2E and ToTTo datasets.
TL;DR Answer 2.1: Expert prompting works the best when the LLM is an “expert”.
Full Answer 2.1: With respect to vanilla, CoT, and expert prompting, for GPT-4, we note that expert prompting outperforms the other two prompting methods consistently. For instance, for the vanilla linearized table representations (Vanilla-T), expert prompting outperforms the CoT and the vanilla prompting method by 1.2% and 3.5%, respectively (Table 4). In contrast, CoT prompting instead of expert prompting leads to the best performance for all other models. For instance, for GPT-3.5 with Vanilla-T table representation, CoT prompting outperforms vanilla and expert prompting by 2.7% and 4.2% (Table 4).
On the other hand, GPT-4 outperforms all other models, as the best average score GPT-4 achieves is 63.8%, compared to 55.2% by GPT-3.5 and 50.0% by Llama-2-70B. We suspect that because of GPT4’s “expertise” on these tasks, expert prompting can further enhance its reasoning ability as GPT-4 can “pretend they are an expert in reading and understanding tables”. In contrast, expert prompting may not fit the less capable LLMs as they may not “pretend an expert” well.
TL;DR Answer 2.2: CoT prompting can sometimes boost up the performance significantly.
Full Answer 2.2: We notice that CoT prompting significantly improves Geminipro’s performance from 38.4% to 57.3% using the bracket table rep-
resentation (Table 4), which outperforms the best performance 55.2% by GPT-3.5. This suggests that proper prompting can make a big difference in LLMs’ performance and unleash the potential within the LLM. On the other hand, it underscores the complexity of LLMs’ evaluation and the importance of prompt engineering, as we may underestimate an LLMs’ performance because of an improper prompt.
TL;DR Answer 2.3: Bracket representation can help LLMs better understand tables.
Full Answer 2.3: Compared to linearizing tables directly (Vanilla-T), adding brackets to distinguish rows in the table boosts up model performances for Geminipro and different versions of Llama-2 models (Table 4).
Figure 5 shows a WikiTQ example from Geminipro, where the vanilla prompting fails to count the number of “1st” place. We suspect that the simple linearized table representation does not have a clear boundary between rows, which may lead to confusion or misinterpretation of data relationships. In addition, adding the row identifier in the sequence does not help while the LLM answers correctly with the bracket representation. We conjecture that LLMs may be familiar with brackets from their pre-training exposure. Since brackets are fundamental components of many programming languages, and Github which contains rich code is often used as a source for pre-training corpora (Touvron et al., 2023), LLMs may have acquired proficiency in recognizing and interpreting bracketed structures.
TL;DR Answer 2.4: Different table representations do not affect the performance of GPT models much.
Full Answer 2.4: Even without any sophisticated prompting methods, the GPT-3.5 and GPT-4
achieve a decent performance (52.5% and 60.3% respectively using the vanilla prompting and linearized table representation from Table 4), demonstrating their strong table understanding abilities. In such cases, brackets or other kinds of table representations may add extra “workload” to the model, which dilutes the models’ attention to the original table content and thus leads to worse performance.
RQ3. How do different image-based prompt methods affect LLMs’ performance on table-related tasks? Test: We test the three image-based table representations in Section 3.2 together with vanilla, chain-of-thought, and expert prompting. We test the Geminipro and GPT-4 model which can take images as the input. We average the accuracy scores across FinQA, LogicNLG, TabFact, and WikiTQ. Appendix C.3 reports LLMs’ performance on E2E and ToTTo datasets.
TL;DR Answer 3.1: CoT prompting helps LLMs reason over images of the table.
Full Answer 3.1: In Table 5, we observe that chain-of-thought prompting helps multimodal LLMs in all image-based table representations. For instance, when using different colors to distinguish
rows in the table (Row-Color), the average accuracy score for GPT-4 improves from 51.8% by vanilla prompting to 60.2% by chain-of-thought prompting. By explicitly outlining the reasoning process, chain-of-thought prompting may help LLMs better understand the context and relationships between different rows and columns in the table, therefore better aligning this visual information with the question text. Such consistent performance improvements suggest that chain-of-thought prompting may enhance information fusion across the text and vision modality.
TL;DR Answer 3.2: Distinguishing rows may lead to better performance for LLMs to reason over images of the table.
Full Answer 3.2: In Table 5, under CoT prompting, GPT4 performs slightly better when using colors to distinguish different rows, which also yields the overall best performance using images of the table. In contrast, under CoT prompting, using colors to distinguish columns yields similar performance to vanilla image (58.0% to 57.8% for GPT-4 and 45.1% to 46.0% for Geminipro), suggesting that these advanced LLMs may not capture row information as well as column information.
Figure 6 shows a WikiTQ example with Geminipro model’s predictions. Since the question asks about the number of games, it requires the model to count how many rows satisfy such a condition. Using colors to distinguish rows may help models visually segment and categorize the data. This visual differentiation may act as a cognitive aid, which reduces the complexity of parsing and interpreting the tabular data.
TL;DR Answer 3.3: The more capable LLM does not necessarily benefit more from the colored images.
Full Answer 3.3: In addition, if we use the vanilla prompt, the different coloring methods may even hurt the performance of GPT-4 (for GPT-4, coloring rows with different colors yields 51.8% compared to 56.2% without adding any color), but helpful for Geminipro (for Geminipro, coloring rows with different colors yields 39.4% compared to 36.8% without adding any color). This suggests that the effectiveness of how different LLMs can leverage colored images varies, and does not depend on the model’s overall performance.
RQ4. Does highlighting relevant cells yield a better performance? Test: We test all six LLMs in Section 3.1 on FinQA which provides relevant cells in the table for each instance. We refer to the relevant cells by adding “Please pay attention to the highlighted cells: (row index, column index, cell value)” in the text prompt, or mark them on the table image directly. Appendix E provides our prompt examples. We use vanilla prompting in this comparison.
TL;DR Answer: Yes.
Full Answer: In Table 6, we notice that in most cases, referring LLMs to specific cells helps LLMs better attend to them, thereby helping LLMs reason over the example. However, LLMs’ performance may get hurt when we refer to the relevant cells through text such as Llama-2-13B and 70B. This may be due to the inherent limitations of textual descriptions for conveying spatial or relational information. In order to relate the mentioned cells in the text, the model needs to figure out the connection between the mentioned cell and the cell in the linearized table, which can be challenging to the model given the complicated table structure.
In addition, LLMs best attend to the table items when there are clues from both text and image. In Table 6, we observe that marking the relevant cells on the image while mentioning them through text leads to the most correctly answered examples (61 examples by GPT-4 and 38 by Geminipro at the last row in Table 6). Such a dual-modality approach that combines visual cues with text references, en-
open-source ( ) LLM we test, Llama-2-70B versus closed-source ( ) LLM we test, GPT-4 on different datasets. The closed-source LLMs always outperform the open-source LLMs and we report the performance difference ∆ between them. For consistency across different datasets, we do not include the performances with highlighting cells*in this comparison. Table 20 in Appendix D provides what combinations of table representation and prompting method yield this performance.
hances LLMs’ overall reasoning ability over the tabular data.
RQ5 and RQ6
We include two additional research questions and our answer to them in Appendix C, including whether these LLMs can reconstruct tables from the image, and whether multimodal LLMs benefit from having both formats simultaneously as input.

heading: 5 Open Problems to Increase the Performance of LLMs on Tabular Data
text: Mathematical reasoning. We observe that LLMs are not good at arithmetic reasoning similar to the findings in prior works (Hendrycks et al., 2021; Imani et al., 2023). As shown in Figure 7, simple arithmetic computing like counting the total number of rows that satisfy certain conditions (‘1st’ in Figure 7) still poses challenges even for GPT-4.
*Except for ToTTo, where the task is to generate the sentence based on the highlighted cells. On ToTTo, we include the highlight information just in text.
This suggests that these previously proposed benchmarks are still valuable in evaluating LLMs, as many of these datasets involve arithmetic reasoning such as FinQA and WikiTQ, which is challenging for even the most sophisticated LLMs like GPT-4. For datasets like E2E for table-to-text generation, though relatively easy for advanced models like GPT-4, smaller models cannot describe the table fully correctly as shown in Figure 8. Therefore, these datasets are still valuable for benchmarking the progress of smaller LLMs.
Appendix F provides more examples from each dataset where LLMs can and cannot answer well.
Closing the gap between open-source and closedsource LLMs In Table 7, we report the best performance among different prompting methods for the best performed open-source Llama-2 models versus the best performed closed-source GPT models on FinQA, LogicNLG, TabFact, and WikiTQ. We note that on these tasks, GPT-4 always performs the best among all the closed-source LLMs we test.
We observe that open-source Llama-2 models obtain decent performances across these benchmarks. as shown in Table 20. Figure 8 shows an example from E2E dataset. The smaller Llama-2 models such as Llama-2-7B or Llama-2-13B make mistakes in counting rows. However, they capture almost all the information in the table including the name, eat type, and area of the restaurant. As the model scales up, the Llama-2 70B model can describe the table accurately.
However, significant performance gaps exist between open-source Llama-2 models and closedsource GPT-4 models. In Table 7, the gap between
open-source Llama-2 models and GPT-4 can be as large as 15% on FinQA and 22.9% on TabFact. Even on LogicNLG which has the smallest performance gap, there is an 8.4% difference between the Llama-2 and GPT models. As Llama models often serve as the foundation models for a wide range of NLP research (Roziere et al., 2023; Xu et al., 2023b), we need the effort from the open-source community to keep developing stronger LLMs to close the gap between open-source and closedsource LLMs.

heading: 6 Conclusion
text: We have explored various representation strategies, including both text-based and innovative imagebased approaches, to understand how to use LLMs effectively in tasks involving tabular data. We demonstrate the effectiveness of image-based representations and reveal the impact of prompting strategies on the performance of LLMs. We believe our insights contribute to the understanding of LLMs and how to optimize LLMs for tabular data processing.

heading: 7 Ethical Statement
text: We conduct our studies on six pre-existing and publically available datasets using various existing LLMs. Prior works have pointed out the potential bias in these LLMs (Bender et al., 2021) which practitioners need to be aware of.

heading: 8 Limitations
text: In this study, we do not intend to exhaust every possible text representation, image representation of tables, or every possible LLM. Moreover, we do not have access to the closed-source LLMs behind their API. Instead, we designed our experiments to try to explore the research questions we raised in this paper. We hope our findings and insights in this paper can inspire future research on table-related tasks.

heading: Acknowledgements
text: We thank Inderjeet Nair for the early discussion on different forms of table representations; Alexander Hanbo Li, and Patrick Ng for discussions of our work and pointers to related work; the authors of the table-related datasets we use in our experiments for making their datasets publically available; the anonymous reviewers for their valuable suggestions. This project was partially supported
by the OpenAI Researcher Access Program and by a Microsoft Foundational Model grant.

heading: A Contributions
text: Idea Proposal. Naihao Deng proposed the idea of evaluating LLMs’ performance on table-related tasks with various text-based and image-based table representations.
Background Survey. Zhenjie Sun conducted a careful survey on table-related tasks. Naihao Deng did the initial background survey on table-related tasks when he was a research assistant in Westlake University with Yue Zhang.
Implementation. Zhenjie Sun came up with various table representations and designed all the prompts. Zhenjie Sun also implemented the evaluation pipeline for autonomous metrics. Naihao Deng and Zhenjie Sun implemented the pipeline for human evaluation.
Experiments. Zhenjie Sun and Naihao Deng conducted all the experiments. Specifically, Zhenjie Sun conducted experiments on GPT-3.5, GPT-4 with text-based table representations, and Geminipro. Naihao Deng conducted experiments on Llama-2 models and GPT-4 with image-based table representations.
Result Aggregation. Zhenjie Sun conducted the result aggregation for all the experiments.
Human Evaluation. Zhenjie Sun, Naihao Deng, Ruiqi He, Aman Sikka conducted the human evaluation for the model generation.
Paper Writing. Naihao Deng drafted the paper. Zhenjie Sun drafted the prompting strategies and metrics, and actively got involved in discussions of result analysis. Naihao Deng and Zhenjie Sun plotted all the tables and figures. Naihao Deng, Zhenjie Sun selected examples that appeared in this paper. Ruiqi He, Aman Sikka provided suggestions on example selections. Rada Mihalcea, Yue Zhang, Lin Ma, and Yulong Chen participated in discussions throughout the entire project and provided revisions and feedback on the paper.

heading: B Statistical Significance Test Results
text: We have observed significant differences between LLMs’ performance. We conduct three statistical significance tests, including Fisher’s Exact test, McNemar’s Test, and proportion Z test for the model predictions.
With p < 0.05:
1. For Fisher’s Exact test, we find a statistically significant difference in GPT-4 performance between T and V inputs for FinQA, LogicNLG, TabFact, WikiTQ (Figure 3), its performance between vanilla and expert prompting for table text input (Table 4), and its performance difference under vanilla and chain-ofthought prompting for image-based table representations (Table 5).
2. For McNemar’s Test, we find statistically significant differences between GPT-4 performance between T and V inputs for FinQA, TabFact, and WikiTQ (Figure 3), as well as between vanilla and expert prompting for table text input (Table 4).
3. For the proportion Z test, we find a statistically significant difference in GPT-4 performance between T and V inputs for FinQA, WikiTQ (Figure 3).

heading: C Research Questions Cont’d
text: C.1 RQ1 Cont’d. Can we use image-based representations of tabular data?
Figure 9 and Figure 10 show the performance comparison between feeding text representations versus image representations of the table to GPT-4 and Geminipro for chain-of-thought and expert prompting, respectively. The results resemble similar trends as Figure 3.
C.2 RQ2 Cont’d. How do different text-based prompt methods affect LLMs’ performance on tabular-related tasks?
Table 8 reports the averaged accuracy scores across FinQA, LogicNLG, TabFact and WikiTQ that use accuracy as the metric. Table 9 and Table 11 report the ROUGE-L scores of LLMs’ generation on E2E and ToTTo dataset, respectively. Table 10 reports the scores annotated manually by the authors. As discussed in Section 3.4, the authors manually check whether the generated sentence captures all the information from the table and does not include any additional or misinformation. We assign “1” for sentences who satisfy the criteria and “0” otherwise.
C.3 RQ3 Cont’d. How do different image-based prompt methods affect LLMs’ performance on tabular-related tasks?
Tables 12 and 13 report the ROUGE-L scores of GPT-4 and Geminipro when we use image representations of tables on E2E and ToTTo dataset, respectively. Table 14 reports the scores annotated manually by the authors.
C.4 RQ5: Can These LLMs reconstruct tables from the image?
Test: We conduct experiments to explore table reconstruction ideas using GPT4 on E2E and FinQA datasets.
TL;DR Answer: Mostly yes.
Full Answer: GPT4 successfully reconstructs 97/100 tables on the E2E dataset, and 72/100 tables on the FinQA dataset. We notice that for simpler tables, GPT4 can reconstruct tables almost perfectly. For instance, in E2E, the only “mistake” the model makes is to output “é” instead of “e” in “Rainbow Vegetarian Café”.
GPT4 also demonstrates a decent capability of reconstructing complicated tables. On FinQA,
GPT-4 manages to reconstruct a 4-row×10-column table with over 100 words (Table 16).
But GPT-4 is more prone to hallucination or messing up with the spatial relations when the table gets more complicated. For instance, for Table 17, GPT-4 reconstructs it as Table 18. As expected, GPT-4 fails to answer the corresponding question to this table when using a table image as the input. However, we notice that when using the text representation of the table, GPT-4 also fails to answer this question. This aligns with what we have reported in.
As expected, GPT-4 fails to answer the corresponding question to this table when using a table image as the input. However, we notice that when using the text representation of the table, GPT-4 also fails to answer this question. This aligns with what we have reported in Figure 3 that On FinQA, GPT-4 better leverages the image representations than the text representations in general.
C.5 RQ6: Do multimodal LLMs benefit from having both formats simultaneously as input?
Test: We test GPT-4 with vanilla prompting for the six datasets.
TL;DR Answer: Not generally true.
Full Answer: As shown in Table 19, it is not generally true that multimodal LLMs benefit from having both formats simultaneously as input.
There are cases where passing both image and text representations would increase the performance (e.g. GPT-4 on WikiTQ and FinQA). In other cases, the performance is comparable to either passing tables as text or image representation or lies in between.

heading: D Comparison of Llama Models and GPT-4 Models
text: Table 20 provides the details of what combination of table representation and prompting method yields the best performance with respect to the Llama-70B and GPT-4 models.

heading: E Prompt Examples
text: Figure 11 gives an example of how we construct our prompt for an instance in WikiTQ.

heading: F LLMs’ Generation Examples on Each Dataset
text: Figure 12 gives examples for WikiTQA, TabFact, LogicNLG, and FinQA datasets we use, how many combinations of LLMs, table representations, and prompting techniques can answer the question cor-
rectly. We notice that LLMs tend to answer well in general if the example focuses on extracting information from the table, but answer poorly if the question involves some arithmetic reasoning such as counting rows and comparing with others (examples from TabFact and LogicNLG), or complicated calculation that involves several steps (the example from FinQA). Figure 13 provides examples from E2E dataset.
Figure 14 provides examples from the ToTTo dataset, the models generally describe information better when there is less information.

heading: G Additional Experiments on GPT-4o, Llama-3 and Gemma models
text: Table 21 provides additional results for vanilla prompting for GPT-4o, Llama-3 and Gemma. We have observed that the GPT-4o performs similarly if we pass the table either through image or text.
Dataset Question / Statement Table Gold Correct (out of 108)
Dataset Question / Statement Table Gold Avg ROUGE-L
source ( ) LLM we test, GPT-4 on different datasets. Four datasets uses accuracy as metrics and two datasets (E2E and ToTTo) uses ROUGE-L as metrics. The closed-source LLMs always outperform the open-source LLMs and we report the performance difference ∆ between them. We include the table representation (“Table Repr”) and prompting methods that yield the best performance next to the columns that report open-source and closed-source LLM scores, respectively. For consistency across different datasets, we do not include the performances with highlighting cells in this comparison.

heading: Future Work
text: We hope our findings and insights in this paper can inspire future research on table-related tasks