Integrating Table Representations into Large Language Models for Improved Scholarly Document Comprehension (2024)

Abstract: We address the challenge of interpreting and reasoning over scientific tables with Large Language Models (LLMs), a crucial aspect of scholarly documents. Despite significant progress in natural language processing, the integration of tabular data into scientific LLMs remains limited. We propose an innovative approach leveraging intermediate task pretraining on table question-answering datasets, followed by model adaptation to comprehend tables in computer science literature. Our findings reveal that incorporating table understanding substantially improves the performance of LLMs on scientific literature understanding tasks, which we showcase in peer-review score prediction. This improvement underscores the importance of utilizing tabular data in the training of scientific language models. The code and models are publicly available.1

heading: 1 Introduction
text: Large Language Models (LLMs) have experienced significant advancements in recent years and have been adapted to numerous natural language understanding and generation tasks (Zhao et al., 2023). Particularly in the scientific community, they have received increasing attention with their applications to scientific literature understanding tasks such as citation prediction (Cohan et al., 2020), paper classification (Zhang et al., 2023d), scientific literature search (Faggioli et al., 2023; Lála et al., 2023) and paper recommendation (Kanakia et al., 2019) to accelerate scientific discovery. In addition, domain-specialized research assistant language models have been developed (Beltagy et al., 2019; Luo et al., 2022; Taylor et al., 2022; Azerbayev et al., 2024).
Although these specialized models on scientific texts demonstrate success in the scientific literature understanding benchmarks such as MAPLE
1https://github.com/buseskorkmaz/ Integrating-Table-Representations-into-LLMs
(Zhang et al., 2023d) and SciRepEval (Singh et al., 2023), the benchmarks and the corpora used in the training of these scientific language models predominantly focus on textual data. A critical component - and the focus of this study - often remains overlooked, which is tables. Tables encapsulate key findings, offering a condensed view of the research outcomes. In this work, contrary to the existing approaches, we hypothesize that understanding tables can significantly enhance the performance of LLMs on scientific literature tasks by providing a more holistic understanding of research papers.
We first tackle the challenge of interpreting tables and reasoning over them to answer questions requiring arithmetic operations and choosing information from specific cells through intermediate task pre-training. Then, we adapt our trained model to comprehend scientific tables in published computer science papers. This training process is designed to enable the models to reason with scientific table data. The scientific tables dataset we use is fundamentally different from the datasets used in intermediate task pre-training for table question-answering, incorporating more extensive summaries of scientific tables. Finally, we demonstrate that utilizing table representations extracted from fine-tuned LLMs with our approach improves the prediction of peer-review scores.
Overall, we develop a pipeline that allows LLMs to incorporate scientific knowledge from tables. The main contributions of this work are: (i) we apply an intermediate task pre-training approach that allows LLMs to understand tables, (ii) we do a detailed comparison of scientific table understanding by different models with different sizes, architectures, and under various settings, and (iii) we show how learning to represent scientific tables improves the understanding of scholarly documents, using the peer-review score prediction as a case study.
293

heading: 2 Related Work

heading: 2.1 Scientific language models
text: The majority of pre-training datasets for scientific LLMs consist primarily of textual data, with a notable absence of tables. Widely used datasets in pre-training such as MAPLE (Zhang et al., 2023d), SciFact (Wadden et al., 2020), SciERC (Luan et al., 2018), ACL-ARC (Bird et al., 2008), SciCite (Cohan et al., 2019), GENIA (Kim et al., 2003), and BC5CDR (Li et al., 2016) include only titles, abstracts, references, or citations. The S2ORC (Lo et al., 2020) dataset includes full texts with parsed tables, yet its potential for enhancing table understanding in LLMs remains largely under-explored.
Models such as SciBERT (Beltagy et al., 2019) have been trained exclusively on words and sentences from scientific texts. Similarly, SPECTER (Cohan et al., 2020) focuses on titles, abstracts, and citations, without incorporating table data into its training process. BioMedGPT (Zhang et al., 2023a) acknowledges the significance of tabular data understanding but leaves it as a future task. Even recently developed models such as SciMult (Zhang et al., 2023c) and SciNCL (Ostendorff et al., 2022), which includes the S2ORC (Lo et al., 2020) dataset in its training mix, fail to leverage table data effectively. SciMult is trained on datasets of MAPLE, SciFact, and SciRepEval (Singh et al., 2023), which do not include tabular data, and SciNCL, despite its access to a dataset with parsed scientific tables (S2ORC) does not utilize table data in the training.

heading: 2.2 Table understanding
text: Recent advancements in table understanding have seen significant contributions. Pasupat and Liang (2015) introduced a compositional semantic parsing approach, which established the WikiTQ dataset for benchmarking. TAPAS by Herzig et al. (2020), leveraged the BERT architecture (Devlin et al., 2019), and advanced table parsing by identifying operations through a classification layer for answer generation. Eisenschlos et al. (2020) focused on enhancing table entailment through pretraining on open-source tables, aligning closely with our approach in Section 3.2. Hegselmann et al. (2023) explored the application of LLMs for few-shot classification of tabular data. Li et al. (2023) recognized the value of information in tables and developed a scientific information extraction pipeline to improve data availability for tabular content within scientific papers.
Moreover, improvements in table understanding have enhanced adjacent tasks such as table-based fact verification, as seen with the TabFact dataset (Chen et al., 2020), and extended to specialized fields such as finance, demonstrated by the TATQA benchmark (Zhu et al., 2021). Zhang et al. (2023b) developed a generalist table understanding model, TableLlama based on LLaMA-2 (7B) (Touvron et al., 2023) using fine-tuned 1.24M tables for 8 different table-based tasks such as table interpretation, augmentation and QA. We evaluate their model for the scientific table understanding task to investigate the capabilities of a generalist model in a scientific domain.

heading: 2.3 Peer-review prediction
text: The utilization of language models in predicting peer review outcomes, as highlighted by Rogers and Augenstein (2020), reflects their potential to understand the scientific literature. Accurately predicting the quality of scientific research through models could address the subjectivity, biases, and inefficiencies identified in the peer review process (Shah, 2022).
The PeerRead dataset (Kang et al., 2018) serves as a foundational dataset for peer-review prediction research, covering acceptance outcomes and review helpfulness. The availability of public peerreview datasets has accelerated the expansion of peer-review research, including studies on review content and decision outcomes (Gao et al., 2019), the introduction of innovative approaches to publication representation (Muangkammuen et al., 2023) and the development of predictive models for review scores.
In peer-review prediction, the accurate construction of scholarly document representations is important to learn the correct relationship between the documents and their peer reviews. The PeerRead dataset (Kang et al., 2018) includes comprehensive details of document bodies and associated peer reviews along with outcomes. Despite the dataset’s richness, the predominant methodology focuses on utilizing only the textual components of documents for representation. For example, peer-review prediction models DeepSentiPeer (Ghosal et al., 2019) and PeerAssist (Bharti et al., 2021) rely on the Science Parse library by AllenAI for extracting information from scholarly documents in PeerRead. Unfortunately, this library does not parse tables. This is a limitation if we are to capture the full scope of a scholarly document for peer review pre-
diction, as findings in result tables can substantially influence review outcomes.

heading: 3 Methodology

heading: 3.1 Datasets
text: WikiTableQuestions (WikiTQ) (Pasupat and Liang, 2015) WikiTQ is a benchmark dataset designed for evaluating the ability of models to perform question-answering (QA) over complex tables sourced from Wikipedia. This dataset challenges models to understand and interpret tabular data in context, requiring both a semantic understanding of questions and the ability to extract and reason relevant information from structured data. The inclusion of WikiTQ in our intermediate task pretraining regimen aims to enhance the model’s proficiency in handling structured data and improve its capability to reason over tables, an essential skill for understanding scientific tables. SQA (Iyyer et al., 2017) The SQA dataset extends the complexity of QA by introducing a sequential aspect, where answers to follow-up questions depend on the context established by previous interactions. However, our end use case is to describe scientific tables that do not have a conversational nature. Hence, we use a portion of the SQA dataset including the first questions in the sequence of questions over a given table. This dataset enables our model to further improve fundamental table understanding by adding diversity to the set of questions. SciGen (Moosavi et al., 2021) SciGen stands out for its focus on generating coherent and contextually accurate textual descriptions from scientific tables, primarily containing numerical data. The ability of arithmetic reasoning to interpret tables in scientific papers and generate appropriate textual
narratives presents the main challenge we aim to address. Thus, we subsequently fine-tune our model to adapt scientific tables on the SciGen dataset following pre-training on WikiTQ and SQA datasets.

heading: 3.2 Experimental Setup
text: Pre-trained LLMs We use FlanT5 (Chung et al., 2022) and LLaMA-2 (Touvron et al., 2023) as pretrained language models. Our task requires learning representations from structured tables. To compare how different architectures adapt to tabular data representation in our problem, we choose T5 (Roberts et al., 2019) and FlanT5 (Chung et al., 2022) to represent encoder-decoder architecture, and LLaMA-2 as a representative of decoder-only architectures. Data pre-processing Following Moosavi et al. (2021), we denote rows with <R>, columns <C> and
associated caption from scientific tables as <CAP>. Figure 2 demonstrates an example of this parsing operation. For LLaMA-2, we also see the benefit of using a special token for instructions 
. We also share the results reported in (Moosavi et al., 2021) over the SciGen dataset for T5 models (Roberts et al., 2019) in our result tables denoted with an asterisk (*) to benchmark our approach.
Intermediate task pre-training Our main goal is interpreting scientific tables to incorporate the learned representations into scientific language models and achieve better results over scientific literature tasks through a more comprehensive understanding of scholarly articles. As an initial experiment, we analyze the capabilities of the chosen LLMs on the SciGen test dataset and report results in Table 1 as a baseline to improve upon during intermediate task pre-training and fine-tuning. This test dataset includes further two subsets focusing on publications from Computational and Linguistics (Test C&L in Table 2) fields and a wide range of subfields of computer science (Test Other). The qualitative examination of generated texts from pretrained language models (red-coloured zero-shot example in Figure 3) concludes that the models are not capable of understanding table structure represented with tokens <R> and <C>.
To address this first challenge, we employ an intermediate task pre-training approach, similarly
(Eisenschlos et al., 2020). We use WikiTQ and SQA datasets to pre-train language models before fine-tuning them on scientific articles in the SciGen dataset. This intermediate step helps the language models to (1) capture the semantic relationships in the tables via our special tokens to represent them, (2) reason over tables to be able to answer questions requires arithmetic operations such as finding the maximum, and minimum values or selecting an answer from a specific cell.
Fine-tuning on scientific tables After the models gain the capability of understanding tables, we move to the next step to obtain specialized language models for scientific tables. At this stage, we utilize the large training dataset under SciGen. We use the provided “text” for each table as a reference and we expect the fine-tuned language model to produce similar text for a given table for the prompt of “Explain the given table”. Further implementation details are given in Appendix A. Figure 1 depicts the end-to-end training methodology explained in this section.
Evaluation metrics Following the evaluations in previous work on SciGen (Moosavi et al., 2021), we use a subset of their metrics in our evaluation such as METEOR (Denkowski and Lavie, 2014), and BertScore (BertS) (Zhang et al., 2019). Considering our generations for scientific tables are expected to be similar to the reference text, we
also add the ROUGE (Lin, 2004) score into our evaluation metrics set.

heading: 4 Understanding Scientific Tables

heading: 4.1 Zero-shot Evaluation
text: We share the evaluation of models in zero-shot task (without fine-tuning or intermediate task pretraining) results in Table 1. These results serve as a baseline for the comparison in Table 2. We can see that while LLaMA-2 is the largest model (with 7B parameters) in the table, the ROUGE and BERTScore of this model are lower than the FlanT5-large (0.78B) and FlanT5-xl (3B). Considering the increasing computational demand in training associated with larger model sizes, we chose to use FlanT5-xl in our detailed experiments under different settings. It is worth noting that, the fine-tuning of SciGen dataset results in Table 2 demonstrates substantial improvement for
LLaMA-2. This indicates that LLaMA-2 could also be adapted to scientific tables, although not with the same capacity as encoder-decoder architecturebased models.
The zero-shot evaluation results for TableLlama Zhang et al. (2023b) further validate the importance of adaptation to the scientific domain since being trained on a large training dataset and a variety of tasks, its performance on SciGen is still close to other language models which are not trained on table specific tasks.

heading: 4.2 Quantitative results
text: In Table 2, we report the scores obtained in each setting and model. When we compare the scores by model size, we can see that the largest model of FlanT5 family (Chung et al., 2022), FlanT5-xl (3B parameters) achieves the best scores on both test datasets introduced in (Moosavi et al., 2021). The scores of METEOR, ROUGE-1 and BertS also
show an increasing trend by model size.
Considering the FlanT5-xl is the most prominent model in our benchmarking set, we conduct the intermediate task pre-training step using different datasets with this model. During the intermediate task pre-training, we use WikiTQ and SQA datasets introduced in Section 3.1. The largest increase in the scores happens when we use the SciGen dataset. It is an expected result since this dataset focuses on scientific tables and has a structure similar to the test datasets. We also see the benefit of WikiTQ in intermediate task pre-training with the increase in scores. Interestingly, when we move to the intermediate task pre-training on SQA after WikiTQ, the scores do not improve. We hypothesize that the difference between the structure of scientific tables and table QA tasks becomes more pronounced after two subsequent pre-training on table QA datasets without fine-tuning on scientific tables. As a final step of our training, we fine-tune the model, which is trained on both WikiTQ and SQA previously, on scientific tables which achieves the highest scores in our comparison of different settings. Consequently, our experiments demonstrate the essential advantage of leveraging intermediate task pre-training on table QA datasets, substantially improving LLMs’ understanding and analysis of scientific tables.

heading: 4.3 Qualitative results
text: We share examples generated under different settings in Figure 3 for a sample table in the Test (Other) dataset, taken from (Yousefi-Azar et al., 2018). The table structure is encoded in the model input by using the tokens mentioned in Section 3.2. The dataset includes a reference text that assists us in quantifying the quality of our generations. The colourful texts in Figure 3 are the generations of the models. The red-coloured text is generated by FlanT5-xl without applying any intermediate task pre-training or fine-tuning. The generated text is non-sensible and indicates the model needs to adapt our table structure to understand the given information and produce a coherent text.
The generation of SciGen further demonstrates our motivation for intermediate task pre-training. Even though the generation is relatively high quality compared to the zero setting and factually correct for the given table sample, it is too generic and it is hard to extract tangible information using this generation. Thus, we find this kind of generation is not helpful for scientific literature understanding tasks. Comparing the SciGen generation, WikiTQ+SciGen output seems to contain more concrete information, however, some of the generated information is not factually correct when checking the table. Finally, the green-coloured generation is produced by the model pre-trained on WikiTQ and SQA, and fine-tuned on SciGen. We see the improvement in the generation quality as the output
is more concise, factually correct, and closer to the given reference. This conclusion aligns with our quantitative analysis findings in Section 4.2.

heading: 5 Peer Review Score Prediction

heading: 5.1 Experiments
text: To demonstrate the potential benefit of learning representations from tabular data in scientific articles, we incorporate tables into the peer-review score prediction task. We use the intersection of PeerRead (Kang et al., 2018) and SciGen (Moosavi et al., 2021) datasets, 55 publications across ICLR 2017, ACL 2017, and CoNLL 2016 as source data. Utilizing the entire content of the publications for peer-review prediction is impractical due to the context window length limitation of language models. Thus, previous approaches develop peer review predictive models using metadata, abstract and introduction sections of the paper (Singh et al., 2023).
In this section, we conduct experiments with table captions or representations generated by the models in addition to the title, abstract, and introduction to evaluate how table comprehension influences the accuracy of peer review score predictions, aligning them more closely with human reviewers’ evaluations. We employ FlanT5-xl, identified in Section 4 as the most effective model, to create summaries of the tables. The summaries’ embeddings serve as input for the prediction model, and we use XGBoost (Chen and Guestrin, 2016) for regression and classification. We then predict rec-
ommendation scores using embeddings from the FlanT5-xl model, which is fine-tuned under different settings. We evaluate our predictions using Mean Squared Error (MSE) and F1-score, defined as follows:
MSE = 1
n
n∑
i=1
(yi − ŷi)2 (1)
where n is the number of samples, yi is the true peer-review score, and ŷi is the predicted value.
F1-score = 2 · precision · recall precision + recall
(2)
where precision = TPTP+FP and recall = TP TP+FN , with TP being true positives, FP being false positives, and FN being false negatives. We share our findings in Table 3.

heading: 5.2 Results
text: The MSE and F1-scores in Table 3 show improvements in all settings when the table representations are used in peer-review prediction, except WikiTQ+SciGen. This finding validates our hypothesis that scientific language models could benefit from learning tabular data to fully interpret scientific literature. We only see a drop in the MSE score of WikiTQ+SciGen generations. We suspect the model in this setting hallucinates more as in the given sample Figure 3 and it misleads the XGBoost algorithm in peer review score prediction. Lastly, we obtain the lowest MSE and
highest F1-score using the embeddings from the WikiTQ+SQA+SciGen setting. This conclusion reinforces the findings in Section 4 and shows the effectiveness of our proposed approach.

heading: 6 Conclusions
text: Scientific language model development and document comprehension have accelerated progress in recent years parallel to advancements in large language models. However, their ability to effectively understand and reason over tabular data in scientific literature has remained under-explored. In this work, we addressed this issue by proposing an approach that combines intermediate task pre-training on table question-answering datasets with model adaptation to comprehend tables in computer science literature.
Our experiments demonstrated that by understanding tables better, LLMs can achieve higher performance in scientific literature understanding tasks. We validated this claim through a case study on peer-review score prediction, where our best-performing model, pre-trained on WikiTQ and SQA datasets and fine-tuned on the SciGen dataset, outperformed other settings in terms of mean squared error and F1-score. These results emphasize the importance of integrating tabular data into the training process of scientific language models.
Moreover, our qualitative analysis showed that the proposed approach generates more informative and contextually relevant summaries of scientific tables compared to generalist table models and models without intermediate task pre-training or fine-tuning. This finding suggests that our method can enhance the comprehension of scientific literature by providing more accurate and descriptive table representations. Future research directions could include extending our approach to other scientific domains, exploring the integration of table representations with other elements of scientific papers (e.g., figures and equations), and developing more sophisticated table encoding techniques. Additionally, incorporating larger and more diverse datasets for pre-training and fine-tuning could further improve the performance of LLMs on scientific literature tasks.

heading: B Additional Examples
text: We share more examples of different samples from the test dataset in this section.

heading: Future Work
text: Future research directions could include extending our approach to other scientific domains, exploring the integration of table representations with other elements of scientific papers (e. g. , figures and equations), and developing more sophisticated table encoding techniques. Additionally, incorporating larger and more diverse datasets for pre-training and fine-tuning could further improve the performance of LLMs on scientific literature tasks