Venue: NAACL 2024

# Efficient End-to-End Visual Document Understanding

with Rationale Distillation

Wang Zhu\({}^{\spadesuit\diamondsuit}\) Alekh Agarwal\({}^{\clubsuit}\) Mandar Joshi\({}^{\clubsuit}\) Robin Jia\({}^{\heartsuit}\) Jesse Thomason\({}^{\heartsuit}\) Kristina Toutanova\({}^{\spadesuit}\)

\({}^{\clubsuit}\)Google DeepMind Google Research University of Southern California

Work done during internship at Google DeepMind

###### Abstract

Understanding visually situated language requires interpreting complex layouts of textual and visual elements. Pre-processing tools, such as optical character recognition (OCR), can map document image inputs to textual tokens, then large language models (LLMs) can reason over text. However, such methods have high computational and engineering complexity. Can small pretrained image-to-text models accurately understand visual documents through similar recognition and reasoning steps instead? We propose Rationale Distillation (RD), which incorporates the outputs of OCR tools, LLMs, and larger multimodal models as intermediate "rationales", and trains a small student model to predict both rationales and answers. On three visual document understanding benchmarks representing infographics, scanned documents, and figures, our Pix2Struct (282M parameters) student model finetuned with RD outperforms the base model by 4-5% absolute accuracy with only 1% higher computational cost.

## 1 Introduction

Information in the digital world is conveyed through text integrated with visual elements, such as complex layouts, figures, and illustrations. Answering user questions based on such visual documents requires models to recognize and connect text and layout to the user need.

While pretrained image-to-text multimodal models have demonstrated strong performance on visual document understanding (VDU) by directly mapping pixel-level input document images to answers corresponding to user queries Kim et al. (2022); Lee et al. (2023); Chen et al. (2023b,c), state-of-the-art approaches benefit from the use of external tools. Tools include OCR systems Chen et al. (2023b); Powalski et al. (2021); Huang et al. (2022), structured table source extraction Liu et al. (2023a), and LLMs reasoning over extracted information and the user query Liu et al. (2023a); Perot et al. (2023). Additional tools such as image captioning, object classification, and search engines have been used for other multimodal tasks Yang et al. (2023); Zhang et al. (2023). However, the accuracy gains from these external components come at the cost of decreased computational efficiency and increased engineering complexity.

In this work, we ask whether we can achieve high accuracy and efficiency by teaching a smaller model to learn from short rationales generated by external tools and expensive LLMs (see Figure 1). We use a small student image-to-text model to perform VDU tasks by decomposing them into rationale prediction and answer steps, predicting the rationale and answer in sequence. The "rationale" can be any intermediate textual information that helps answer a question correctly: for instance, it could be a subset of relevant text from the image as well as layout, structured information, and rea

Figure 1: We synthesise the ability of recognizing and summarizing text, deplotting structured plots, and program generation into one small model and perform efficient rationale-based visual document understanding.



soning (see Figure 2).

The training data for VDU tasks of interest does not generally contain annotated "rationales." It is also not known what types of sufficiently succinct rationales, even if available, would be useful for a small image-to-text model. We take inspiration from related works on chain-of-thought distillation Shridhar et al. (2023); Zhang et al. (2023) for text and multimodal tasks, borrowing techniques and adding novel components to address the specific challenges within the visual document understanding domain. We use chains of tools at training time to derive short rationales representing salient subtasks of the problem--recognizing text and layout, and deriving programs to encode numerical reasoning. To increase the quantity and validity of example rationales, and the student's robustness to incorrect predictions, we design data augmentation schemes and DAGGER-style Ross et al. (2011) loss functions, which improves the student's ability to benefit from intermediate predictions.

Our method takes advantage of task decomposition and reasoning, but offers the following advantages over other tool-using models:

* No OCR or other external tools used during inference, reducing engineering complexity.
* Only a short, query-dependent rationale is predicted versus longer structures typically extracted by external tools, saving computation.
* Computation is increased by only about 1% (in FLOPS) compared to models that predict the answer directly.

We conduct experiments on three VDU benchmarks: InfoVQA Mathew et al. (2022), DocVQA Mathew et al. (2021), and ChartQA Masry et al. (2022). We show accuracy improvements over models that predict answers directly. For models based on Pix2Struct-Base (282M parameters), improvements are 4.0 and 4.6 points in ANLS on InfographicVQA and DocVQA respectively, and 3.3 / 7.7 points in relaxed accuracy on ChartQA's augmented and human sets, with similar improvements for larger Pix2Struct models (1.3B parameters).

## 2 Task definition

In VDU, a model is given an image \(I\) and user question \(q\), and predicts text answer \(a\). We focus on training a single small image-to-text model with parameters \(\theta\) for this task. Prior work in VDU trains such models by maximizing the training data log-likelihood according to an image-to

Figure 2: For training examples, we first generate the full OCR of each image with Google Cloud OCR. Depending on the dataset, we either use LLM-Summarizer (few-shot prompted PaLM 2-L) to generate text evidence (top), or use LLM-Programmer (also PaLM 2-L) to generate a program based on both the OCR and available structured table source for the image (bottom).

 

text (or image+text-to-text) model that directly generates the text answer \(a\) given the input and makes predictions through greedy decoding, _i.e._, \(\hat{a}=\operatorname*{arg\,max}p_{\theta}(a\mid q,I)\)Kim et al. (2022); Lee et al. (2023); Chen et al. (2023); Wang et al. (2022).

We assume that external tools such as OCR systems, LLMs, larger image-to-text models, or structured input image source information may be available at training time, but not inference time. We use such tools and metadata to derive rationales \(r\) paired with training input-output examples \((I,q,a)\), and train a small student image-to-text model to predict rationales \(r\) as an intermediate reasoning step, before predicting the answer \(a\).

## 3 Rationale Distillation

We propose Rationale Distillation (RD), which distills rationales from a predefined set of tools, and trains a student model to predict the relevant rationales before predicting the answer.

_Rationales_ are sequences of text tokens of relevant information to arrive at the answer. We consider two kinds of rationales \(r\): natural language text evidence derived from the output of an OCR system; and tabular representation of charts in the input image concatenated with simple custom programs with predefined operations. The tools we leverage are: an OCR tool Google Cloud OCR); Plot-to-Table, a converter that converts charts or plots to structured tables; a LLVMer (designed by us), which summarizes OCR text to evidences relevant to the question using a prompted PaLM 2-L model Anil et al. (2023); a LLM-Programmer (also designed by us and based on PaLM 2-L), which generates simple programs for numerical reasoning tasks; and a Multimodal-Verifier based on PaLI-X Chen et al. (2023), which verifies the quality of the rationales. We provide detailed descriptions of these tools in Appendix A. As the tools add heavy computation (for LLMs) or engineering complexity (for OCR), we depend on none of them at inference time.

In this section, we first discuss the process of generating the two types of rationales from tools (SS3.1). We then describe a data augmentation scheme for increasing the number of examples with rationales, and making student models more robust to potentially noisy rationales (SS3.2). Finally, we discuss training and inference for student models to predict the rationale and the answer (SS3.3, 3.4).

### Rationale generation from tools

InfoVQA and DocVQA require a strong ability to recognize text, so we first use the OCR tool to extract the text from the image, then perform 5-shot prompting with LLM-Summarizer to generate question evidence (Figure 2, top). ChartQA focuses on numerical reasoning on charts, so we extract the full OCR text, obtain structured tables using Plot-to-Table,1 and then prompt LLM-Programmer (using 8 in-context examples) to generate a program to derive the answer. The concatenation of the structured table and the program are then used as the rationale (Figure 2, bottom). Detailed prompt templates are in Appendix E.

Footnote 1: We use provided ChartQA structured tables directly.

### Rationale augmentation and filtering

We aim to enable a small student model to reason over visual documents ranging over diverse formats and complexity. The expensive tools can typically generate high-quality rationales from such data, but it is a significant challenge for a small student model to match the quality of these rationales from a limited amount of training data. To overcome this challenge, we devise a data augmentation approach

Figure 3: We first crop along the longer edge of the image to create multiple smaller square images. We generate rationales using the appropriate subset of tools (OCR, LLM-Summarizer, LLM-Programmer, Plot-to-Table) on these images, then categorize the examples and rationales with Multimodal-Verifier (PaLI-X).

 

based on image cropping to greatly enlarge the number of examples available for rationale prediction, and to teach the model to use variable quality rationales in generating the final response.

```
1:Input: image \(I\); question \(q\); answer \(a\); rationale \(r\); multimodal verifier with parameter \(\phi\).
2:Output: A tuple (the category of the rationale, the assigned answer used for training).
3:if\(\arg\max_{\bar{a}}p_{\phi}(\hat{a}\mid I,q,r)\neq a\)then
4:return "irrelevant", "None"
5:endif
6:if\([p_{\phi}(a\mid I,q,r)]^{\lambda}\geq p_{\phi}(a\mid I,q)\)then
7:return "useful", \(a\)
8:else
9:return "relevant but not useful", \(a\)
10:end if
```

**Algorithm 1** Rationale Filtering

Cropping-based augmentation.We crop the original image along the longer dimension, resulting in multiple square images (Figure 3). To minimize the possibility that the most relevant segment does not fit within any crop, we use a sliding window with adjacent croppings overlapping by half the image size (Algorithm 2, appendix). For an input image \(I\), we obtain \(k\) cropped images \(i_{1},\dots,i_{k}\) and generate corresponding rationales for them as detailed above. As an example, in the InfoVQA dataset we observe an average of \(k\approx 4\).

Filtering relevant and useful examples.While cropping significantly increases the size of our training dataset, many of the images might not contain information pertaining to the answer, and we may not be able to extract reasonable rationales. Including such examples in our dataset can amplify noise and make the problem more challenging for the student. So we carefully filter the augmented data to extract examples which are useful for rationale and/or answer prediction. We use a powerful Multimodal-Verifier (PaLI-X) with parameter \(\phi\) to design two filters on VDU tasks (Algorithm 1). We analyze the validity of PaLI-X as a verifier model in Appendix B.4.

(1) The _relevance filter_ checks if the cropped image \(i_{j}\) contains information for answering the question by comparing greedy decoding with the rationale as input against the gold answer: \(\operatorname*{arg\,max}_{\tilde{a}}p_{\phi}(\hat{a}\mid i_{j},q,r_{j})=a,j \in\{1..k\}\) (row #3 of Algorithm 1). For examples failing this filter, we replace the answer \(a\) with None in the training data, assuming the cropped image is insufficient to generate the answer. For instance, the first cropped image of Figure 3 does not contain the gold answer "Instagram" and the example falls within the irrelevant category. We still use the rationale \(r_{j}\) for rationale prediction, since it could help distill the tool into the student model.

(2) The _rationale filter_ applies to examples that pass the relevance filter, and checks if the probability of the gold answer is sufficiently increased given the rationale (row #6 of Algorithm 1). We use a factor \(\lambda=2\) to avoid small perturbation caused by changing the format of the model prompt by concatenating the rationale. For examples that pass the relevance filter but not the rationale (row #9), we regard the rationale \(r_{j}\) as low-quality, and do not use it for learning rationale prediction. For instance, the second cropped image of Figure 3 contains the gold answer "Instagram", but the tools do not generate a useful rationale.

We classify \((i_{j},r_{j})\) pairs into three categories (rows #4,7,9 of Algorithm 1), which determine their assigned answer \(\bar{a}=a\) or None and the way their rationales are used in training.

Dataset balancing.Most examples fail the relevance filter, and more than half of the ones that pass fail the rationale filter. We subsample the examples with label None (row #4) such that their number \(n_{\text{row \#4}}\leq n_{\text{row \#9}}-n_{\text{row \#7}}\).

### Training student models

In Rationale Distillation, we perform multi-task training for the student model, using tasks derived from the original and augmented data annotated with rationales. Tasks differ by their encoder and decoder inputs and decoder outputs (Table 1). We weight the four tasks equally (_i.e._, 0.25 for each), and train on a linear combination of them, with loss defined over the target output.

\begin{table}
\begin{tabular}{l c c c} Task name & Encoder input & Decoder input & Target output \\ \hline GRA & \(I\) & - & \(q,r,a\) \\ ASR & \(I\) & \(q,\hat{r}\) & \(a\) \\ ORACI & \(i_{j}\) & - & \(q,r_{j},\check{a}\) \\ ALRCI & \(i_{j}\) & \(q,r_{j}\) & \(\bar{a}\) \\ \multicolumn{4}{c}{} \\ \end{tabular}
\end{table}
Table 1: We compute loss on the target output tokens for four student training tasks. Encoder input images have questions \(q\) rendered as the header. Rationale \(r\) (resp. \(r_{j}\)) is generated by tools on image \(I\) (resp. \(i_{j}\)). Rationale \(\hat{r}\) is generated by students.

 

Distilling the tools directly.This vanilla Question, Rationale and Answer (QRA) distillation setup teaches the model to take in the original image and predict \(q\) (which can be read out from the image header), \(r\) ( the intermediate rationale generated by the tools), and then by the answer \(a\).

Robustifying against student rationale errors.To help make the student model robust to its own mistakes, the Answer with Student Rationale (ASR) task provides question \(q\) and student generated rationale \(\hat{r}\) as decoder input for the student model to predict the answer. To generate such student rationales \(\hat{r}\), we use a separately trained Pix2Struct-based student model, which learns to predict only rationales.

We sample three student generated rationales for each input example and use them as the low-quality rationales \(\hat{r}\). Since the training loss for ASR is only applied to the answer prediction, the RD student is not encouraged to replicate these noisy rationales, but to be able to recover from potential errors and predict the gold answer. We note that other than the difference of a separate student model generating the rationale, this is akin to student-forcing or DAGGER style approaches to structured prediction Ross et al. (2011).

Leveraging cropped images.In Question, Rationale and Answer on Cropped Images (QRACI), we use cropped images \(i_{j}\) with rationales identified as useful (row #7 of Algorithm 1) or irrelevant (row #4), to learn to predict those rationales and the original answer or None, respectively. Answer with Low-quality Rationale on Cropped Images (AL-RCI) is similar to ASR, taking cropped images as encoder input and providing low-quality rationales (row #9) in the decoder input.

### Model architecture and inference details

Pix2Struct is an encoder-decoder model using a Transformer image encoder for an input image, and a Transformer-based decoder generating text. Following Lee et al. (2023), we render the question \(q\) as the header of the image \(I\) for visual document understanding tasks and do not provide the question through a textual input channel. We take 
 and <answer> as separators, and use the following encoding format for the decoder sequence: [_question_] 
 [_rationale_] <answer> [_answer_]. As the decoder sequence length of Pix2Struct is 128 tokens,2 we trim the sequence before [_answer_] to 108 tokens and leave 20 tokens for the answer.

Footnote 2: Defined using Pix2Struct’s tokenizer.

If the rationale has programs, like in ChartQA, we put both the structured table and the program in the [_rationale_] slot, using the format [_rationale_] = [_table_] <program> [_program_]. As the structured table is usually long, we trim the sequence before [program] to 64, leaving 44 tokens for the program.

During inference, we evaluate only on the original, non-cropped images with greedy decoding. To avoid generating answer None, we force the model to decode non-None after the answer token.

Note that student model's intermediate predictions are relatively short. The overall floating-point operations (FLOPs) compared to a baseline model that directly generates answers are increased by less than 1% (see Appendix D for a derivation).

## 4 Experimental results

We study the impact of rationale distillation across three benchmarks, analyze the contribution of each component of our approach, and the extent to which

\begin{table}
\begin{tabular}{l l r r r r r r r r} \multirow{2}{*}{Model} & \multirow{2}{*}{Method} & \multicolumn{3}{c}{Dev} & \multicolumn{5}{c}{Test} \\ \cline{3-10}  & & InfoVQA & DocVQA & ChartQA & InfoVQA & DocVQA & ChartQA & aug. & human \\ \hline \multirow{3}{*}{Base} & Ans-Only & 36.8 & 72.3 & 75.9 & 34.3 & 38.2 & 72.1 & 81.5 & 30.3 \\  & QID & 38.2 & 75.5 & 76.2 & 35.4 & 39.5 & 75.7 & 82.3 & 32.5 \\  & RD (Ours) & **41.3** & **76.3** & **78.9** & **36.7** & **42.2** & **76.7** & **84.8** & **38.0** \\ \cline{1-1}  & Oracle & 48.1 & 82.5 & 84.7 & 43.1 & - & - & - & - \\ \hline \multirow{3}{*}{Large} & Ans-Only & 39.6 & 76.0 & 77.3 & 36.3 & 40.0 & 76.6 & 83.8 & 35.2 \\  & QID & 41.0 & 77.8 & 78.5 & 37.8 & 41.9 & 77.9 & 85.0 & 35.9 \\  & RD (Ours) & **43.5** & **79.2** & **81.6** & **39.3** & **44.3** & **79.0** & **88.6** & **40.6** \\ \cline{1-1}  & Oracle & 53.5 & 84.0 & 85.8 & 46.5 & - & - & - & - \\ \hline (\geq\$B) & SOTA & - & - & - & - & \({}^{\dagger}\)62.4 & \({}^{\ddagger}\)88.6 & \({}^{\ddagger}\)91.0 & \({}^{\ddagger}\)67.6 \\ \end{tabular}
\end{table}
Table 2: Pix2Struct-based results on three benchmarks. We show Rationale Distillation consistently outperforms the Ans-Only and QID baselines on both Base and Large models. Results marked by \({}^{\dagger}\) are from Chen et al. (2023c), and ones marked by \({}^{\ddagger}\) are from Liu et al. (2023a).

 

a single student model can match the capabilities of the external tools and LLMs it learns from.

### Dataset metrics

InfoVQA and DocVQA use the average normalized Levenshtein similarity (ANLS) score as the evaluation measure. ChartQA uses relaxed accuracy (RA) and includes an easier augmented evaluation set and a harder human-generated evaluation set.

#### Baselines

Pix2StructWe compare with the original Pix2Struct fine-tuning approach for both Base and Large models, where the model takes in an image \(I\) with the question rendered as a header as encoder input and directly predicts \(a\).

QidFine-tuning tasks QRA and QRACI predict the question as part of the decoder output. To detect improvements due to reading out the question as an intermediate step, we compare to the question-in-decoder (QID) setup, where the Pix2Struct model takes in \(I\) in the encoder input and predicts the sequence \(q,a\) separated by <answer>.

OracleTo establish an upper bound on performance of the student model if it was able to condition on the tool-generated high-quality rationales, we also compare to an oracle method on the development set. We use the tool generated rationale \(r\) during evaluation to get an oracle measure that uses information about the gold answer \(a\).

We also describe other existing VDU approaches and compare RD to them in Table 9 (Appendix B).

### Main results

Table 2 evaluates our rationale distillation (RD) method against baselines.

Overall trendsOverall, RD shows consistent improvements on InfoVQA (4.0 and 4.3 points), DocVQA (4.6 and 2.4 points) and ChartQA-human (7.7 and 5.4 points) test sets for both base and large model variants (respectively) over the Pix2Struct baseline. We also see that including the question in the decoder brings benefits across all datasets and variants. Next, we discuss the value of rationale distillation in comparison to this stronger QID baseline.

Textual rationalesTable 2 shows consistent improvements due to OCR and LLM-Summarizer rationales compared to the QID baseline. RD records improvements of 2.7 and 2.4 points on InfoVQA and 1.0 and 1.1 points on DocVQA for base and large variants respectively for the test set.

Table and program rationalesOn ChartQA, we use rationales including Plot-to-Table (underlying tables for charts), as well as programs derived by LLM-Programmer (based on this table and OCR). Using such rationales results in improvements of 2.5 and 3.6 points respectively on base and large variants on the augmented set over the QID baseline. We see even larger improvements: 5.3 and 4.7 points for base and large models, respectively, on the harder human eval set which requires more complex mathematical reasoning.

Accuracy and efficiency trade-off.We show that efficiency and accuracy can be improved at the same time. The performance of the Base model with RD is better than that of the Large model with Ans-Only; the inference FLOPs of the former (\({\sim}\)2.65E+12) are also lower than those of the latter (\({\sim}\)4.63E+12; Appendix D shows a derivation).

On the other hand, Pix2StructLarge with RD still shows gaps compared to the SOTA methods -- PaLl-3 with OCR [8] on InfoVQA and DocVQA, and a tool use case with deplotting and prompted LLM [10]. It is worth noting that these methods use more than 10 times the FLOPs of the Pix2StructLarge model and also use more data.

### Analysis

Using Base-sized models, we first explore two simple inference method variants that can improve model performance. We then ablate the impact of the different tasks designed to drive student model learning, and compare RD to additional baselines using cropping-based augmentations and tool-generated rationales in alternate ways. Finally, we analyze the quality of our student-generated rationales relative to ones predicted by external tools, and examine the types of questions that benefit most from Rationale Distillation.

\begin{table}
\begin{tabular}{l r r r r} \hline \multirow{2}{*}{Method} & \multicolumn{3}{c}{Dev Set} \\ \cline{2-5}  & InfoVQA & DocVQA & \begin{tabular}{c} ChartQA \\ aug. \\ \end{tabular} & 
\begin{tabular}{c} human \\ \end{tabular} \\ \hline RD & 41.3 & 76.3 & 78.9 & 36.7 \\ RD+Voting & 41.7 & 76.6 & 79.4 & 37.0 \\ \hline \end{tabular}
\end{table}
Table 3: RD on Pix2StructBase with voting during inference. Decoding with voting shows small but consistent improvements across datasets.

 

Top-\(n\) voting in inference.We can naturally apply top-\(n\) voting during inference, which is similar to making predictions using self-consistency in chain-of-thought [23]. We simply perform beam search decoding with a beam size of \(n=5\) and aggregate the probabilities of the distinct answers appearing in these hypotheses. We choose the answer (that is not None) with the highest aggregate probability as the final prediction. From Table 3, we see that this leads to small but consistent improvements across datasets, albeit at an increased computation cost.

What if we use an external calculator on the generated programs?Using a calculator - an additional but computationally inexpensive tool - could further enhance the capabilities of our models. For valid programs generated by student models, we use a calculator to carry out computations dictated by the programs, and take the output of the calculator to replace model output. For invalid programs, we keep using the model generated answer prediction. We observe that on ChartQA, the calculator use, when combined with voting, leads to further improvements of 0.4 RA on the augmented set and 3.3 RA on the human set.

How much does each of the tasks aid the student in predicting helpful rationales?In Table 4, we tease apart the contribution of each training task. First, we see that a model which uses standard supervised training with QRA _(i.e._, predicting the question, rationale and answer) performs worse than the QID baseline. This result suggests that it is important to make the student model robust to its own errors and expose it to rationales with varying degrees of relevance to the question.

Augmenting QRA with ASR (training with predicted rationales) results in a gain of about 1.9 points absolute (row #3). The additional image, rationale and answer examples obtained through image cropping and verifier categorization bring further improvements of 1.2 points (row #6).

Cropping-based augmentation is substantially less beneficial without rationales.As a traditional data augmentation method in vision, cropping may improve the accuracy of visual document understanding independent of the usage of intermediate rationales. We perform experiments to understand the gain from cropping in the baseline systems. We compare cropping augmentation with and without rationales using Pix2Struct-base on InfoVQA and show the gains with rationales are substantially higher. For setting without rationales, we study both the Ans-Only and QID setups.

Under the Ans-Only setup, we use one best (scored by pix2struct-infovqa-large) cropped image per example for augmentation, adding roughly the same amount of data as in the rationale augmentation setup. The performance with augmentation using cropped images is 37.1 on the InfoVQA dev set, which indicates a small improvement of 0.3 over the analogous setting without augmentation (w/o augmentation Ans-Only is 36.8).

Under the QID setup, we generate augmentation examples using all crops of the input images (\(\sim\)4 cropped images per original image); we then form a training set which is a mixture of the original examples and the cropped image examples. We perform a grid search over downsampling ratios for the cropped examples at 2x, 4x (roughly the same amount of data as rationale augmentation), 8x, and 16x. Results for QID without augmentation via cropping are 38.2, and the best results with augmentation was 38.4 with 8x downsampling. The results for all settings were: 37.5, 38.0, 38.4, and 38.3 for the 2x, 4x, 8x, and 16x downsampling configurations, respectively.

These findings suggest that cropping augmentation without rationales leads to a small improvement, and low-quality augmentation may lead to losses. The gains from augmentation for RD are substantially higher.

Directly distilling the tools via multi-task training is beneficial, albeit less effective than RD. Prior work has predicted teacher rationales as an auxiliary task in a multi-task training setting

\begin{table}
\begin{tabular}{c l c} \hline \hline \# & Task & InfoVQA Dev Set \\ \hline
1 & QRA & 36.7 \\ \hline
2 & QID & 38.2 \\ \hline
3 & QRA and ASR & 40.1 \\
4 & QRA, ASR and QRACI & 41.0 \\
5 & QRA, ASR and ALRCI & 40.5 \\
6 & All 4 tasks & 41.3 \\ \hline \hline \end{tabular}
\end{table}
Table 4: We conduct ablation study of different student training task combinations on the InfoVQA dev set: Question, Rationale, Answer (QRA), Answer with Provided Rationale (APR) and analoguous tasks on Cropped Images (CI). We show the importance of both training to predict the gold rationales and training to predict the answer based on the noisy rationales (row # 3), as well as the usefulness of image cropping augmentation (row # 6).

 

(Hsieh et al., 2023), while predicting the answer directly during inference time. Here we evaluate whether multi-task training is helpful and whether predicting rationales at inference time brings additional value. We experiment with multitask training on InfoVQA for base-sized Pix2Struct models. Here we do not use data augmentation via cropping. The training tasks are to predict the answer, the question, and the text evidence derived from OCR and LLM-Summarizer. At inference, we predict the answer only. The tasks take the image (including the rendered question), denoted as i+q, and an instruction and produce corresponding outputs, as follows: (1) given i+q and instruction "ans", predict the answer; (2) given i+q and the instruction "ques", predict the question; (3) given i+q and the instruction "or", predict the text evidence. During inference, we use the instruction from task (1).

We experiment with combinations of tasks and sampling data proportions. The ablation results in Table 5 shows the effectiveness of distilling the tools with multitask training. The baseline that uses a single task predicting the answer obtains ANLS 36.8 (as shown in Table 2), and multi-task training improves performance to 38.7 for the best mixture, while not adding any inference-time cost. However performance when predicting rationales at inference time (RD) is higher, at 40.1 ANLS.

What is the usefulness of the rationale generated by the student in comparison to external tools?On InfoVQA, we analyze the usefulness of the student-generated rationale in comparison to evidence from the OCR tool and several ways to sub-select fragments of similar length from it including LLM-Summarizer without access to gold answer (based on PaLM 2-L) (Figure 4). The systems are shown (from left to right) in order of increasing computation costs and engineering complexity. All methods except QID are evaluated with Pix2Struct-Base trained with RD, using corresponding rationales as decoder input during inference.

Since OCR outputs can be very long, we experiment with different methods for selecting 50-token segments. The simplest variant truncates the OCR output to the first segment of 50 tokens which results in a small gain (1.6 points) over QID. More complex methods which select segments based on TF-IDF (7.2 points) or BERT-embedding based (6 points) similarity to the question result in larger gains. Finally, the rightmost red bar shows the performance with rationales from few-shot prompted PaLM 2-L. For this experiment, we modified the prompting template for PaLM 2-L, to generate rationales from the OCR without being given the answer. Specifically, we ask PaLM 2-L to predict the evidence first, and the answer next and use only the evidence (and not the answer) from PaLM 2-L as student decoder input. This variant performs the best with a 7.7 point improvement over QID. Overall, these results indicate a significant room for improvement in rationale prediction for student models. We also see that an external OCR tool would still provide benefits at the cost of added computation by the OCR system and, since OCR is relatively efficient, the more significant cost of increased engineering complexity and potential service fees for production solutions.

Breakdown analysis of the improvement.The InfoVQA leaderboard provides a breakdown of model performance over subsets categorized by answer type, evidence type, and question operations. We compare the performance of Ans-Only models (ANLS 38.2) and RD+Voting (ANLS 42.2) in

\begin{table}
\begin{tabular}{l c r} Train tasks & Weight per task & Dev ANLS \\ \hline ans/ques/ocr & 1:1:1 & 37.9 \\ ans/ques/ocr & 4:1:1 & 38.4 \\ ans/ques/ocr & 8:1:2 & 38.7 \\ ans/ques & 2:1 & 37.0 \\ ans/ocr & 2:1 & 38.2 \\ \hline \end{tabular}
\end{table}
Table 5: Distilling the OCR directly with multitask training on InfoVQA is beneficial, but worse than RD.

Figure 4: We analyze the usefulness of student generated rationales. The systems are shown in order of increasing engineering complexity. All red bars use a pipeline with Google Cloud OCR during inference. RD trades off between accuracy and complexity.

 

Table 6. We observe large improvements when answers are text spans in the image or in the question. The former type indicates the helpfulness of the intermediate rationales; the latter suggests the helpfulness of decoding the question before answering.

We see a 7.1 point gain when the evidence comes from a table or list, 8 points when the evidence comes from text, which implies the student can extract better rationales from such parts of the images, in comparison to parts with more complex layouts such as figures and maps.

We did not use programs as rationales for InfoVQA, and we do not see large improvement on arithmetic and counting questions. Using programs as parts of the rationales in this and other types of tasks is a promising direction for future work.

## 5 Related work

Using tools to augment the input in a prediction problem can be seen as using additional reasoning steps, _i.e._, calling a tool with a set of arguments and integrating its result with the rest of the context. Much prior work on VDU has relied on calling OCR Tang et al. (2023); Appalaraju et al. (2021); Huang et al. (2022), object detector Kim et al. (2023), or de-plotting tools Liu et al. (2023a). Such works have not attempted to recognize text or structured data as an intermediate reasoning step using the same small model.

On the other hand, the specific structure of reasoning chains through prompting LLMs has been shown to have significant impact Wei et al. (2022); Zhou et al. (2023); Khot et al. (2023); Yao et al. (2023). Distilling these text rationales from large teacher models has been shown successful by chain-of-thought distillation works on NLP benchmarks Shridhar et al. (2023); Li et al. (2023) and ScienceQA Zhang et al. (2023); Wang et al. (2023). Toolformer Schick et al. (2023) trains smaller language models to call tools. Generic multimodal tool use solutions based on LLMs have also been proposed Yang et al. (2023). However, these works do not replicate the results of tool output and replace them for efficiency.

We marry the powerful ideas of taking intermediate reasoning steps from tools for accuracy, and distilling to small student models for efficiency, as we have proposed in RD.3

Footnote 3: We overview more related works in Appendix C.

## 6 Conclusions

We showed that the visual document understanding ability of small image-to-text models can be improved by our proposed Rationale Distillation. In RD, we obtain rationales for training examples using external tools and LLMs, and train small end-to-end student models to predict rationales as intermediate reasoning steps. We demonstrated the importance of designing student training tasks that make the model robust to irrelevant rationales.

RD leads to substantial improvements via textual evidence distillation on the text-heavy InfoVQA & DocVQA datasets, and via Plot-to-Table and program distillation on the numerical reasoningsfocused ChartQA dataset. Analysis shows the gains transfer to stronger models such as MatCha (Appendix B) and larger Pix2Struct models. Marginalizing over rationales and using a cheap calculator tool at inference time bring additional consistent benefits. Controlled experiments show that RD offers a tradeoff between performance and computational cost/engineering complexity, in comparison to systems relying on tool pipelines.

### Limitations

Our study shows RD can teach small models to successfully generate and utilize two types of rationales: summarized OCR evidence, and structured table concatenated with a simple program. A broader set of tools, such as object detection, image segmentation and captioning tools, can be further explored as rationales to enhance the ability of visual document understanding.

To use resources sparingly, we evaluate on the Pix2Struct series of models up to a size of 1.3B parameters (including the stronger MatCha

\begin{table}
\begin{tabular}{l c c c c c c c c c c c c} Method & \multicolumn{4}{c}{Answer type} & Evidence \\  & Image & Question & Multiple & Non & Table/ & Textual & Visual & Figure & Map & Comparison & Arithmetic & Counting \\  & span & span & spans & Span & List & & object & & & & & \\ \hline Ans-Only & 41.5 & 43.8 & 16.6 & 30.1 & 33.5 & 49.7 & 23.8 & 36.3 & 32.6 & 23.4 & 40.4 & **18.9** \\ RD+Voting & **46.6** & **46.7** & **18.8** & **30.4** & **40.6** & **57.7** & **28.0** & **37.9** & **36.5** & **28.1** & **41.2** & 17.7 \\ \hline \end{tabular}
\end{table}
Table 6: Breakdown ANLS score on different types of questions and answers from InfoVQA test set. RD benefits questions related to text or table evidence most.

 

model; see Appendix B). In the future, RD could also be evaluated on other more powerful pre-trained models for visual document understanding, such as PaLI-3 [25] or ERNIE-Layout [146].

We focus on single-page visual document understanding, and have not explored the potential of RD on multi-page images. Multi-page image problems may have longer-distance dependencies, and require student models to generate more complex rationales as the intermediate reasoning steps.

We inherit the ethical concerns of existing LLMs and multimodal models, such as privacy considerations and potential misuse. Here we use public peer-reviewed datasets to evaluate our method. For use in deployed applications, the data for RD should be constructed with careful data curation. Privacy-sensitive documents which contain personal information, should be excluded from the training data to prevent potential privacy breaches and unintended consequences.

## Acknowledgements

We are grateful to Adam Fisch, Kenton Lee, Peter Shaw, Yi Luan, Julian Eisenschlos, Ming-Wei Chang, Fangyu Liu, Hexiang Hu, and other members of the Google research community for helpful discussion and feedback on this work.

## References

* document understanding by language-image network. In Empirical Methods in Natural Language Processing (EMNLP), Cited by: SS1, SS2.
* R. Anil, A. M. Dai, O. Firat, M. Johnson, D. Lepikhin, A. Passos, S. Shakeri, E. Taropa, P. Bailey, Z. Chen, et al. (2023) PALM 2 technical report. arXiv preprint arXiv:2305.10403. Cited by: SS1, SS2.
* S. Appalaraju, B. Jasani, B. Urala Kota, Y. Xie, and R. Manmatha (2021) DocFormer: end-to-end transformer for document understanding. In 2021 IEEE/CVF International Conference on Computer Vision, ICCV 2021, External Links: Link, Document Cited by: SS1, SS2.
* W. Chen, X. Ma, X. Wang, and W. W. Cohen (2023) Program of thoughts prompting: disentangling computation from reasoning for numerical reasoning tasks. Transactions on Machine Learning Research. Cited by: SS1, SS2.
* X. Chen, J. Djolonga, P. Padlewski, B. Mustafa, S. Changpinyo, J. Wu, C. Riquelme Ruiz, S. Goodman, X. Wang, Y. Tay, S. Sakheri, M. Dehghani, Daniel Salz, M. Lucic, M. Tschannen, N. Nagrani, L. Hu, M. Joshi, B. Pang, C. Montgomery, P. Pietrzyk, M. Ritter, A. Piergivanni, M. Minderer, F. Pavetic, A. Waters, G. Li, I. Alabdulmohsin, L. Beyter, J. Amelot, K. Lee, A. P. Steiner, Y. Li, D. Keysers, A. Arnab, Y. Xu, K. Rong, A. Kolesnikov, M. Seyedhosseini, A. Angelova, X. Zhai, N. Houlsby, and R. Soricut (2023) PaLI-X: on scaling up a multilingual vision and language model. arXiv preprint arXiv:2305.18565. Cited by: SS1, SS2.
* X. Chen, X. Wang, L. Beyer, A. Kolesnikov, J. Wu, P. Voigtlaender, B. Mustafa, S. Goodman, I. Alabdullmohsin, P. Padlewski, D. Salz, X. Xiong, D. Vlasic, F. Pavcitek, K. Rong, T. Yu, D. Keysers, X. Zhai, and R. Soricut (2023) PaLI-3 vision language models: smaller, faster, stronger. arXiv preprint arXiv:2310.09199. Cited by: SS1, SS2.
* M. Elbayad, J. Gu, E. Grave, and M. Auli (2020) Depth-adaptive transformer. In 8th International Conference on Learning Representations, ICLR 2020, Cited by: SS1, SS2.
* C. Hsieh, C. Li, C. Yeh, H. Nakhost, Y. Fujii, A. Ratner, R. Krishna, C. Lee, and T. Pfister (2023) Distilling step-by-step! outperforming larger language models with less training data and smaller model sizes. In Findings of the Association for Computational Linguistics: ACL 2023, Cited by: SS1, SS2.
* Y. Huang, T. Lv, L. Cui, Y. Lu, and F. Wei (2022) LayoutLMv3: pre-training for document AI with unified text and image masking. In Proceedings of the 30th ACM International Conference on Multimedia, Cited by: SS1, SS2.
* T. Khot, H. Trivedi, M. Finlayson, Y. Fu, K. Richardson, P. Clark, and A. Sabharwal (2023) Decomposed prompting: a modular approach for solving complex tasks. In The Eleventh International Conference on Learning Representations (ICLR), Cited by: SS1, SS2.
* G. Kim, H. Lee, D. Kim, H. Jung, S. Park, Y. Kim, S. Yun, T. Kil, B. Lee, and S. Park (2023) Visually-situated natural language understanding with contrastive reading model and frozen large language models. In Empirical Methods in Natural Language Processing (EMNLP), Cited by: SS1, SS2.

 

Kenton Lee, Mandar Joshi, Iulia Turc, Hexiang Hu, Fangyu Liu, Julian Eisenschlos, Urvashi Khandelwal, Peter Shaw, Ming-Wei Chang, and Kristina Toutanova. 2023. Pix2Struct: Screenshot parsing as pretraining for visual language understanding. In The 40th International Conference on Machine Learning (ICML).
* Li et al. (2023) Liumian Harold Li, Jack Hessel, Youngjae Yu, Xiang Ren, Kai-Wei Chang, and Yejin Choi. 2023. Symbolic chain-of-thought distillation: Small models can also "think" step-by-step. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (ACL).
* Liu et al. (2023a) Fangyu Liu, Julian Martin Eisenschlos, Francesco Piccinno, Syrine Krichene, Chenxi Pang, Kenton Lee, Mandar Joshi, Wenhu Chen, Nigel Collier, and Yasemin Altun. 2023a. Deplot: One-shot visual language reasoning by plot-to-table translation. In Findings of the 61st Annual Meeting of the Association for Computational Linguistics.
* Liu et al. (2023b) Fangyu Liu, Francesco Piccinno, Syrine Krichene, Chenxi Pang, Kenton Lee, Mandar Joshi, Yasemin Altun, Nigel Collier, and Julian Martin Eisenschlos. 2023b. Matcha: Enhancing visual language pretraining with math reasoning and chart derendering. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (ACL).
* Luo et al. (2021) Junyu Luo, Zekun Li, Jinpeng Wang, and Chin-Yew Lin. 2021. Chartocr: Data extraction from charts images via a deep hybrid framework. In Winter Conference on Applications of Computer Vision (WACV).
* Masry et al. (2022) Ahmed Masry, Xuan Long Do, Jia Qing Tan, Shafiq Joty, and Enamul Hoque. 2022. ChartQA: A benchmark for question answering about charts with visual and logical reasoning. In Findings of the Association for Computational Linguistics: ACL 2022.
* Masry et al. (2023) Ahmed Masry, Parsa Kavehzadeh, Xuan Long Do, Enamul Hoque, and Shafiq Joty. 2023. Unichart: A universal vision-language pretrained model for chart comprehension and reasoning. In Empirical Methods in Natural Language Processing (EMNLP).
* Mathew et al. (2022) Minesh Mathew, Viraj Bagal, Ruben Tito, Dimosthenis Karatzas, Ernest Valveny, and CV Jawahar. 2022. Infographicvqa. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision.
* Mathew et al. (2021) Minesh Mathew, Dimosthenis Karatzas, and CV Jawahar. 2021. Docvqa: A dataset for vqa on document images. In Proceedings of the IEEE/CVF winter conference on applications of computer vision (WACV).
* Peng et al. (2022) Qiming Peng, Yinxu Pan, Wenjin Wang, Bin Luo, Zhenyu Zhang, Zhengjie Huang, Yuhui Cao, Weicheng Yin, Yongfeng Chen, Yin Zhang, Shikun Feng, Yu Sun, Hao Tian, Hua Wu, and Haifeng Wang. 2022. ERNIE-layout: Layout knowledge enhanced pre-training for visually-rich document understanding. In Findings of the Association for Computational Linguistics: EMNLP 2022.
* Perot et al. (2023) Vincent Perot, Kai Kang, Florian Luisier, Guolong Su, Xiaoyu Sun, Ramya Sree Boppana, Zilong Wang, Jiagi Mu, Hao Zhang, and Nan Hua. 2023. LDDX: Language model-based document information extraction and localization. arXiv preprint arXiv:2309.10952.
* Pope et al. (2023) Reiner Pope, Sholo Douglas, Akanksha Chowdhery, Jacob Devlin, James Bradbury, Jonathan Heek, Kefan Xiao, Shirani Agrawal, and Jeff Dean. 2023. Edificently scaling transformer inference. Proceedings of Machine Learning and Systems.
* Powalski et al. (2021) Rafal Powalski, Lukasz Borchmann, Dawid Jurkiewicz, Tomasz Dwojak, Michal Pietruszka, and Gabriela Palka. 2021. Going full-tilt boogie on document understanding with text-image-layout transformer. In International Conference on Document Analysis and Recognition. Springer.
* Ross et al. (2011) Stephane Ross, Geoffrey Gordon, and Drew Bagnell. 2011. A reduction of imitation learning and structured prediction to no-regret online learning. In Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics, Proceedings of Machine Learning Research.
* Schick et al. (2023) Timo Schick, Jane Dwivedi-Yu, Roberto Dessi, Roberta Raileanu, Maria Lomeli, Luke Zettlemoyer, Nicola Cancedda, and Thomas Sciolom. 2023. Toolformer: Language models can teach themselves to use tools. In Advances in Neural Information Processing Systems (NeurIPS).
* Shazeer and Stern (2018) Noam Shazeer and Mitchell Stern. 2018. Adafactor: Adaptive learning rates with sublinear memory cost. In Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Proceedings of Machine Learning Research.
* Shridhar et al. (2023) Kumar Shridhar, Alessandro Stolfo, and Mrinnaya Sachan. 2023. Distilling reasoning capabilities into smaller language models. In Findings of the Association for Computational Linguistics: ACL 2023.
* Tafjord et al. (2019) Oyyind Tafjord, Peter Clark, Matt Gardner, Wen-tau Yih, and Ashish Sabharwal. 2019. QUAREL: A dataset and models for answering questions about qualitative relationships. In The Thirty-Third AAAI Conference on Artificial Intelligence.
* Talmor et al. (2019) Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. 2019. CommonsenseQA: A question answering challenge targeting commonsense knowledge. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics (NAACL).
* Tang et al. (2023) Ziening Tang, Ziyi Yang, Guoxin Wang, Yuwei Fang, Yang Liu, Cheng Guang Zhu, Michael Zeng, Cha Zhang, and Mohit Bansal. 2023. Unifying vision, text, and layout for universal document processing. In Conference on Computer Vision and Pattern Recognition (CVPR).

 

Jianfeng Wang, Zhengyuan Yang, Xiaowei Hu, Linjie Li, Kevin Lin, Zhe Gan, Zicheng Liu, Ce Liu, and Lijuan Wang. 2022. GIT: A generative image-to-text transformer for vision and language. _Transactions on Machine Learning Research_.
* Wang et al. (2023a) Lei Wang, Yi Hu, Jiabang He, Xing Xu, Ning Liu, Hui Liu, and Heng Tao Shen. 2023a. T-sciq: Teaching multimodal chain-of-thought reasoning via large language model signals for science question answering. In _Thirty-Eighth AAAI Conference on Artificial Intelligence (AAAI-24)_.
* Wang et al. (2023b) Peifeng Wang, Zhengyang Wang, Zheng Li, Yifan Gao, Bing Yin, and Xiang Ren. 2023b. SCOTT: Self-consistent chain-of-thought distillation. In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (ACL)_.
* Wang et al. (2023c) Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2023c. Self-consistency improves chain of thought reasoning in language models. In _The Eleventh International Conference on Learning Representations (ICLR)_.
* Wei et al. (2022) Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. 2022. Chain-of-thought prompting elicits reasoning in large language models. In _Advances in Neural Information Processing Systems (NeurIPS)_.
* Yang et al. (2023) Rui Yang, Lin Song, Yanwei Li, Sijie Zhao, Yixiao Ge, Xiu Li, and Ying Shan. 2023. GPT4Tools: Teaching llm to use tools via self-instruction. _arXiv preprint arXiv:2305.18752_.
* Yao et al. (2023) Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L. Griffiths, Yuan Cao, and Karthik R Narasimhan. 2023. Tree of thoughts: Deliberate problem solving with large language models. In _Thirty-seventh Conference on Neural Information Processing Systems_.
* Ye et al. (2023) Jialo Ye, Anwen Hu, Haiyang Xu, Qinghao Ye, Ming Yan, Guohai Xu, Chenliang Li, Junfeng Tian, Qi Qian, Ji Zhang, Qin Jin, Liang He, Xin Alex Lin, and Fei Huang. 2023. UReader: Universal OCR-free visually-situated language understanding with multimodal large language model. In _Findings of the Association for Computational Linguistics: EMNLP 2023_.
* Yih et al. (2016) Wen-tau Yih, Matthew Richardson, Chris Meek, Ming-Wei Chang, and Jina Suh. 2016. The value of semantic parse labeling for knowledge base question answering. In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (ACL)_.
* Zhang et al. (2023) Zhuosheng Zhang, Aston Zhang, Mu Li, Hai Zhao, George Karypis, and Alex Smola. 2023. Multimodal chain-of-thought reasoning in language models. _arXiv preprint arXiv:2302.00923_.
* Zhou et al. (2023) Denny Zhou, Nathanael Scharli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuurmans, Claire Cui, Olivier Bousquet, Quoc Le, and Ed Chi. 2023. Least-to-most prompting enables complex reasoning in large language models. In _The Eleventh International Conference on Learning Representations (ICLR)_.
* Zhu et al. (2023) Wang Zhu, Jesse Thomason, and Robin Jia. 2023. Chain-of-questions training with latent answers for robust multistep question answering. In _In Empirical Meth-ods in Natural Language Processsin (EMNLP)_.

 

Implementation details

### Description of used tools

OCRFor all datasets, we begin with calling an off-the-shelf external OCR tool (Google Cloud OCR), which takes the image as input and outputs the full text recognized in the image together with location information (see Figure 2).

LLM-SummarizerOCR outputs can be quite long for some images, and not all text in the input image is directly relevant to a given question. To minimize computation spent on intermediate rationale prediction steps, we employ another powerful tool -- a prompted large language model PaLM 2-L Anil et al. (2023), to generate a significantly shorter span of text (less than 100 tokens), given the question, answer, and the full image OCR text (see Figure 2 top for an example). We sample a single evidence with temperature of 0.1 to obtain these rationales from PaLM 2-L.

Plot-to-TableIn addition to relevant text on the screen, some visual document domains and types of problems can benefit from other types of intermediate structure. An example is understanding charts and figures, whose underlying structured source data is not well captured by OCR systems. Such structured source data is available in some datasets, _e.g._, ChartQA provides structured data tables extracted by ChartOR Luo et al. (2021); but they can also be inferred for unannotated images through tools like DePlot Liu et al. (2023a).

LLM-ProgrammerFor problems involving numerical reasoning, we use a prompted LLM, PaLM 2-L, to generate a simple program capturing common numerical reasoning patterns corresponding to user queries, given the question, answer, the full image OCR text and the structured table (see Figure 2 lower half for an example). The programs are limited to the following formats: Div(a,b); Mul(a,b); Avg(a list of numbers); Sum(a list of numbers); Diff(a,b); Greater(a,b); Less(a,b); Find(str). All programs except Find(str) have execution steps in the prompt templates, which explain how to connect the programs to arithmetic and comparison operations. The last program type is applicable if numerical reasoning of the other types is not needed, and has no operation involved. Note that the program rationale is not executed by default, but is only used to guide the model towards the correct answer.

Multimodal-VerifierTo determine the helpfulness of the rationale generated by other tools and the relevance of image augmentations, we employ a multi-task trained, large multimodal model PaLIX 55B Chen et al. (2023b). We construct the text encoder input in the following format:

[rationale] Answer in en: [question]

The verifier takes in the image \(I\) as input to the vision encoder, the question \(q\) and the rationale \(r\) as input to the text encoder. We use the log-probability of the gold answer (with and without conditioning on the rationale), and the correctness of the predicted answer (through greedy decoding), to define two measures of rationale helpfulness.

### Algorithm for rationale augmentation

Here we list the detailed algorithm for rationale augmentation described in SS3.2.

```
1:Input:image \(I\); question \(q\); answer \(a\); tools for rationale generation.
2:Output:a set of cropped images, and a corresponding set of rationales.
3:Initialize the counter \(j\gets 0\), the cropped image set \({\cal I}\leftarrow\varnothing\) and the rationale set \({\cal R}\leftarrow\varnothing\).
4:Get the height \(h\) and the width \(w\) of the image \(I\).
5:if\(h\geq w\)then
6:while\(wj<h\)do
7:\(start\gets wj/2\)
8:\(end\leftarrow\min(wj/2+w,h)\)
9:image \(i_{j}\leftarrow\text{crop }[start,end]\) on the height of \(I\).
10: Get rationales \(r_{j}\) for \(i_{j},q,a\) from tools.
11:\({\cal I}\leftarrow{\cal I}\cup i_{j};{\cal R}\leftarrow{\cal R}\cup r_{j};j \gets j+1\).
12:endif while
13:else
14:while\(hj<w\)do
15:\(start\gets hj/2\)
16:\(end\leftarrow\min(hfj/2+h,w)\)
17:image \(i_{j}\leftarrow\text{crop }[start,end]\) on the width of \(I\).
18:Get rationales \(r_{j}\) for \(i_{j},q,a\) from tools.
19:\({\cal I}\leftarrow{\cal I}\cup i_{j};{\cal R}\leftarrow{\cal R}\cup r_{j};j \gets j+1\).
20:endwhile
21:endif
22:return\({\cal I},{\cal R}\) 



### Student rationale generation for ASR

For student rationale generation, we cannot directly use the student trained on the whole training set, as it is likely to remember and replicate the tool-generated rationale but this would not be representative of its behavior on unseen data.

On InfoVQA and DocVQA, we split the training data of into 3 folds. We train 3 student models, each takes in 2 folds as the train data and generates student rationale for the remaining fold. On ChartQA, to avoid the distribution shift from the augmented set and human set, we split both augmented set and the human set into 3 holds, in total 6 folds. We train 6 student models, each takes in 5 folds for training and generates student rationale for the remaining fold.

Here, the student models are only trained to generate the question and the rationale, not the answer. The output format of the student models is

[question] 
 [rationale]

For each example, we sample 3 rationales to create the ASR training set.

### Hyper-parameters

Following the setup in Lee et al. (2023), for Pix2Struct-Base, we use an input sequence length of 6155 patches for InfoVQA, and 4096 patches for DocVQA and ChartQA. We train with a batch size of 128 for InfoVQA, and 256 for DocVQA and ChartQA, on 32 v3-Google Cloud TPUs.

For Pix2Struct-Large, we use an input sequence length of 3072 patches and train with a batch size of 64 for all datasets, on 64 v3-Google Cloud TPUs.

We train all the model with 10k steps, optimizing using Adafactor Shazeer and Stern (2018). The learning rate schedule uses a linear warmup of 1k steps to 0.01, followed by cosine decay to 0. On InfoVQA and DocVQA, we select the model with the best ANLS score on the dev set for evaluation. On ChartQA, we select the model with the best RA on the dev augmented set for test evaluation. We report all the results under a single-run setup.

### Cost of external tool calls

Many of the tools we used are publicly accessible, such as Google Cloud OCR4 and PaLM-2 text bison5. Based on the pricing, an estimate of the cost is approximately $150 for OCR and $400 for PaLM-2 to generate training data for student models.

Footnote 4: [https://cloud.google.com/use-cases/ocr?hl=en](https://cloud.google.com/use-cases/ocr?hl=en)

Footnote 5: [https://pypi.org/project/google-generativeai](https://pypi.org/project/google-generativeai)

### Scientific artifacts and licenses

We evaluate on three public datasets, InfoVQA, DocVQA and ChartQA, in our experiments. InfoVQA and DocVQA data is shared for non-commercial, research and educational purposes, which aligns with our use. ChartQA is under GNU General Public License v3.0. The questions in all three datasets are in English. We put the statistics of our evaluated datasets in Table 7.

We finetune public models Pix2Struct and MatCha. They are under Apache License 2.0.

## Appendix B Additional experimental analysis

### Model ablations

We show that RD also benefits stronger pretrained model such as Liu et al. (2023b), while decoupling rationale and answer prediction is harmful.

What if we use a stronger pretrained model tailored to math reasoning as in ChartQA?We initialize our student model parameters with MatChaLiu et al. (2023b) instead of Pix2Struct before finetuning with RD on ChartQA (Table 8). MatCha is based on Pix2Struct-Base but has stronger numerical reasoning and other abilities obtained through additional pretraining on relevant data. We see that RD leads to consistent improvements over stronger MatCha models specialized for this domain.

Decoupling rationale and answer predictionRD uses the same student model (with a single set of parameters \(\theta\)) to predict rationales and answers.

\begin{table}
\begin{tabular}{l l r r r} Dataset & Domain & Train & Dev & Test \\ \hline InfoVQA & Documents & 23,946 & 2,801 & 3,288 \\ DocVQA & Documents & 39,463 & 5,349 & 5,188 \\ ChartQA-human & Illustrations & 7,398 & 960 & 1,250 \\ ChartQA-aug. & 20,901 & 960 & 1,250 \\ \end{tabular}
\end{table}
Table 7: Statistics of the datasets we evaluate on.

\begin{table}
\begin{tabular}{l c r r r} Method & \multicolumn{2}{c}{ChartQA Dev Set} & \multicolumn{2}{c}{ChartQA Test Set} \\  & aug. & human & aug. & human \\ \hline Ans-Only & 83.5 & 40.4 & 88.5 & 36.6 \\ QID & 84.6 & 40.2 & 89.7 & 37.5 \\ RD & **86.0** & **40.9** & **90.8** & **42.1** \\ \hline \end{tabular}
\end{table}
Table 8: We initialize the student model with MatCha, which has stronger numerical reasoning skills. RD also improves MatCha for ChartQA.



In Figure 4, "Another Student" refers to using a student model, with a separate set of parameters, only responsible for rationale prediction. While training separate models for predicting different intermediate steps has been shown beneficial for ScienceQA Zhang et al. (2023), this configuration results in slightly worse performance on InfoVQA dev set. Moreover, it also adds engineering complexity, storage, and compute.

**Selecting appropriate rationales is important.** Instead of using a simple customized program, we construct the rationale for ChartQA by structured table concatenated with text evidence. The text evidence describes information in the figure that is relevant to the question and is predicted by PaLM 2-L given the question, answer, structured table, and OCR, but does not specify a program that can be executed to obtain the answer. For example, for the input in the lower half of Figure 2, the text evidence generated by PaLM 2-L in this setting is _"No confidence value in 2017 is 5, confidence value in 2017 is 93"_. The same RD training on evidence-based rationales achieves 83.4 / 33.0 RA on the ChartQA's augmented and human test sets, which is 1.4 / 5.0 points lower than the program-based rationales.

### Comparison to other approaches

We make an additional comparison to other approaches, which may have different setups, such as the use of tools or LLMs at inference time, or the use of additional pretraining, in Table 9. We show that except the powerful pretrained model PaLI-3 (5B parameters), RD is better than other approaches under the setup of pixel-level image-to-text model without the use of external tools at inference time.

UniChart Masry et al. (2023) is pretrained on chart-specific objectives, but on a larger corpus than MatCha. The pretraining data is augmented by knowledge distillation from LLMs. Without further pretraining, RD shows better performance on ChartQA, initialized with MatCha. DUBLIN Aggarwal et al. (2023) proposes pretraining objectives at four different levels: language, image, document structure, and question-answering. It demonstrates high performance on InfoVQA and DocVQA, at the cost of sacrificing the ability to understand charts. In addition, UReader Ye et al. (2023) designs a shape-adaptive cropping module to process high-resolution images. It is jointly fine-tuned on multiple VDU tasks with low-rank adaptation approach. Cream Kim et al. (2023) utilizes contrastive learning to align the visual representation of the image and text representation of OCR and objects (generated from tools). We show that RD is better than or close to Cream even under the setup where Cream uses tools in inference.

UDOP Tang et al. (2023) uses external OCR tool for text layout information at training and inference time. It is also pretrained on the IIT-CDIP scanned documents corpus, achieving great performance gains on InfoVQA and DocVQA.

### Qualitative analysis

We randomly select 5 examples in the dev set of InfoVQA to illustrate that tool generated rationales extract relevant information from the visual context, which are helpful to answer the question (Table 10).

We also randomly select 20 examples from the dev set of InfoVQA for a qualitatively analysis of student generated rationales (Table 12). The first five examples are for the same inputs as the tool-generated rationale examples. We observe that for 3 examples out of 5, the student generated rationales match the tool generated ones. In the table, we list the student generated and TF-IDF extracted rationales, along with the question and the ground truth answer. We compute the TF-IDF weight for each OCR block in the image, and measure the cosine similarity of the question to these OCR blocks. Starting from the closest OCR block to the question, we gradually add more OCR blocks to the final TF-IDF string until it reaches 50 tokens under Pix2Struct tokenizer. Note that this process is also applied to the TF-IDF and BERT embedding analysis in Figure 4.

For more than 50% of the student generated rationales, answers can be inferred from them without looking at the images. Also, 90% of the student generated rationales are relevant to the answer. It is possible for the student model to generate an irrelevant rationale, such as in the last row of Table 12, the student rationale (_27 % fake or empty 28 % inactive 43% good_) is irrelevant to the question (_Who uses the twitterid @Ev?_) as well as the answer (_twitter co-founder evan williams_). This observation verifies the importance of robustifying against student rationale errors during training 



### Validity of PaLI-X as a verifier

We choose PaLI-X for filtering as it was the SOTA before PaLI-3 on our evaluated benchmarks (without OCR pipeline). We have verified PaLI-X's filtering ability on two axes.

First, we checked if higher likelihood on PaLI-X with rationales as the input implies the prediction is more likely to be correct. For the correct predictions, \({\sim}75\%\) have log-likelihood greater than -0.25; for the wrong prediction, \({\sim}70\%\) have log-likelihood lower than -0.6, which shows PaLI-X is not over-confident on wrong predictions (see Figure 5 for details).

Second, we evaluate on a small scale whether if the relevance filter is satisfied, the generated rationales are more useful. We manually examined \({\sim}5\) examples and verified those rationales are actually useful. We give one example here where the PaLI-X prediction is wrong without the rationale, but correct with the rationale.

**Question**: What was the name of 'Snapchat' when it was launched?

**Rationale**: Snapchat, an image messaging app, is launched in September after being tested as an IOS only app known as Picaboo in July. One of the founding members is pushed out before Snapchat is released., SHORT VIDEOS, MEGA VIEWS,

Figure 5: The distribution of the log-likelihood of PaLI-X prediction; \(y\)-axis is the number of examples. Orange bars show correct predictions and blue bars show the wrong predictions.



Vine, a video hosting service specialising in short videos, goes public in June, Twitter acquires Vine for $30 million in the October of that year.

**PaLI-X's answer w/o rationale**: Snapchat

**PaLI-X's answer w/ rationale**: Picaboo

## Appendix C Extended related works

Here we summarize related research in text only and visual language understanding, focusing on methods using intermediate reasoning steps.

Tool use in visual language understandingUsing tools to augment the input in a prediction problem can be seen as using additional reasoning steps of specific type, _i.e._, calling a tool with a set of arguments and integrating its result with the rest of the context. Much prior work on visual document understanding has relied on an OCR component Tang et al. (2023); Appalaraju et al. (2021).

PaLI-X Chen et al. (2023b) and the smaller PaLI-3 model Chen et al. (2023c), which are image-and-text encoders paired with text decoders, achieve strong results both with and without additional OCR input. Since OCR extractions can be very long, _e.g._, InfoVQA has images with OCR more than 1k tokens, the recognized text often needs to be truncated to a given maximal token length given pretrained model assumed token limits and efficiency considerations. Other architectures are heavily centered on the recognized document text, with examples being TILT Powalski et al. (2021) and LayoutLM Huang et al. (2022).

In addition to OCR, de-plotting has been used as a pre-processing step to either augment or entirely replace the input image representation Liu et al. (2023a). Both object detection and OCR are used as an auxiliary input by cream Kim et al. (2023) to augment the vision feature.

Such works have not attempted to recognize text or structured data as an intermediate reasoning step using the same small model, as we have proposed in RD.

Tool use and chain-of-thought distillationDistilling text rationales from large teacher models has been shown successful by chain-of-thought distillation works Shridhar et al. (2023); Li et al. (2023); Wang et al. (2023b) on NLP benchmarks, such as CommonsenceQA Talmor et al. (2019) and QuaRel Tafjord et al. (2019).

MMCoC7 Zhang et al. (2023) and T-Sci Wang et al. (2023a) have utilized annotated or decomposed reasoning chains for improving vision-language reasoning on ScienceQA, which is not representative of the visual document understanding challenges we focus on (_e.g._ text-only models can reach accuracy of over 79% on this benchmark). In addition, these works only distill using our QRA task, which we show is insufficient to teach the student model to produce high-quality rationales and be robust to potential errors. We also use a single small model instead of two different models for rationale and answer generation, reducing complexity and engineering cost, and focus on short rationales for efficiency. Finally, we use a broader set of tools instead of just one LLM chain-of-thought tool.

Toolformer Schick et al. (2023) trains smaller language models to call tools. Generic multimodal tool use solutions based on LLMs have also been proposed Yang et al. (2023). However, these works do not replicate the results of tool output and replace them for efficiency.

Other related work on text-only models with intermediate reasoning stepsIntermediate reasoning in text-only models has been successful through prompting large language models to perform a chain-of-thought Wei et al. (2022). More traditionally in NLP, smaller models have been shown to be able to successfully learn to generate semantic parses before predicting final answers, including when such parses are not directly annotated in training data Yih et al. (2016). Decomposing intermediate questions is also known to help small models on multistep text question answering Zhu et al. (2023). Marginalizing over multiple intermediate rationale possibilities has brought consistent gains Wang et al. (2023c).

The specific structure of reasoning chains (which can be guided by tailored prompting strategies for LLMs) used has been shown to have significant impact Zhou et al. (2023); Khot et al. (2023); Yao et al. (2023). In addition to text as intermediate predictions, generating programs has also been shown useful Chen et al. (2023a).

## Appendix D Detailed FLOPs analysis

We show that RD only increases the FLOPs of the Base model on InfoVQA by around 1%, those of the Large model by around 2%, and uses around 10% the FLOPs of the SOTA model, as listed in Table 11.

We only consider the computation of transformer blocks of the encoder and the decoder, and ignore 

the small cost in the last linear layer for token generation. Most of the computation cost is from the attention and feed-forward layers, and we ignore the activation and normalization layers. Notice that matrix multiplication of with dimension \([N,P]\times[P,M]\) uses FLOPs of \(NM(2P-1)\); for simplicity, we use \(2NMP\) to approximate.

For each self-attention layer, we suppose an input sequence length of \(d_{q}\), a hidden size of \(d_{h}\). The query, key, value matrix computation takes \(6d_{q}d^{2}_{h}\), the multiplication of these three matrices takes \(4d^{2}_{q}d_{h}\), and the linear transformation towards the output takes \(2d_{q}d^{2}_{h}\). The total is \(8d_{q}d^{2}_{h}+4d^{2}_{q}d_{h}\).

For each cross-attention layer, we suppose a query input sequence length of \(d_{q}\), and a key-value input token sequence of \(d_{k}\). The query, key, value matrix computation takes \(2d_{q}d^{2}_{h}+4d_{k}d^{2}_{h}\), the multiplication of these three matrices takes \(4d_{q}d_{k}d_{h}\), and the linear transformation towards the output takes \(2d_{q}d^{2}_{h}\). The total is \(4d_{q}d^{2}_{h}+4d_{k}d^{2}_{h}+4d_{q}d_{k}d_{h}\).

For one feed-forward layer, suppose the sequence length from the attention layer is \(d_{q}\) and the hidden size from the attention layer is \(d_{h}\) and the feed-forward size is \(d_{f}\), the total computation is \(6d_{q}d_{f}d_{h}\) if gated activation is used, otherwise \(4d_{q}d_{f}d_{h}\).

Now we derive the formula of FLOPs for encoder-decoder models. We use \(d_{e}\) and \(d_{d}\) to denote the encoder sequence length, and the whole decoder sequence length, respectively. Given the models we discuss here all have same hidden dimension for the encoder and the decoder, we use \(d_{h}\) to denote the hidden size and \(d_{f}\) to denote the feed-forward size. For simplicity, we assume a batch size of 1. The computation cost of each encoder layer, denoted with FCE, is

\[\text{FCE}(d_{e},d_{h},d_{f}) =8d_{e}d^{2}_{h}+4d^{2}_{e}d_{h}+4d_{e}d_{f}d_{h}\] \[+2\llbracket\text{Gated}\rrbracket d_{e}d_{f}d_{h},\]

where \(\llbracket\text{Gated}\rrbracket\) is the indicator function on whether the model uses gated activation. Similarly, without caching the past attention matrices, the computation cost of each decoder layer, denoted with \(\text{FCD}_{\text{exact}}\), is

\[\text{FCD}_{\text{exact}}(d_{e},d_{d},d_{h},d_{f})=4d_{e}d^{2}_{ h}+\sum_{t=1}^{d_{d}}4d_{h}t^{2}\] \[+(12d^{2}_{h}+4d_{e}d_{h}+4d_{f}d_{h}+2\llbracket\text{Gated} \rrbracket d_{f}d_{h})t.\]

Notice the query, key, value matrices from the encoder output only have to be constructed once through the decoding time steps.

Instead, if we consider KV-caching and reusing the past attention matrices in the decoding (Pope et al., 2023), we can achieve the following at step \(t\):

* reuse the first \(t-1\) rows of the query, key, value matrices;
* reduce the matrix multiplication cost by a factor of \(t\) with block matrix computation;
* for both self-attention and cross-attention, we only have to care about the last row of the output matrix.

Given the decoding with caching, we reduce the computation cost of each decoder layer to \(\text{FCD}_{\text{approx}}\), written as

\[\text{FCD}_{\text{approx}}(d_{e},d_{d},d_{h},d_{f})=4d_{e}d^{2}_ {h}+\sum_{t=1}^{d_{d}}4d_{h}t\] \[+(12d^{2}_{h}+4d_{e}d_{h}+4d_{f}d_{h}+2\llbracket\text{Gated} \rrbracket d_{f}d_{h}).\]

This formula matches the one provided by Elbayad et al. (2020). For a \(N\)-layer encoder-decoder model, the total computation cost is \(N(\text{FCE}+\text{FCD}_{\text{approx}})\) and \(N(\text{FCE}+\text{FCD}_{\text{exact}})\) with and without caching, respectively.

Based on the formula derived above, we start to compute FLOPs for specific models. Taking InfoVQA as an example, the student generated rationales have \(41.8\) tokens on average, the questions have \(15.3\) tokens on average and the answers have \(5.0\) tokens on average.

#### Pix2Struct-Base

The model has \(N=12\), \(d_{h}=768\), \(d_{f}=2048\) and uses gated activation. For InfoVQA, we have \(d_{e}=6155\), \(d_{d}=5\) for answer-only generation, and \(d_{d}=62\) for RD generation (including the question, rationale, and the answer). Without caching, the total FLOPs computation is \(2.63\)E+\(12\) for answer-only generation, and \(3.46\)E+\(12\) for RD generation, resulting in a \(\sim 30\%\) increase of computation. With caching, the

\begin{table}
\begin{tabular}{l c} Method & FLOPs \\ \hline Plx2Struct-Base, Ans-only & \(2.62\)E+\(12\) \\ Plx2Struct-Base, RD & \(2.65\)E+\(12\) \\ \hline Plx2Struct-Large, Ans-only & \(4.63\)E+\(12\) \\ Plx2Struct-Large, RD & \(4.72\)E+\(12\) \\ \hline PaLI-3, w/o OCR & \(4.81\)E+\(13\) \\ \hline \end{tabular}
\end{table}
Table 11: FLOPs of evaluated approaches. RD only increase the FLOPs of Base model by around 1%, Large model by 2%, and uses less than 10% the FLOPs of the SOTA model.

 

total FLOPs computation is \(2.62\)E+\(12\) for answer-only generation, and \(2.65\)E+\(12\) for RD generation, resulting in a only \(\sim 1\%\) increase of computation.

Pix2Struct-LargeThe model has \(N=18\), \(d_{h}=1536\), \(d_{f}=3968\) and uses gated activation. Similarly, for InfoVQA, we have \(d_{e}=3072\), \(d_{d}=5\) for answer-only generation, and \(d_{d}=62\) for RD generation. With caching, the total FLOPs computation is \(4.63\)E+\(12\) for answer-only generation, and \(4.72\)E+\(12\) for RD generation, resulting in a only \(\sim 2\%\) increase of computation.

PaLi-3We also estimate FLOPs for PaLI-3 [Chen et al., 2023c], which is constructed by a 2B ViT-G/14 vision encoder and a 3B UL2 language encoder-decoder.

The vision encoder has \(N=48\), \(d_{h}=1536\), \(d_{f}=8192\), and does not use gated activation. For evaluating on InfoVQA, the model uses the resolution of 1064\(\times\)1064, which has \(d_{e}=5776\) patches. The FCE formula gives the computation cost of \(2.90\)E+\(13\).

The language encoder-decoder has \(N=24\), \(d_{h}=1024\), \(d_{f}=16384\), and uses gated activation. We consider the extra text tokens (\(15\) on average) from the question but not the ones from the OCR input. Hence, we have \(d_{e}\geq 5791\) and \(d_{d}=5\). With caching, the total computation cost of the UL2 language transformer is at least \(1.91\)E+\(13\).

Combing two parts, the 5B PaLI-3 model uses FLOPs of at least \(4.81\)E+\(13\) on the setup of the InfoVQA task, which is 10 times more than Pix2Struct-Large with the RD generation.

## Appendix E Prompt templates

We list the prompt templates for rationale generation on InfoVQA, DocVQA and ChartQA in Fig. 6, Fig. 7 and Fig. 8, respectively. The former two use 5-shot prompting for LLM-Summarizer and the last uses 8-shot prompting for LLM-Programmer.

 

* [10] M. C. B., and R. A. S., "Theoretical study of the effect of a new model on the influence of the social activity in the social network", in _Proceedings of the 20th International Conference on Social Networks_, pages 35-46, New York, NY, USA, 2007.
* [11] J. D., "A simple approach to the impact of social activity in social networks", in _Proceedings of the 20th international conference on Social Information Theory_, volume 89 of _Lecture Notes in Computer Science_, pages 1-10, Springer, Heidelberg, 2000.
* [12] P. F., "Social networks: The role of social activity in social networks", in _Proceedings of the 20th international conference on Social information theory_, volume 10 of _Lecture Notes in Computer Science_, pages 1-10, Springer, Heidelberg, 2000.
* [13] G. H., "Social networks: An introduction to social networks", in _Proceedings of the 20th international conference on Social information theory_, volume 10 of _Lecture Notes in Computer Science_, pages 1-10, Springer, Heidelberg, 2000.
* [14] K. E., "Social networks: The role of social activity in social networks", in _Proceedings of the 20th international conference on Social information theory_, volume 10 of _Lecture Notes in Computer Science_, pages 1-10, Springer, Heidelberg, 2000.
* [15] K. E., "Social networks: The role of social activity in social networks", in _Proceedings of the 20th international conference on Social information theory_, volume 10 of _Lecture Notes in Computer Science_, pages 1-10, Springer, Heidelberg, 2000.
* [16] K. E., "Social networks: The role of social activity in social networks", in _Proceedings of the 20th international conference on Social information theory_, volume 10 of _Lecture Notes in Computer Science_, pages 1-10, Springer, Heidelberg, 2000.
* [17] K. E., "Social networks: The role of social activity in social networks", in _Proceedings of the 20th international conference on Social information theory_, volume 10 of _Lecture Notes in Computer Science_, pages 1-10, Springer, Heidelberg, 2000.
* [18] K. E., "Social networks: The role of social activity in social networks", in _Proceedings of the 20th international conference on Social information theory_, volume 10 of _Lecture Notes in Computer Science_, pages 1-10, Springer, Heidelberg, 2000.
* [19] K. E., "Social networks: The role of social activity in social networks", in _Proceedings of the 20th international conference on Social information theory_, volume 10 of _Lecture Notes in Computer Science_, pages 1-10, Springer, Heidelberg, 2000.
* [20] K. E., "Social networks: The role of social activity in social networks", in _Proceedings of the 20th international conference on Social information theory_, volume 10 of _Lecture Notes in Computer Science_, pages 1-10, Springer, Heidelberg, 2000.
* [21] K. E., "Social networks: The role of social activity in social networks", in _Proceedings of the 20th international conference on Social information theory_, volume 10 of _Lecture Notes in Computer Science_, pages 1-10, Springer, Heidelberg, 2000.
* [22] K. E., "Social networks: The role of social activity in social networks", in _Proceedings of the 20th international conference on Social information theory_, volume 10 of _Lecture Notes in Computer Science_, pages 1-10, Springer, Heidelberg, 2000.
* [23] K. E., "Social networks: The role of social activity in social networks", in _Proceedings of the 20th international conference on Social information theory_, volume 10 of _Lecture Notes in Computer Science_, pages 1-10, Springer, Heidelberg, 2000.
* [24] K. E., "Social networks: The role of social activity in social networks", in _Proceedings of the 20th international conference on Social information theory_, volume 10 of _Lecture Notes in Computer Science_, pages 1-10, Springer, Heidelberg, 2000.
* [25] K. E., "Social networks: The role of social activity in social networks", in _Proceedings of the 20th international conference on Social information theory_, volume 10 of _Lecture Notes in Computer Science_, pages 1-10, Springer, Heidelberg, 2000.
* [26] K. E., "Social networks: The role of social activity in social networks", in _Proceedings of the 20th international conference on Social information theory_, volume 10 of _Lecture Notes in Computer Science_, pages 1-10, Springer, Heidelberg, 2000.
* [27] K. E., "Social networks: The role of social activity in social networks", in _Proceedings of the 20th international conference on Social information theory_, volume 10 of _Lecture Notes in Computer Science_, pages 1-10, Springer, Heidelberg, 2000.
* [28] K. E., "Social networks: The role of social activity in social networks", in _Proceedings of the 20th international conference on Social information theory_, volume 10 of _Lecture Notes in Computer Science_, pages 1-10, Springer, Heidelberg, 2000.
* [29] K. E., "Social networks: The role of social activity in social networks", in _Proceedings of the 20th international conference on Social information theory_, volume 10 of _Lecture Notes in Computer Science_, pages 1-10, Springer, Heidelberg, 2000.
* [30] K. E., "Social networks: The role of social activity in social networks", in _Proceedings of the 20th international conference on Social information theory_, volume 10 of _Lecture Notes in Computer Science_, pages 1-10, Springer, Heidelberg, 2000.
* [31] K. E., "Social networks: The role of social activity in social networks", in _Proceedings of the 20th international conference on Social information theory_, volume 10 of _Lecture Notes in Computer Science_, pages 1-10, Springer, Heidelberg, 2000.
* [32] K. E., "Social networks: The role of social activity in social networks", in _Proceedings of the 20th international conference on Social information theory_, volume 10 of _Lecture Notes in Computer Science_, pages 1-10, Springer, Heidelberg, 2000.
* [33] K. E., "Social networks: The role of social activity in social networks", in _Proceedings of the 20th international conference on Social information theory_, volume 10 of _Lecture Notes in Computer Science_, pages 1-10, Springer, Heidelberg, 2000.
* [34] K. E., "Social networks: The role of social activity in social networks", in _Proceedings of the 20th international conference on Social information theory_, volume 10 of _Lecture Notes in Computer Science_, pages 1-10, Springer, Heidelberg, 2000.
* [35] K. E., "Social networks: The role of social activity in social networks", in _Proceedings of the 20th international conference on Social information theory_, volume 10 of _Lecture Notes in Computer Science_, pages 1-10, Springer, Heidelberg, 2000.
* [36] K. E., "Social networks: The role of social activity in social networks", in _Proceedings of the 20th international conference on Social information theory_, volume 10 of _Lecture Notes in Computer Science_, pages 1-10, Springer, Heidelberg, 2000.
* [37] K. E., "Social networks: The role of social activity in social networks", in _Proceedings of the 20th international conference on Social information theory_, volume 10 of _Lecture Notes in Computer Science_, pages 1-10, Springer, Heidelberg, 2000.
* [38] K. E., "Social networks: The role of social activity in social networks", in _Proceedings of the 20th international conference on Social information theory_, volume 10 of _Lecture Notes in Computer Science_, pages 1-10, Springer, Heidelberg, 2000.
* [39] K. E., "Social networks: The role of social activity in social networks", in _Proceedings of the 20th international conference on Social information theory_, volume 10 of _Lecture Notes in Computer Science_, pages 1-10, Springer, Heidelberg, 2000.
* [40] K. E., "Social networks: The role of social activity in social networks", in _Proceedings of the 20th international conference on Social information theory_, volume 10 of _Lecture Notes in Computer Science_, pages 1-10, Springer, Heidelberg, 2000.
* [41] K. E., "Social networks: The role of social activity in social networks", in _Proceedings of the 20th international conference on Social information theory_, volume 10 of _Lecture Notes in Computer Science_, pages 1-10, Springer, Heidelberg, 2000.
* [42] K. E., "Social networks: The role of social activity in social networks", in _Proceedings of the 20th international conference on Social information theory_, volume 10 of _Lecture Notes in Computer Science_, pages 1-10, Springer, Heidelberg, 2000.
* [43] K. E., "Social networks: The role of social activity in social networks", in _Proceedings of the 20th international conference on Social information theory_, volume 10 of _Lecture Notes in Computer Science_, pages 1-10, Springer, Heidelberg, 2000.
* [44] K. E., "Social networks: The role of social activity in social networks", in _Proceedings of the 20th international conference on Social information theory_, volume 10 of _Lecture Notes in Computer Science_, pages 1-10, Springer, Heidelberg, 2000.
* [45] K. E., "Social networks: The role of social activity in social networks", in _Proceedings of the 20th international conference on Social information theory_, volume 10 of _Lecture Notes in Computer Science_, pages 1-10, Springer, Heidelberg, 2000.
* [46] K. E., "Social networks: The role of social activity in social networks", in _Proceedings of the 20th international conference on Social information theory_, volume 10 of _Lecture Notes in Computer Science_, pages 1-10, Springer, Heidelberg, 2000.
* [47] K. E., "Social networks: The role of social activity in social networks", in _Proceedings of the 20th international conference on Social information theory_, volume 10 of _Lecture Notes in Computer Science_, pages 1-10, Springer, Heidelberg, 2000.
* [48] K. E., "Social networks: The role of social activity in social networks", in _Proceedings of the 20th international conference on Social information theory_, volume 10 of _Lecture Notes in Computer Science_, pages 1-10, Springer, Heidelberg, 2000.
* [49] K. E., "Social networks: The role of social activity in social networks", in _Proceedings of the 20th international conference on Social information theory_, volume 10 of _Lecture Notes in Computer Science_, pages 1-10, Springer, Heidelberg, 2000.

 

In 2009, how many pedestrian men died? In 2009, 157 Pedestrian Deaths, [http://www.nj.gov/info/fatalace/2009_fatal_crash.pdf](http://www.nj.gov/info/fatalace/2009_fatal_crash.pdf), MALE ; 112, FEMALE ; 45, MALEP: 45,

\begin{tabular}{p{36.8pt} p{36.8pt} p{36.8pt}} \hline What percentage of clothing and consumer electronic products of men photographed by mobile shoppers, taken together? & 22 \%, 22 \%, 32 \%, 18 \%, 4 \%, 5 \%, 13 \%, 2 \%, 20 \%, 30 \%, PRODUCTS PHOTOGARHED BY MOBILE SHOPERS, 15 \%, At work, 25 \%, In the \\ \hline What is the value of New York Knicks? & NEW YORK KNICKS $ 3.30B & NEW YORK KNICKS $ 3.30B, NEW YORK METS $ 2.00B, NEW YORK \\ \hline How much more is the value of Barcelona FC when compared to Real Madrid (Sbn)? & BARCLONA FC § 3.64B, NEW YORK KNICKS $ 3.30B, LOS ANGELONA FC § 3.00B, CHICAGO BULLLS § 2.60B, GOLDEN STATE WARRIORS $ 2.60B, CHICAGO BULLLS § 2.50B, BRO & REAL MADRID, $ 3.58B, BARRECLONA FC § 3.64B, A mountain of sponsorship and advertising cash keeps Man U king of the soccer castle, though Barcelona \\ \hline Which is the second last tip for staying healthy? & Don't touch your, face, Avoid close contact with someone who’s, sick, Clean and disinfect surfaces and objects people frequently touch & Tips for staying healthy, ON, What to do if you feel sick, Stay home, Most people with COVID - 19 have mild to moderate symptoms and can recover at home. Rest up and prevent germs from spreading by staying home \\ \hline What percent of adults in age group 65+, buy their food based on the ’availability of nutritious food’? & 33 \%, 28 \%, 21 \%, 32 \%, 11 \%, 17 \%, 15 \%, Making it easier for the 50+ to get more nutritious foods, 1 \%, 65 \%, Help find information on fruits & Food Availability, AARP@cunk>, FOUNDATION, A recent AARP Foundation survey of 1,000 low - income adults age 50+ reveals that, in the past 12 months, two in \\ \hline Who provides statements for the presentencing investigation report? & ANALYSIS OF LEGAL HISTORY, ANALYSIS OF LEGAL HISTORY, ADISARD of the DV Criminal History including Domestic Incident port ( DIR ) history, How many arteries in DV related crimes? Convictions?, • Stalking history, • Prolective orders?, • Level of compliance if under supervision before?, • Current release status, • Jail days credited, Domestic Incident Report ( DIR ) history, • \\ \hline What happened first; Gaza conflict or Scottish independence? & GAZA CONFLIICT August 1 : 64K Peak Shares SCOTTISH INDEPENDENCE September 14 35K DENCE March 17 \\ \hline Who uses the twitter id @Ev? & 27 \% fake or empty 28 \% inactive 43 \% good & Twitter co - founder Evan Williams @Ev, WHOLESAELS, IN DARK CORNERS OF THE INTERNET , THEY DIY TOOLS TO OVERRIDE TWITTER’S RULES, THE \\ \hline \end{tabular} Table 12: We show 20 random selected examples with student generated or TF-IDF extracted rationales. The first 5 examples are the same as in Table 10, where 60% of student generated rationales match the tool generated ones. For more than 50% of the student generated rationales, answers can be inferred from them without looking at the images. 90% of the student generated rationales are relevant to the answer, others are irrelevant.

 

OCR string from image: H, EVOLUTION OF THE SKATEBOARD, 1940, 1959 1960 1964 1970, 1975, 1980, 1990, 2000, SIDEWAK SURFBORADS The first skateboards started with wooden boxes, or boards., which kids added roller skate wheels to in the late 40's and early 50's, ROLLEER DERBY SKATEBORARD The Roller Derby Skate Company was the company who coined the name stateboard. They were the first company to mass produce the Roller Derby Statedor. Their factory was in La Mradia, CA. By 1959, people could purchase. the boards nationwide at Roller Derby arenas. NASH SHARK In the 1960's, another company by the name of NASH came out with their own skateboards, and they called it the Shark. Today its known as the hash Shark Stateboard, GANDS FIBREFLEX PINTAL IN 1964, the G & S FiberFlex Pintali was born. It was made by writers for sorters. G & S stand for Larry Gordian and Flojof Smith. In the 60's, these gives became one of the largest and most successful stakeholder board companies. , BANANA BOARD In the mid 1970's, a new board hit the streets. It was called the Banan aboard. The Banana boards are skinny, flexible boards made out of polypropylene that have ribs on the underside for structural support. ROAD RIBC ERCUSIER in 1975 Road folder came out with the first ever sidebaoard that had precision bearings made just for stakeholders. This would bring an end to decades of loose ball bearings. OLD SCHOWGL KSHTALL is the 1980's, skateboards changed for vert skaters. The ideal board to rider wet was the Fishtal deck. People still kacted street with these short nosed, wide vert, soft wheeled boards. POP SIGKLE, POP SIGKLE, IL the 1997's.stackexchange during started focusing more on street skatebandring. Most boards are 7 I/4 to 8 in and 30-32 inches long with a largely symmetrical shape with a relatively narrow width. , The board hasr shared much from the 90's till now, but the concave may be a little deeper . However , people are starting to ride their own custom shaped boards more and more ! Question: when was nash shark introduced? Answer: 1960 Evidence: NASH SHARK In the 1960's, another company by the name of NASH came out with their own skateboards, and they called it he Shark. Today it's known as the Nash Starkeboard.

...(omit two examples) OCR string from image: State, Government, Chad Foust FIF, [ great]. Reasons to hire me as Art Director , PRESENTATION, [ reason : five] +, TENT, years experience, 2001, 2002, 2003, 2004 2005, 01 D\({}_{0}\)2 03 d 045, creating beautiful presentation design, for Community Groups, Direct Marketing Sales [282], 2006, 2007, 2008, 2009, 2010, 2011, 07 06, 08, 09, 10, now, Real Estate Ventures, Non - Profitt Sector, Youth Camps O, [ reason : four] , Motion Graphics, < FRVE serves DIRECTING creative teams, Lower Thirds, Lorenipsiny door sitmart, consistent or displacing eilt, sed de uismodem temporoiler, le 

OCR string: B & W, Broown & Williamson TOBACCO CORPATION RESEARCH & DEVELOPMENT, TQ.; R. H. Honeycutt, CC.; T.F. Riethl, FROMM.;C. J. Cook, DATE ; May 8, 1995, SUBJECT ; Review of Existing Brainstorming Ideas / 483, INTERNAL CORRESPONDENCE, The major function of the Product Innovation Group is to develop marketable novel products that would be profitable to manufacture and sell. Novel is defined as: of a new kind, or different from anything seen or known before. Innovation is defined as: something new or different introduced; act of innovating ; introduction of few things or methods. The products may incorporate the latest technologies, materials and know - how available to give them a unique taste or look . The first task of the Product Innovation Group was to assemble, review and categorize a list of existing brainstorm ideas . Ideas were grouped into two major categories labeled appearance and taste / aroma. These categories are used for novel products that may differ from a visual and / or taste - aroma point of view compared to conventional cigarettes. Other categories include a combination of the above, filters, packaging and brand extensions . Appearance, This category is used for novel cigarette constructions that yield visually different products with minimal changes in smoke chemistry, \(\bullet\) Two cigarettes in one . Multi - plug to build your own cigarette . Switchable menthol or non menthol cigarette.

Question: Who is in cc in this letter?

Answer: TF. Riethl
Evidence: TO; R. H. Honeycutt, CC.; T.F. Riethl, FROMM; C. J. Cook.

OCR string: Confidential RJRT PR APPROVAL, DATE ; SUBJECT ; 1/8/93 - Lu glass PROPOSED RELEASE DATE ; FOR RELEASE TO : CONTACT : P. CARTER, for response, ROUTE TO I. Home, Peggy Carter, Maura Payne, David Fishel Tom Griscom Diane Barrows, Ed Blackmer, Tow Rucker, Initial, Ace, out, OB7, tus , TVR, Return to Peggy Carter ,PR ,16 Reynolds Building, Date, 1/8/93, Source : [https://www.windustroductuments.ucsfedu/docs/xnbD0037](https://www.windustroductuments.ucsfedu/docs/xnbD0037), 51142 3977

Question: what is the date mentioned in this letter?

Answer: 1/8/93

Evidence: DATE ; SUBJECT ; 1/8/93 - Lu gas, Date, 1/8/93

OCR string: DOMESTIC PRODUCT DEVELOPMENT ( cont'd .), Project Marlboro, - POL 0330 - L6 tar / puff - 80mm has been produced and currently is in C.I. for analytical . - POL 0331 - 1.6 tar / puff - 84mm was produced 6/1/90. Samples have been submitted to C.L. - Marlboro Double Batch - RL & FCB was produced 6/4/90. Samples have been submitted for analytical testing . - POL 3634 - RL Evaporator Upgrade - Scheduling for primary at the Mt / Chs been completed . Fabrication is scheduled for the week of 6/18/90 on Semiworks , Marlboro Menthol, Marlboro Menthol 80mm and SBMm were subjectedly smoked by the Richmond Panel. After further review of the data and specifications, another model of the 83mm with zero ventilation will be made at Semikowts within the next 2-3 weeks . Bucks, Burcks K.S. Lights and Full Flavor with various aftercut modifications were smoked by the Richmond Panel. Particular models were selected from the group and POL testing will be done on these prototypes ,, Miscellaneous, Additional tipping papers of Marlboro Lights have been received and currently are being analyzed for lip release coatings. Cigarettes will be produced and submitted to O / C Panel for evaluation of lip release . 3; Source : [https://www.industrydocuments.ucsfedu/docs/khvj0037](https://www.industrydocuments.ucsfedu/docs/khvj0037), 2022155853 Question: what mm Marlboro Menthol were subjectively smoked by the Richmond Panel Answer 80mm and SBMm

Evidence: Marlboro Menthol, Marlboro Menthol 80mm and SBMm were subjectively smoked by the Richmond Panel.

OCR string: SFE - GC were also demonstrated in quantitative measurements of phenolics in woodsmoke analysis .W.T. Foreman ( US. Geological Survey , CO ) extracted the C. cartridge with SFE to recover pesticides in high yield , DETERMINATION OF POLAR VOLATILE ORGANICS (PVOC ) IN AMBIENT AIR, The polar compounds are those containing hetero - atoms such as nitrogen , sulfur and oxygen . The single most difficult problem in developing protocols for analyzing polar compounds at trace level in air is probably moisture. Sampling of sidestream smoke components shared similar difficulty . The moisture in the ambient air closed up the cryogenic trap and prevented sample enrichment . The evaporation of water vapor in the source of the mass spectrometer interfered with the high vacuum and the detection of co - eluting compounds. The present EPA TO - 14 method requires the use of Naphion dryer to eliminate water . Unfortunately, the Naphon tube is also permeable to many molar compounds such carbonyls and alcohols. Method TO - 14 with carrier sampling is only for nonpolar organic compounds, e.g. aromatics and hydrocarbons . Source :[https://www.inolustrydocuments.ucsfedu/docs/ylo037](https://www.inolustrydocuments.ucsfedu/docs/ylo037), 2022155945 Question: Which hetero-atoms does polar compounds contain?

Answer: nitrogen, sulfur andoxygen.

Evidence: The polar compounds are those containing hetero - atoms such as nitrogen , sulfur and oxygen .

OCR string: CUT TOBACCO ; BLEND ; MT - 768 DBT391 BW - 6071, BEST PROYOTYPE , 327391, LBS AT 12.5%, SOLID LBS, LBS AT TARGET, STRIPS : FLUEUE UURE 

Please generate the program as the intermediate step to answer the question based on the OCRs and tables. The tables show the layout of the plot, but the numbers may be inaccurate or incomplete. Please check if these numbers appear in the OCR; if not, please ignore them in the tables.

The only available functions of the programs are

DiVa(b): Mul[a,b]; Aug(last of numbers): Sum(list of numbers); Diff(a,b); Greater(ab,k); Less(ab,h): Find(str).

OCR: Public Expects Political Division to Persist Level nation's political division in five years will be ... Don't know, More, Same, 36 %, 41 %, Less, 58 %, Survey conducted Dec 2-7, 2014. PEW RESEARCH CENTER Table: EntityValue [105,517] Same,411 Morre.36 [Less,17] Question: What is the difference in value between Same and sum of More and Less? Answer: 12

Program: Dirf(41,Sum(36, 17)) Execution: Dirf(41,(36+17))-Diff(41-53)=[41-53]=12

OCR: T - Series, YouTube Movies, Music, Cocomleon - Nursey Rhymes, PewDiePie, SET India, Gaming, 89.2, Kids Diana Show, 79.1, WWE, Sports, Additional Information, 77.6, 75.2, 25.5, 50.75, 115, 112, 110, 105, 100, 137, 125, 183, 150, 175, 200, 225, Number of subscribers in millions, *, 155, 59, e Statista 2021, Snow source Table: CharacteristicNumber of subscribers in millions ! TeSeries,183.0 | YouTube Movies,137.0 | Music,115.0 | Cocomleon - Nursey Rhymes,112.0 | WebDiePie,110.0 | User Indica,105.0 | Ganjen89.2 | Kids Diana Show,79.1 | WWE,77.6 | Sports,75.0 Question:What is the average number of subscribers of the most 3 popular Youtube channels? Answer: 145

Program: Agg(183.0,137.0/115.0) Execution: {183.0}+137.0/115.0)-3145.0

OCR: Overwhelting Majority of Russians Say Breakup of USSR Was Bad for Russia Day out think the dissolution of the Soviet Union was a good thing or bad thing for Russia ? Good, thing, 17 %, Don t, Bad, known, 14 %, thing, 69 %, Source : Spring 2015 Global Attitudes survey _ Q34 , PEW RESEARCH CENTER Table: EntityValue | Bad thing_69 | Goodthing,17 | Don't know,14 Question: What is the percentage of Don't know in the chart? Answer: 14

Program:FindPercentage of Don't know)

OCR: In Canada, only a quarter of the public has confidence, in Trump Among Canadians ... 100 %, Favorable view of the U.S., 72 & 63, 59, 59, 40, 88, 83, 81, 76, 68, 66, 65, 64, 43, 55, 39, 28 Confidence in US, president, 25, 22, 0 2002, 2006, 2010, 2014, 2018, Bush, Obama, Trump, Source: Spring 2018 Global Attributes Survey _Q17A_ & Q35a.. PEW RESEARCH CENTER Table: YearConfonce in US, president,view of the U.S. favorable [2002,59.72 | 2006,40.59] [2010,88.68 | 2014,81.64 | 2018,25.39 Question: Is the average of highest and lowest value of green bar greater than 80? Answer: No Program: Greater(Any(72.39I,80) Execution: Greater((72+39I/2.80) Generator(55.5,80)\(>\)55.5-80?No

OCR: Pakistani Say It's Important to Educate Both Girls and Boys Education is more important for ... Boys and girls equally 86 %, 7 % 5 %, 2 %, Dont, Boys, Girls, know, Source: Spring 2014 Global Attitudes, survey _PEW RESEARCH CENTER Table: EntityValue | Boys and girls equally,86 | Gits,5 | Boys Girls,75 | Don't know.2 Question: New sum of three smallest segment, multiply it 5, is the result greater than largest segment? Answer: No Program: Greater(Mul(Sum(5, 5, 2), 5), 86) Execution: Greater(Mul(5+5+2,5),86)-Greater(Mu([14,5],86)-Greater(14^5,86)-Greater(70,86)-70-86?No

OCR: Americans Give China Mostly Negative Ratings, U.S. views of China, 80 %, 43, 52, 35, 0, 2005, Unfavorable, 55, 54, 51, 52, 50, 49, 42, 42, 40, 40, 39 39, 38, 36, 36, 37, 38, 35, 29, 2007, 2009, Source: Spring 2015 Global Attitudes survey _Q12L_ , PEW RESEARCH CENTER, Favorable, 2011, 2013, 2015

Table: YearFavorable,Unfavorable | 2005.03, 2007,5250.39 | 2009.50.38 | 2011.49,36 | 2013.35,52 | 2015.38,54 Question: How many values are below 40 in unfavorable graph? Answer: 6

Program: FindCount of values below 40 in Unfavorable graph)

OCR: How often people interact with people of other races, ethnicities varies widely, % who say they race or ethnicity, interact with people of a different, Never / Rarely, Occasionally /. Frequency, India, 27 %, 66 %, South Africa, 34, 66, Venezuela, 40, 60, Lebanon, 40, 57, Colombia, 46, 53, Jordan, 48, 51, Kenya, 48, 51, Tunisia, 59, 40, Philippines, 61, 38, Vietnam, 64, 33, Mexico, 69, 30, Note: Don't know responses not shown ... Source " Mobile Technology and Its Social Impact Survey 2018 ... Q38b ... Attudes Toward Diversity in 11 Emerg Economics " PEW RESARCH CENTER Table: EntityNewer/RarelyOccasionally) Forequently | Mexico,69.30.0 | Phillippines,61.38.0 | Kenya,48,Ann | Jordan,48,51.0 | Colombia,46,53.0 | Lebanon,40,Banul |Yenezuela,40,60.0 | South Africa,34,66.0 | India,27,66.0 | Consortium: I see the median of the green bar smaller than the median of the blue bar? Answer: No Program: Less(51,48) Execution: 51+48?No

OCR: : No 65.88 %, - Yes 34.12 %, + 99, old, Additional Information, e Statista 2021, Show source Table: CharacteristicsShare of respondents | Yes,34.12% | No,65.88% Question: What is the ratio of yes to no? Answer: OST8 Program: Div(34.12x,65.88%) Execution: 34.12x/65.88%-0.518

OCR: [oorr] Table: [table] Question: [query] Answert: [answer] Program:

Figure 8: ChartQA prompt template.