DocVXQA: Context-Aware Visual Explanations for Document Question Answering
Authors: Mohamed Ali Souibgui
Changkyu Choi
Andrey Barsky
Kangsoo Jung
Ernest Valveny
Dimosthenis Karatzas
Venue: ICML 2025 poster (2025)
Keywords: Document Intelligence
Explainability
Self-explainable Deep Learning
Vision and Language
Primary area: applications->computer_vision

Abstract: We propose **DocVXQA**, a novel framework for visually self-explainable document question answering, where the goal is not only to produce accurate answers to questions but also to learn visual heatmaps that highlight critical regions, offering interpretable justifications for the model decision. To integrate explanations into the learning process, we quantitatively formulate explainability principles as explicit learning criteria.
Unlike conventional relevance map methods that solely emphasize regions relevant to the answer, our context-aware DocVXQA delivers explanations that are contextually sufficient yet representation-efficient. This fosters user trust while achieving a balance between predictive performance and interpretability in document visual question answering applications. Extensive experiments, including human evaluation, provide strong evidence supporting the effectiveness of our method.

Doc VXQ: Context-Award Visual Explanations for Document Question
Answering
Mohamed Ali Souibgui 1
Changkyu Choi *2 Andrey Barsky' Kangsoo Jung? Ernest Valveny
Dimosthenis Karatzas'
Abstract
We propose DocVXQA, a novel framework for visually self-explainable document question an- swering. The framework is designed not only to produce accurate answers questions but also to learn visual heatmaps that highlight contextu- ally critical regions, thereby offering interpretable justifications for the model's decisions. To inte- grate explanations into the learning process, we quantitatively formulate explainability principles as explicit learning objectives. Unlike conven- tional methods that emphasize only the regions pertinent to the answer, framework delivers ex- planations that are contexiually sufficient while re- maining reprexentation-efiiie trust while achieving a balance between predic- tive performance and interpretability VoA applications. Extensive experiments, including human evaluation, provide strong evidence sup- porting the effectiveness of our method. The code is available at https:/ httgghgthhhh..oo com/ da i 92002 2022000///ccx2..
The up why
specialty blends, spart
digareto tastes Camel, because other aigaratle blerded best Turk ah and domestic tobaccoe for a taste that has Camel. distnctive flvor and wo'ld id-oolees emoothness.
Based on secrets we ve learned over 85 years, Camal has developed collection of exolis formules, splove and fobeozo papers, Now, Camell introduces a sperial set digenelles that complements our core blend with new flavors and rare tobaccoe.
Gamel Pleasure burn since 1813.
The insert elso sums up why Camel, and its specialty blands, stand apart:
No other digaretto tastes lika Camel because no other oigaretle blended Camel We match best Turk ah domestic tobacoce for taste disfnctive faver and wo id-olass smeothness.
Based secrats
developed 85 years, papers, Now. Camel introduces spices special digarettes that complements corn new flavors and rare
collection
This fosters user
Gamel Pleasure burn since 1913.
Figure f. An illustration of the relevant regions DocVQA model (highlighted zones), produced by extracting the raw at- tention maps from the last layer (top) and by using our method (bottom) the question " Pllessrre burn since 1913', Which cigarette's 's tagline is this?". Here, the answer given correctly by the model is "Camel".
1. Introduction
tual and visual modalities at scale (Wang et al., 2024; Zhao et al., 2024; Rasheed et al., 2024; Parashar et al., 2024).
Document visual question answering (QocVQA) is essen- tial for automatically extracting information from visually complex, multi-modal documents including invoices, re- ports, and contracts (Appalaraju et al., 2024; Blau et al., 2024; Tito et al., 2023; Mathew et al., 2021). This task requires interpreting the question and generating through natural language, given specific document, Re- cently, DocVQA models have increasingly leveraged large vision-language models for their ability to process both tex-
Despite these promising advances in terms of utility, Doc relies on large, opaque neural network archi- tectures that function as "black boxes," offering limited transparency into how they generate answers. This lack of transparency is particularly concerning in high-stakes domains such as finance (Wu et al., 2023), healthcare (Li et al., 2024), and law (Abdallah et al., 2023), where trust in AI-driven caaisinn-mmking crucial. In addition, ex- isting explainable methods for VQA primarily rely on visualizing attention maps to highlight relevant regions in- fluencing the model's predictions. However, as illustrated in Figure (above), these attention-based explanations are often noisy or imprecise, failing to justify the reasoning be- hind the answer. Instead of offering clear insights, they tend to broadly highlight all occurrences of the answer without
'Equal contribution 'Computer Vision Center, Universitat Autônoma de Barcelona. Spain *UiT The Arctie University of Norway, Norway 'Inria, France. Correspondence to: Mohamed Ali Souibgui <msouibgui evc.uab.cat>.
Proceedings of the ndnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

DocVXQA: Context-Aware Visual Explanations for Document Question Answering
contextualizing why it was selected.
planations along with answer predictions. (2) We formalize explainability as an explicit learning objective with the in- formation bottleneck principle. (3) We make our design model-agnostic and lightweight, thus, compatible with ex- isting pretrained Doc models and requiring minimal ar- chitectural changes. (4) We conduct extensive experiments, including human evaluations, to validate the effectiveness of the proposed approach.
To address this limitation and enhance the transparency of Doc models, we propose novel self-explainable framework called Doc, which learns to provide context-aware explanations to the answers in the form of relevance maps. Note that selffexplainable models generate interpretable outputs by design, in contrast to post-hoc ap- proaches that provide explanations retrospectively (Moller et al., 2024; Choi et al., 2024; Gautam et al., 2023; 2022). While these methods have been explored from a methodolog- ical perspective, their application in the context of Doc remains largely unexplored. To the best of our knowledge, our framework is the first self- eelf-xxplainable DocVQA model designed to generate visual explanations that align with tex- tual answers. This approach enhances both transparency and user trust in model predictions.
2. Related Work
2.1. Document visual question answering
Typical Voc models are reaching good performances using visual and textual features within waanformer-bbsee encoder-decoder architectures to generate answers (Ap- palaraju et al., 2024; Tito et al., 2023; Powalski et al., 2021; Xu et al.. 2020). In this setup, text extrtacted with OCR systems. However, despite their strong best-case perfor- mance, these methods are prone to errors introduced by OCR inaccuracies and face difficulty in domains that are more reliant on visual features.
To enhance transparency without disrupting established Doc models, we propose a strategy that learns gen- erating explanations while maintaining compatibility with available pretrained models. We build upon the widely used Pix2Struct (Lee et al., 2023) model, trained for DocVQA task. With minimal alterations to the existing architecture- thus avoiding significant impacts performance the need for extensive retraining-ovl our method first learns a mask over the document image as an explanation, and then uses the masked image as an input to the same network to answer the question. This sequential learning process ensures that the generated explanations directly contribute to the answer.
In contrast, OCR-free end-to-end models (Lee et al., 2023; Aggarwal et al., 2023; Kim et al., 2022; Davis et al., 2022; Kim et al., 2021) predict answers directly from document im- ages, utilizing pre- mre-trainngg objectives to interpret text with- out relying on OCR. More recently, Large Vision-Language Models (VLMs) like GPT-4 (Achiam et al., 2023) set new benchmarks in Doc performance, but their high com- putational requirements and closed-source nature limit their accessibility. There has also been gradual emergence of smaller, open source models trained with instruction tuning (Wang et al., 2024; Zhang et al., 2023; Ye et al., 2023), but all these models fall short in providing explanations that support the reason behind their predictions. Among the OCR-free models, Pix2Struct (Lee et al., 2023) is designed for sisually-situated language understanding tasks. It uses a pretraining strategy called screenshot parsing, converting masked web page screenshots into simplified HTML rep- resentations. The model incorporates variaile-rssolution input, preserving the aspect ratios for diverse document lay- outs. Thus, avoiding the significant reduction in original image resolution, which make it successful in performing DoA without requiring external OCR systems.
The core contribution of our VXEA lies in learning effective mask designed to enhance human understanding. Grounded in philosophical foundation of good explana- tion (Choi et al.. 2024; Sokol & Flach, 2020), our approach aims to develop a method that learns mask representations that are both contextually sufficient and representation- efficient. To achieve this, we frame the trade-off between these objectives within the information bottleneck princi- ple (Tishby& & Zaslavsky, 2015), ensuring a balance between retaining relevant contextual information and minimizing redundancy in the representation. Additionally, to prevent overfitting in mask learning and to enhance the generalizabil- ity of the explanations, we integrate additional pretrained models (Faysse et al., 2024; Yu et al., 2024a), which infer the mask representation as a prior and interact with the main model to refine the explanation learning process, ensuring robust and reliable outcomes. This approach offers more precise and contextually relevant justifications. Notably, although our focus is on Pix2Struct (Lee et al., 2023), our method is inherently aggostig,, allowing seamless integration with various VQA architectures.
2.2. Explainability in vision and language
Current assion-and-language model explanations rely heav- ily on self-attention mechanisms to generate relevance maps, highlighting the regions that contribute to the model's de- cision (Bousselham et al., 2024; Chefer et al., 2021b;a). However, attention maps are often noisy (Abnar & Zuidema, 2020), making it challenging to extract meaningful insights into the model's reasoning. Recently, in the domain of
Our main contributions are as follows: (1) We introduce Doc the first self-exxlaine eeff-explainable Doc framework that learns to generate visually grounded context-aware ex-
2

---

DocVXQA: Context-Aware Visual Explanations for Document Question Answering
Doc new approaches have been introduced to pro- vide more fine-grained grounding by explicitly localizing answer regions (Mohammadshirazi et al., 2024; Zhou et al., 2024). These methods, however, require annotated answer locations during training, leading to the development of new datasets containing such annotations (Giovannini et al., 2025). While effective in the case of extractive VOA tasks (where the answer string found explicitly in the doc- ument), these approaches introduce a significant annotation cost and remain limited in their ability to capture the con- text that explain why an answer is correct. Consequently, there remains a need for explanation techniques that not only identify specific answer locations but also provide a reasoning context beyond simple spatial answer grounding.
function (Bang al., 2021; Rudin, 2019; Alvarez Melis & Jaakkola, 2018), and architectural adjustments of the net- work to enhance transparency (Gautam et al., 2023; 2022; Alvarez Melis & Jaakkola, 2018). These approaches mark a shift from the traditionally opaque "black-box' neural net- works to a more transparent and understandable framework.
2.4. Information theoretic learning
Information theory provides a robust foundation for quanti- fying data representation through metrics such as entropy, divergence, and mutual information (Principe, 2010; Shan- non, 1948). Building on this foundation, recent learning systems have embraced these principles (Skean et al., 2025; Yu et al., 2024b; 2019; Tishby & Zaslavsky, 2015) and evolved through various innovative approaches (Jenssen, 2024; Choi et al., 2024).
2.3. Learning to explain
Model explainability has gained significant attention in recent years, aiming to provide insights into neural net- work drecision-making (Covert et al., 2021). A common approach is generate saliency maps that highlight input regions relevant to predictions (Schulz et al., 2020; Selvaraju et al., 2017). In this context, the saliency map is a set of relevance scores over input pixels, intended to reveal the model's reasoning. Earlier explainability methods provide post-hoc explanations, i.e., get explanation without chang- ing the trained model weights or architecture (Shrikumar et al., 2017: Selvaraju et al., 2017; Ribeiro et al., 2016; Bach et al., 2015). Gradient- based methods such as Grad- CAM (Selvaraju et al., 2017), LRP (Bach et al., 2015), and DeepLIFT (Shrikumar& et al., 2017) calculate saliency maps through backpropagation of model output gradients to the input or feature space. While these methods are easy to implement, they often suffer from gradient shattering, lead- ing to noisy and imprecise explanations. Another line of post-hoc explanation research is perturbation-basee meth- ods. This approach observes output changes by processing a set of perturbed images, each occluding different regions of the original image. Examples include LIME (Ribeiro et al., 2016), Occlusion (Zeiler& & Fergus, 2014), and RISE (Petsiuk, 2018). While these methods often produce more reliable attributions, their effécctivness is constrained by the high computational cost of sampling sufficient number of meaningful perturbations.
3. Method
3.1. Visually self-explainable Doc Enhancing context awareness through a learnable mask
We propose DocVXQA, novel framework for visually self-explainable document question answering. At a high level, this framework provides not only the answer to a ques- tion but also a visual explanation in the form of a relevance map (Müller, 2024; Zhang et al., 2024). The core innovation of our method lies in learning the relevance map-based ex- planations while simultaneously improving their reliability, To achieve this, we quantitatively formalize philosophically defined explainability principles (Sokol & Flach, 2020) and integrate them into unified objective function, enabling end-to-end training. Our framework builds upon the widely recognized Pix2Struct architecture (Lee et al., 2023), adapt- ing it effectively for this domain.
trade-off In formulating explain- ability principles, VXQA draws inspiration from the in- formation bottleneck (IB) framework (Tishby & Zaslavsky, 2015), which seeks to construet compact representation T' of input data X that preserves only the information essential for predicting a target variable Y. This frame- work balances the trade-off between reducing the mutual information I(X; T) to compress irrelevant data and max- imizing [(T; Y) to retain predictive power. To implement celf-explainabiliiy within the conventional DocVQA setting, we strategically detine the bottleneck representation T as the masked input, T X O M, where @ represents the Hadamard product that masks out irrelevant information within X, and M is a learnable mask. This formulation pro- vides a self-explaining mask in the form of region discovery, highlighting the most relevant regions of the input (Choi et al., 2024; Zhmoginov et al., 2021). With these terms, the
Minimality
Unlike post-hoc approaches, self-explainabbe methods (Choi et al., 2024; Møller et al., 2024; Rudin, 2019; Alvarez Melis & Jaakkola, 2018) embed explainability directly into the training process, ensuring that models learn to generate in- terpretable explanations alongside predictions. Instead of relying on external attribution techniques, these models are designed to provide built-in reasoning. Among the vari- ous approaches to self- celf-explainable deep learning methods, two main strategies stand out: the integration of mathemati- cally formulated explainability principles into the objective
3

---

DocVXQA: Context-Aware Visual Explanations for Document Question Answering
Q: How much is the eecess of expenditures over income'?
Q: How much is the 'excess of expenditures over income'?
ColPali
3
ENC
6
MASK
DEC
TEXT
A: 3,038,444
Pix2Struct
Figure 2. Overview of the proposed DocVXQA framework: ① The input question and the full document image are sent to both the pretrained Pix2Struct ENC-DEC and pretrained ColPali model. X The mask head (MASK) generates a learnable mask based on the decoder output and positional embeddings. @ ColPali provides a mask prior, highlighting the relevant regions of the input document in relation to the question. @ The learnable mask, guided by the mask prior, combined with the original document to create the masked image, where only the highlighted parts are kept visible. 3 The masked image is processed through the Pix2Struct network. @ The text head (TEXT) predicts the answer to the question based the masked input.
objective function LIв is expressed as:
As shown in the figure, our framework also incorporates a pretrained ColPali model (Faysse et al., 2024). The details of its role and integration are discussed in Sec. 3.4.
min LıB = X
x M) - -) (1)
with hyperparameter /3 balancing the contrasting terms.
3.2. Learning sufficient explanation via cross-entropy
These two terms reflect the philosophical foundation of ex- planation, and the principles that define what an effective explanation should be (Choi et al., 2024; Sokol & Flach, 2020). We frame these principles as minimality and suffi- ciency, using them as learning criteria to guide the model as direct learning signals. Minimality emphasizes present- ing only the most pertinent information for understanding a model's decision, while sufficiency ensures that explanations remain consistent and contribute decisively predicting correct answer.
Upon learning the bottleneck representation, the mask M must forward contextually sufficient information to the model answer the question by maximizing the mutual information I(X O M; Following (Amjad & Geiger, 2019), we approximate this by minimizing the cross-entropy loss CE Y),
max I(X O M;Y) -H( (
(2)
min H(Y X o M) = CE
Figure 2 provides an overview of our framework. We build on a pretrained Pix2Struct model as the baseline, updating the encoder, the newly added mask head, and the text head to enable the learning of a self-explainable mask representa- tion. When question and a document image are provided as inputs, the model first predicts a mask, introducing the minimality term I(X; X @ M) to encourage the removal of irrelevant information. The masked input is then processed by the model, together with the same question, to predict the answer based on the masked representation, captured by I(X @ M; Y). Note that the decoder of the Pix2Struct model remains frozen to ensure consistent text prediction.
where H(Y and H(Y X O M) are the marginal and conditional entropies of Y, respectively. Ý denotes the model prediction. Note that H (YY) is constant because the label Y is fixed.
3.3. Learning minimal explanation via continuity loss with L.I norm
Minimality is achieved by reducing the mutual informa- tion between the full document input X and the masked input X @ M, resulting in a representation effiicin mask.
4

---

DocVXQA: Context-Aware Visual Explanations for Document Question Answering
Contrary to the belief that estimating mutual information is inherently challenging (Poole al., 2019; Belghazi al., 2018), recent studies (Skean et al., 2024; Yu et al., 2019) have demonstrated that it can be computed deterministically when combined with kernel density estimation (Yu et al., 2019; Giraldo et al., 2014). This approach to quantifying mutual information deterministic and integrates seam- lessly with gradient descent over mini-batches. However, despite extensive experimentation, this approach did not yield the expected results. A potential limitation may stem from the model's sensitivity to hyperparameter tuning, par- ticularly when dealing with high-dimensional image data, which could have restricted its capacity to reach optimal performance.
derstanding model as a source of prior knowledge to guide mask learning. We utilize its inference results of this model as prior information to facilitate interactive mask learning within our framework. While many models can be used for this purpose, we employ ColPali (Faysse et al., 2024) in our implementation. ColPali is a vision-language retrieval model. Its architecture integrates visual patches and textual tokens to generate multi-vector representations. These rep- resentations are then subjected to late interaction matching mechanism (Liu et al., 2024; Khattab & Zaharia, 2020), which maximizes the interaction between each question to- ken and its corresponding relevant tokens in the document image. This provides a degree of explainability by visu- alizing question-document interactions through heatmaps, offering insights into the model's decision decision-making process. Although these visualizations often fail align with human intuition provide comprehensive explanations due to their reliance on attention distributions, we use them prior mask and refine them within our VXcA framework to enhance relevance and interpretability.
As an alternative, we propose novel step by integrating continuity loss, as known as anisotropic total variation (Bui et al., 2023; Johnson et al., 2016) combined with the L1 norm to enforce sparsity and smoothness in the generated relevance masks. The LI norm promotes sparsity by penal- izing the magnitude of mask values, ensuring that only the most essential regions of the input are highlighted, thereby producing concise and focused explanations. In parallel, the continuity loss enhances spatial coherence by minimiz- ing abrupt transitions between adjacent regions in the mask. This is achieved by penalizing the L1 norm of horizontal and vertical differences in the reconstructed 2D relevance mask. Together, these complementary objectives enable the model to highlight minimal yet structurally coherent regions of importance. By fostering smoothness and mini- mality, this approach mitigates the risk of producing noisy or fragmented masks, ensuring that the explanations are not only interpretable but also intuitively aligned with human understanding.
3.5. Proposed learning objective
Our model achieves the following: (1) it learns to make predictions with masked input documents via cross-entropy loss LCE (Sec.3.2), (2) it minimizes the highlighted re- gions to ensure concise and focused explanations via L11 (Sec.3.3), and (3) it generates more context-aware explana- tions through tokens interaction with the ColPali signals via LMSE (Sec.3.4). The complete objective function LaxQa is defined as:
min Cvxos -MLMsE(MpiMM
(3)
3.4. Learning context-aware explanations through token interactions
Through our experiments with training the model using the objective function in Equation (1), we identified that the learned mask often overfits to regions directly corre- sponding to the answer (see Figure 7 in Appendix). This overfitting reduces the mask's utility, rendering it mere visual reproduction of the textual answer rather than provid- ing meaningful, context-aware insights. Such masks lack the ability to highlight broader, equally relevant information related to the question, leading to unreliable explanations.
where LMSE aligns the learned mask with the prior Mp derived from ColPali, Lt encourages sparsity in the high- lighted regions, and LcE represents the cross-entropy loss for prediction accuracy. The hyperparameters 7 and /3 bal- ance the contributions of each term. All components of the objective function are optimized simultaneously in an end-to-end fashion. As explanations M emerge naturally through this integrated optimization process, our framework inherently exhibits self-explainabiliyy
To address this issue, we emphasize the need to regularize the mask-learning process, not only to mitigate overfitting but also enable learning more generalizable explanations. Specifically, aim to create context-award explanations that identify and integrate the most relevant information across the broader input space. To achieve this, we incor- porate a publicly available, pretrained vision- lasgun-aang un-
3.6. Postprocessing
For more human understandable relevance maps, a postpro- cessing stage is applied, consists in enclosing the connected relevance regions with bounding boxes and keep only boxes with high relevance score. This step, applied only in the inference stage, is further detailed in Appendix A.I.
5

---

DocVXQA: Context-Aware Visual Explanations for Document Question Answering
METHOD
MASK THRESH.
DOCVQA (MATHEW AL.,, 2021)
PFL.-DoCVOK (TITO Al.,, 2024)
Acc.
ANLS f
Pix. RATIO 4
Acc. t
ANLS
Pix. RATIO
PIXZSTRUCT, UNMASKED (LEE ET AL.,, 2023) OUR MODEL, UNMASKED
0.56
0.68
1
0.80
0.92
0,51
0.65
1
0.57
0.79
0.05
0.34
0.46
0.38
0.06
0.13
0.27
0.10
0.20
0.32
0.20
0.04
0.10
0.17
RAW ATTENTION
0.25
0.07
0.13
0.08
0.01
0.04
0.09
0,50
0.03
0.08
0.04
0.00
0.01
0,05
ATTENTION ROLLOUT
0,01
0.03
0.07
0.04
0.00
0.02
0.03
(ABNAR & ZUUDDMA,, 2020)
1 x 10-6
0.06
0.11
0.07
0.00
0.03
0.05
GRAD-CAM (SELVARAJU ET AI.,, 2017)
0.50
0.01
0.05
0.18
0.00
0.03
0.16
COLPALL+PIX2STRUCC (FAYSSE ET AL., 2024)
0.50
0.38
0.50
0.23
0.28
0.39
0.18
0.70
0.38
0.54
0.23
0.43
0.66
0.22
OURS
0.80
0.30
0.46
0.11
0.33
0.54
0.13
Table Evaluation various explainability methods under different mask threshold settings on the Voc task. The results demonstrate the trade-offs between the interpretability and utility of the different approaches. The unmasked approaches are set for reference as upper bounds for utility.
4. Experiments and Results
raw attention (obtained from Pix2Struct's last layer) and attention rollout (Abnar Zuidema, 2020), which use aggregated attention weights across all layers to provide a more holistic view of the model's focus.
4.1. Experimental Setup
We posit that a good explanation map should highlight all the critical regions necessary to answer a question while minimizing irrelevant areas correctly, thus being both suffi-
Gradient Baselines: We implemented Grad-CAM (Sel- varaju et al., 2017), adapting it for the autoregressive sequence output layer by applying it to each output token from the decoder and aggregating the resulting maps. Retrieval Baseline: This approach utilizes ColPali (Faysse et al., 2024), a retrieval-base framework. Here, the in- put (question answer) is first processed by ColPali on its
cient and minimal. We experimentally compare our method against several baseline techniques to evaluate both qualities. The evaluation process is structured as follows: explanation masks generated using different methods, then postpro- cessed and binarized to keep only the relevant information when applied to the original document image. After that, masks are applied and the resulting image is passed to our fine-tuned Pix2Struct to predict the answer. Predicted an- swer is compared with the ground truth (GT) using accuracy and average normalized Levenshtein similarity (ANLS) to assess the sufficiency. Furthermore, the proportion of high- lighted relevant regions in the mask relative the total image area, referred to as the pixel ratio, is calculated to measure mask minimality. An ideal method achieves fa- vorable trade-off between explanation utility (accuracy and ANLS) and minimality (pixel ratio). The experiments are done on two datasets, VQA (Mathew al., 2021) and PFL-DocVQA (Tito al., 2024).
generate heatmap the explanation. This heatmap is then applied to the image, which is subse- quently passed to the Pix2Struct model for prediction. We call this baseline
4.3. Results
Quantitative Evaluation The results obtained are sum- marized in Table I, where different methods are evaluated to balance utility and explainability. As a reference for upper bound utility, we include results for unmasked images using the original Pix2Struct model and our trained model. Al- though model exhibits slight decrease in performance, this can be attributed the challenge of simultaneously managing both masked and clear images during training.
4.2. Baselines
Our approach stands out among the explainable methods, achieving the best utility in terms accuracy and ANLS while maintaining a modest pixel ratio. This highlights the method's 's ability focus on the most critical regions of the
We compare our approach against three categories of base- line methods:
Attention Baselines: In this category, include both
6

---

DocVXQA: Context-Aware Visual Explanations for Document Question Answering
Original Image
Raw Attention (0.5)
Raw Attention (0.1)
Raw Attention (0.05)
Att. rollout
Grad-CAM
Ours (0.7)
Figure 3. comparison of explanations generated by different methods for the question "What the total amount?" with the model's 's answer being "$100,000", Relevance maps with diffcrent (thresholds) applicd the input image keep only the relevant regions, Best viewed at high zoom. Additional qualitative results across diverse contextual scenarios are provided in Appendix B.3.
input image, providing concise yet effective explanations. In contrast, raw attention exhibits reasonable performance when the mask threshold is set very low, as this results in large pixel ratios that cover a significant portion of the im- age. However, as the threshold increases, raw attention's 's utility metrics degrade significantly, suggesting it strug- gles maintain performance when forced to operate with smaller, more answer-overfitted explanations. Attention rollout, while offering a more comprehensive aggregation of attention weights across layers, performs poorly overall. Its failure to accurately localize relevant regions likely due to the propagation errors (missing the contextual regions) across layers, which focused the relevant regions only around the answer. Grad-CAM, despite its popularity, also shows weak performance in this context. This can be attributed to the challenges of adapting Grad-CAM to autore- gressive sequence models, where token-level aggregation may reduce its ability to generate precise explanations. Fi- nally, the ColPali Pix2Struct approach demonstrates com- petitive utility metries with reasonable pixel ratios on the Doc VQA dataset, however, our method achieves even bet-
ter utility at the same ratio, as well significantly better performance on the PFL-DocVQA dataset, which further shows the efficacy of our approach.
Qualitative Evaluation To reflect the results presented in Table we illustrate the relevance masks generated by different methods in Figure 3. Each method was tasked with explaining the model's response to the question "What is the total amount?". where the predicted answer is "$180,000", The explanation masks are overlaid on the input image to highlight regions deemed relevant for the prediction. From the visualizations, it is evident that our method produces explanations that are both precise and comprehensive. Com- pared baseline methods, such as Attention Rollout, and Raw Attention, our approach suc- cessfully isolates all the textual elements contributing the final answer in relevance map that is easy visually pro- cess by human, while avoiding irrelevant regions. Notably, Grad-CAA CAA focuses on certain parts the document that do not contain all the required information to answer the question by the model.

---

DocVXQA: Context-Aware Visual Explanations for Document Question Answering
S
M
TI
ACC.
ANLS 1
PIXEL RATIO +
07
0.51
0.65
1
0.19
0.36
0.02
0.38
0.54
0.23
Table 2. Ablation study on the impact of different learning objec-
0.4
tives. S: sufficieney, minimality, TI: Token Interactions.
0.3
Ancuracy (no-Routrnneesingg)
ANLS Pestgracessing)
Picel Rat Noo-Pottprocessiin
METHOD
CONTEXT f
CLARITY 1
0.1
Accuracy PPostprocessing
ANLS (Postpracesong)
RAW ATTENTION (0.25)
2.90 + 0.48
2.75 + 0.42
Pixel Ratio ins)
0.0
0.4
0.5
0.7
3.8
0.9
RAW ATTENTION (0.50)
2.26 = 0.54
2.34 I 0.60
Applied Threshoid
COLPALI+PIX2STRUCC 3.97 = 0.25
3.02 = 0.42
Figure 4. Performance of our method with and without postpro- cessing, under different thresholds applied to the relevance masks.
OURS (0.7)
4.49 + 0.26
3.56 0.41
Table 3. Results of the human evaluation of the explanation quality for different methods. Participants rated scale of I 5.
When these masked images were fed back into the model, all methods except Grad-CAM produced the correct answer. However, this does not necessarily indicate good relevance maps. In such scenarios, highlighting only any number will lead the DoA use it as answer, even without evidence that it corresponds to the "total". Our method lever- ages the most clear and context-aware explanation. This results in an accurate visualization of the reasoning process, which enhances users' ability to understand the model's decision-making, and fosters greater trust and confidence in the system.
This indicates that the model identifies the entire image as relevant, maximizing coverage but lacking precision. When both sufficiency and minimality losses are employed, the Pixel Ratio drastically decreases to 0.02, demonstrating the effectiveness of minimality in constraining the mask to small regions that are necessary to answer the question. However, this extreme reduction leads to overfitting, as the model disregards accompanying context that may be important for generalization. Consequently, both Accuracy and ANLS sig- nificantly decrease on the test data, highlighting a trade-off between precision and informativeness. Finally, the inclu- sion of token interactions loss (S+M+TI) balances these trade-offs, improving mask coverage on the context leading to achieving significant gains Accuracy and ANLS.
Effect of Post-processing To study the impact of postpro- cessing and thresholding on the resultant masks, we perform the experiment presented in Figure 4. The experiment high- light the trade-offss between precision and coverage when ap- plying different thresholds and when using postprocessing. Lower thresholds achieve higher Accuracy and ANLS but at the cost of significantly larger Pixel Ratios, indicating less precise localization of relevance regions. As the threshold increases, Pixel Ratio decreases, leading to more focused masks but with a corresponding decline in Accuracy and ANLS. When postprocessing is applied, the results show an improvement in the accuracy and ANLS with reduction in Pixel ratio with threshold > 0.7. This improvement under- scores the importance of postprocessing refining the outputs of our method, balancing relevance and mask quality.
Human Evaluation The goal of developing explainable DocVQA models is enhance human trust in these systems. To experimentally assess the quality of the explanations, we conducted human evaluation study with 42 impartial par- ticipants, blind to the details of our method. We used 10 example document images. For each image, participants were shown a corresponding question, the ground truth (GT) answer, and four different explanation masks generated from relevance maps of the top-performing methods seen in Ta- ble 1, based on the trade-offs between Accuracy, ANLS, and Pixel Ratio-the this includes ColPali, raw attention, and our own method. Participants were asked to evaluate each explanation on two key criteria, rated on a scale of to 5:
critical step in
Learning Objectives In our method, three learning objec- tives are used during training: Sufficiency (S), Minimality (M), and Token Interactions (TT). To evaluate their individ- ual and combined contributions to performance, we con- ducted an ablation study, as shown in Table 2. The first row, where only the sufficiency loss is applied, achieves the highest Accuracy and ANLS but has the largest Pixel Ratio.
Context-awareness of the explanation: Participants re- sponded to the question: "How confident are you (1-5) to answer the question using only this masked image?" This question assesses whether the explanation directly supports the answer and provides all the necessary context
8

---

DocVXQA: Context-Aware Visual Explanations for Document Question Answering
answer.
aligned with human reasoning. This is achieved by formu- lating and including fundamental explainability principles directly into the learning process. Our extensive evaluations demonstrate the effectiveness of our framework.
Participants responded to the question: "Does the masked image include only all the necessary information in a clear, concise, and comprehen- sive way to fully explain the model's reasoning? Rate on a scale (1-5)." " This question evaluates the extent to which the explanation provides thorough, focused and human comprehensive coverage of the relevant content.
Clarity of the explanation:
Future research will explore the generalization of our method across diverse Voc architectures and datasets, as well as optimizing the trade-off between explainability and task performance. We envision DocVXQA as a step- ping stone toward robust, transparent, and user-centric Al solutions in document intelligence.
The results of the human evaluation are presented in Table 3. As shown, our method outperforms all baselines in both context-awareness and clarity, achieving the highest scores, compared to ColPali and Raw Attention variants. These results highlight our method's ability to generate clear planations that enable participants to confidently answer questions without the original image, while also providing comprehensive and focused masks that include only the nec- essary information. Thus, using our method enhances user trust and understanding of the model's predictions.
Acknowledgments
We thank all participants for their contributions to the hu- man preference study. This work has been supported by the Consolidated Research Group 2021 SGR 01559 from the Research and University Department of the Catalan Gov- ernment, and by project 11722023-144426DD 100 funded by MCIU/AEI/1O 1039/500 10001 1033 and FSE+. This work has been funded by the European Lighthouse on Safe and Secure AI (ELSA) from the European Union's Horizon Europe programme under grant agreement No 101070617. The work of Changkyu Choi is supported by Visual Intelli- gence, centre for research- baaeed innovation funded by the Research Council of Norway and its consortium partners (RCN grant no. 309439).
Furthermore, conducted another human preference study to directly compare the masks of our method against the top-performing baseline in the previous study, that is Col- Pali+Pix2Struct. In total, 12 participants evaluated 21 ran- domly selected question-answe As result, our method was preferred in 163 trials (64.7%; 95% CI [58.4%, 70.6%], p << 0.001), with all participants (12/12) favoring our approach overall. Thus, our method is offering significantly more compact and interpretable explanations than ColPali+Pix2Struct.
pairs (252 trials in total).
Impact Statement
To the best of our knowledge, this work is the first to introduce explainability into the DocVQA applications. As DocVQA systems are increasingly deployed in high- stakes domains such as banking, healthcare, and public administration-where automated document understanding aims to reduce human involvement- the demand for trans- parent and trustworthy models becomes paramount. Our approach addresses this urgent gap by enabling interpretabil- ity of DocVQA models, which is essential for ensuring re- liability, fostering user trust, and facilitating responsible deployment of Al in sensitive real-world contexts.
Model Agnosticity Our DocV framework is de- signed to be model-agnostic. To demonstrate its architec- tural flexibility, we implemented with Donut (Kim al., 2021) an alternative to Pix2Struct. Notably, Donut and Pix2Struct differ fundamentally in how they incorporate the question. Pix2Struct renders the question directly onto the input image, while Donut tokenizes the question and uses it to condition the decoder during generation. To accommo- date this difference, we adapt our method by modifying the mask head to also incorporate the encoded question. Quali- tative results presented in Appendix B.4 illustrate that our method consistently identifies relevant regions across both backbones, validating its model-agnosta design.
References
Abdallah, A., Piryani, B., and Jatowt, A. Exploring the state of the art in legal qa systems. Journal of Big Data, 10(1): 127, 2023.
5. Conclusion
This paper introduced DXQA the first self- exlf-explainable model for DOc,QA, enhancing interpretability by learn- ing relevance maps that elucidate the model's decisions. Unlike conventional approaches that rely on post-hoc ex- planations or merely justify answers without deep contex- tual grounding, VXQA inherently generates visual and context-award explanations to the answers, ensuring that the model's process remains transparent, interpretable, and
Abnar, S. and Zuidema. W. Quantifying attention flow in transformers. In Proceedings of the 58th Annual Meet- ing of the Association for Computational Linguistics, pp. 2020.
Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L.,, Almeida, D., Altenschmidt, J., Altman, S.,

---

DocVXQA: Context-Aware Visual Explanations for Document Question Answering
Anadkat, S., al. Gpt-4 technical report. arXiv preprini arXiv:2303.00877, 2023.
Chefer, H., Gur, S., and Wolf, L.. Generic attention-model explainability for interpreting bi-modal and encoder- decoder transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 397- 406, 2021a.
Aggarwal, K., Khandelwal, A., Tanmay, K., Khan, O. M., Liu, Q. Choudhury, M., Chauhan, H. H., Som, S., Chaudhary, V., and Tiwary, S.
un- derstanding by language-image network. arXiv preprint arXiv:2305. 142/1/////2 2023.
Chefer, H.,, Gur, S., and Wolf, L. Transformer interpretabil- ity beyond attention visualization. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 782-791, 2021b.
Alvarez Melis, D. and Jaakkola, T. Towards robust in- terpretability with self-explaining neural networks. Ad- vances in neural information processing systems, 31, 2018.
Choi, C., Yu, S., Kampffimeyer, M., Salberg, A.-B., Hande- gard, N. O., and Jenssen, R. Dib-x: Formulating explain- ability principles for self-explainable model through information theoretic learning. In ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 7170-7174. IEEE, 2024.
Amjad, R. A. and Geiger, B. C. Learning representations for neural network-baser classification using the information bottleneck principle. IEEE transactions on pattern analy- sis and machine intelligence, 42(9):2225-2239, 2019.
Appalaraju, S., Tang, P., Dong, Q., Sankaran, N., Zhou, Y., and Manmatha. R. Doeformerv2: Local features for document understanding. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 709- 718, 2024.
Covert, I,, Lundberg. S., and Lee, S.-I. Explaining by remov- ing: A unified framework for model explanation. Journal of Machine Learning Research, 22(209):1 2021.
Davis, B., Morse, B., Price, B., Tensmeyer, C.,, Wigington, C., and Morariu, V. End-to-end document recognition and understanding with dessurt. In European Conference on Computer Vision, pp. 280-296. Springer, 2022.
Bach, S., Binder, A., Montavon, G., Klauschen, F., Müller, K.-R., and Samek, W. On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation. PloS one, 10(7):e013140, 2015.
Faysse, M., Sibille, H., Wu, T., Omrani, B., Viaud, G., Hudelot, C., and Colombo, Colpali: Efficient document retrieval with vision language models. arXiv preprint arXiv:2407.0144 2024.
Bang, S., Xie, P., Lee, H., Wu, W., and Xing, F. Explaining a black-box by using a deep variational information bot- tleneck approach. Proceedings of the AAAI conference on artificial intelligence, volume 35. pp. 11396-11 11044, 2021.
Gautam, S., Boubekki, A., Hansen, S., Salahuddin, S., Jenssen, R., Höhne, M., and Kampffmeyer, M. Protovae: A trustworthy celf-explainable prototypical variational model. Advances in Neural Information Processing Sys- lems, 35: 11119974-99555 2022.
Belghazi,, I., Baratin, A., Rajeshwar, S., Ozair, S., Bengio, Y., Courville, A., and Hjelm, R. Mine: Mutual in- formation neural estimation. In Proceedings of the 35th International Conference Machine Learning (ICML), volume 80, pp. 531-540. PMLR, 2018.
Gautam, S., Höhne, M. M.-C., Hansen. S., Jenssen, R., and Kampffmeyer, M. This looks more like that: Enhancing -ell--nxi models by prototypical relevance propa- gation. Paltern Recognition, 136: 109172, 2023.
Blau, T., Fogel, S., Ronen, R.. Golts, A., Ganz, R., Ben Avra- ham, E., Aberdam, A., Tsiper, S., and Litman, R. Gram: Global reasoning for multi-page vqa. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 15598-15607, 2024.
Giovannini, S., Coppini, F., Gemelli, A., and Marinai, Boundingdocs: a unified dataset for document ques- tion answering with spatial annotations. arXiv preprint arXi::2501.034003 2025.
Bousselham, W., Petersen, F., Ferrari, V., and Kuehne, H. Grounding everything: Emerging localization properties in vision-language transformers. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 3828-3837, 2024.
Giraldo, L. G. S., Rao, M., and Principe, J. C. Measures of entropy from data using infinitely divisible kemelss. IEEE Transactions on Information Theory, 61()5:335- 548, 2014.
Bui, K., Lou, Y., Park, F., and Xin, J. Weighted anisotropic- isotropic total variation for poisson denoising. In 2023 IEEE International Conference on Image Processing (ICIP), pp. (020-1024, IEFE, 2023.
Jenssen, R. Map it to visualize representations. In The Twelfth International Conference on Learning Represen- tations, 2024.
10

---

DocVXQA: Context-Aware Visual Explanations for Document Question Answering
Johnson, J., Alahi, A., and Fei-Fei, L. Perceptual losses for real-time style transfer and super-resolution. In Com- puter Vision-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Pro- ceedings, Part II 14, pp. 694-711. Springer, 2016.
Møller, B. L.,, Igel, C., Wickstrøm, K. K., Sporring, J., Jenssen, R., and Ibragimov, B. Finding nem-u: Explain- ing unsupervised representation learning through neural network generated explanation masks. In Forst st In- ternational Conference on Machine Learning, 2024.
Müller, R. How explainable ai affects human performance: A systematic review of the behavioural consequenees of saliency maps. International Journal of Human- Computer Interaction, pp. 1-32, 2024.
Khattab, O. and Zaharia, M. Colbert: Efficient and effective passage search via contextualized late interaction over bert. Proceedings of the 43rd International ACM SIGIR conference on research and development in Information Retrieval, pp. 39-48, 2020.
Parashar, S., Lin, Z.,, Liu, T., Dong, X., Li, Y., Ramanan, D., Caverlee, J., and Kong, S. The neglected tails in vision-languast models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12988-12997, 2024.
Kim, G., Hong, T., Yim, M., Park, J., Yim, J., Hwang, W., Yun, S.. Han, D., and Park, S. Donut: Document understanding transformer without ocr. arXiv preprint arXiv:2111.1564, 2021.
Petsiuk, V. Rise: Randomized input sampling for explanation of black-box models. arXiv preprint arXiv:1806.0777 07421. 2018.
Kim, G., Hong, T., Yim, M., Nam. J., Park, J., Yim, J., Hwang, W., Yun, S., Han, D., and Park, S. Ocr-free docu- ment understanding transformer. In European Conference on Computer Vision, pp. 498-517. Springer, 2022.
Poole, B., Ozair, S., Van Den Oord, A., Alemi, A., and Tucker, G. On variational bounds of mutual information. In International Conference on Machine Learning, pp. 5171-5180. PMLR, 2019.
Lee, K., Joshi, M., Ture, R., Hu, H., Liu, F. Fisensch- los, J. M., Khandelwal, U., Shaw, P., Chang, M.-W,, and Toutanova, K. Pix2struct: Screenshot parsing pretraining for visual language understanding. In Inter- national Conference on Machine Learning, pp. 18893- 18912. PMLR, 2023.
Powalski, R., Borchmann, E., Jurkiewicz, D., Dwojak, T., Pietruszka, M., and Palka, G. Going full-tilt boogie on document understanding with text-image-layout trans- former. In Document Analysis and DecoonitiionICDDA 2021: 16th International Conference, Lausanne, Switzer- land, September 5-10, 2021, Proceedings, Part Il 16, pp. 732 747. Springer, 2021.
Li, C., Wong, C., Zhang, S., Usuyama, N., Liu, H., Yang. J., Naumann, T., Poon, H., and Gao, J. Llava-med: Training large language-and-vvsion assistant for biomedicine in one day. Advances in Neural Information Processing Systems, 36, 2024.
Principe, J. C. Information theoretic learning: Renyi's 's entropy and kernel perspectives. Springer Science & Business Media, 2010.
Liu, Q. Guo, G., Mao, J., Dou, Z., Wen, J.-R., Jiang, H., Zhang, X., and Cao, Z. An analysis on matching mech- anisms and token pruning for late-interaction models. ACM Transactions on Information Systems, 42(5):1-28, 2024.
Rasheed, H., Maaz, M., Shaji. S., Shaker, A., Khan, S., Cholakkal, H., Anwer, R. M., Xing, E.,, Yang, M.-H., and Khan, F. S. Glamm: Pixel grounding large multimodal model. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 13009- 13018, 2024.
Mathew, M., Karatzas, D., and Jawahar, C. Doevqa: A dataset for vqa on document images. In Proceedings of the IEEE/CVE winter conference on applications of computer vision, pp. 2200-2209 2021.
Ribeiro, M. T., Singh, S., and Guestrin, C. trust you?" explaining the predictions of any classifier. Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining, pp. 1135-1144, 2016
why should
Mathew, M., Bagal, V., Tito, R., Karatzas, D., Valveny, E., and Jawahar, C. Infographievga. In Proceedings of the IEEE/CVF Winter Conference Applications of Computer Vision, pp. 1697-1706, 2022.
Rudin, C. Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead. Nature machine intelligence, 1(5):206- 215, 2019.
Mohammadshirazi, A., Neogi, P. P. G., Lim, S.-N., and Ramnath, R. Dlava: Document language and vision assis- tant for answer localization with enhanced interpretability and trustworthiness. arXiv preprint arXiv:2412.00151, 2024.
Schulz, K., Sixt, L., Tombari, F., and Landgraf, T. Re- stricting the flow: Information bottlenecks for attribution. arXiv preprint arXiv:2001. 000396, 2020.

---

DocVXQA: Context-Aware Visual Explanations for Document Question Answering
Selvaraju, R. R.,, Cogswell, M., Das, A., Vedantam, R., Parikh, D., and Batra, D. Grad-cam: Visual explana- tions from deep networks via gradient-based localization. In Proceedings of the IEEE international conference on computer vision, pp. 61 618-626,77 2017.
Ye, J., Hu, A., Xu, H., Ye, Q.. Yan, M., Xu, G., Li, C.,, Tian, J., Qian, Q.. Zhang, J., al. Ureader: Universal ocr- free visually-situate language understanding with mul- timodal large language model. In The 2023 Conference on Empirical Methods in Natural Language Processing, 2023.
Shannon, C. E. A mathematical theory of communication. The Bell system technical journal, 27(3):379-42 423,, 1948.
Yu, S., Giraldo, L. G. S., Jenssen, R., and Principe, J. C. Multivariate extension of matrix-based rényi's a-order entropy functional. IEEE transactions on pattern analysis and machine intelligence, (011)):2900-2960 2019.
Shrikumar, A., Greenside, P., and Kundaje, A. Learning important features through propagating activation differ- ences. In International conference on machine learning, pp. 3145-3153 PMIR, 2017.
Yu, S., Tang, C., Xu, B., Cui, J., Ran, J., Yan, Y., Liu, Z., Wang, S., Han, X., Liu, Z., et al. Visrag: Vision- based attrivall-augmente generation on multi-modality documents. arXiv preprint arXiv:2410.10594 2024a.
Skean, O., Dhakal, A., Jacobs, N., and Giraldo, L. G. S. Frossl: Frobenius norm minimization for self self-supervised learning. Proceedings of the European Conference on Computer Vision (ECCV), 2024.
Yu, S., Yu, X., Lokse, S., Jenssen, R., and Principe, J. C. Cauchy-schwarz divergence information bottleneck for regression. In The Twelfth International Conference on Learning Representations, 2024b.
Skean, O., Dhakal, A., Jacobs, N., and Sanchez Giraldo, L. G. Frossl: Frobenius norm minimization for efficient multiview self-supervised learning. In European Confer- ence on Computer Vision, pp. 69-85. Springer, 2025.
Zeiler, M. D. and Fergus, R. Visualizing and understand- ing convolutional networks. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part I 13. pp. 818- 833. Springer. 2014.
Sokol, K. and Flach, P. Explainability fact sheets: framework for systematic assessment of explainable ap- proaches. In Proceedings of the 2020 conference on fairness, accountability, and transparency, pp. 56-67, 2020.
Zhang, H. Torres, F., Sicre, R., Avrithis, Y., and Ayache, S. Opti-cam: Optimizing saliency maps for interpretability. Computer Vision and Image Understanding, 248: 104001, 2024.
Tishby, N. and Zaslavsky, N. Deep learning and the infor- mation bottleneck principle. In 2015 ieee information theory workshop (ihw), pp. 1-5. IEEE, 2015.
Tito, R., Karatzas, D., and Valveny, E. Hierarchical multi- modal transformers for multipage docvqa. Pattern Recog- nition, 144:109834, 2023.
Zhang, Y., Zhang, R., Gu, J., Zhou, Y., Lipka, N., Yang, D., and Sun, T. Llavar: Enhanced visual instruction tuning for text-rich image understanding. arXiv preprint arXiv:2306.177 2023.
Tito, R., Nguyen, K., Tobaben, M., Kerkouche, R., Souibgui, M. A., Jung, K., Jälkö, J., D' Andecy, V. P., Joseph, A., Kang, L., et al. Privacy-aware document visual question answering. In International Conference on Document Analysis and Recognition, pp. 199-218. Springer, 2024.
Zhao, Y., Pang. T., Du, C., Yang. X., Li, C., Cheung, N.- M. M., and Lin, M. On evaluating adversarial robustness of large vision-language models. Advances in Neural Information Processing Systems, 36, 2024.
Wang, P., Bai, S., Tan, S., Wang, S., Fan, Z., Bai, J., Chen, K., Liu, X., Wang, J., Ge, W., al. Qwen2-vl: Enhancing vision-language model's perception of the world any resolution. arXiv preprint arXiv:2409. 12111 2024.
Zhmoginov, A., Fischer, I., and Sandler, M. Information- bottleneck approach to salient region discovery. In Ma- chine Learning and Knowledge Discovery in Databases: European Conference, ECML. PKDD 2020, Ghent, Bel- gium, September 14-18, 2020, Proceedings, Part III, pp. 531-546. Springer, 2021.
Wu, S., Irsoy, O., Lu, S., Dabravolski, V., Dredze, M., Gehrmann, S., Kambadur, P., Rosenberg, D., and Mann, G. Bloomberggpt: A large language model for finance. arXiv preprint arXiv:2303. /7644, 2023.
Zhou, Y., Chen, Y., Lin, H., Yang, S., Zhu, L., Qi, Z., Ma, C., and Shan, Y. Doge: Towards versatile vi- sual document grounding and referring, arXiv preprint 11X 2024.
Xu, Y., Li, M., Cui, L., Huang, S., Wei, F., and Zhou, M. Layoutlm: Pre-training of text and layout for document image understanding. In Proceedings of the 26th ACM SIGKDD international conference on knowledge discov- ery & data mining, pp. 1192-1200, 2020.
12

---

Appendix
To ensure consistency, we continue the numbering of figures and tables from the main paper. The first references in the Appendix begin with Figure and Table
A. Implementation Details
A.1. Postprocessing
The relevance map produced by method is passed throught postprocessing step that envolve stages: (1) background removal, (2) connected region bounding, In the following we detail each step, noting that for fair comparison, these three steps are applied to all the methods presented in Table I:
Background Removal. Since the output mask may highlight irrelevant background regions, we first filter out the highlighted regions that fall input image tokens that represent background. other words, we discard tokens whose encoded patch variance are below a threshold of 0.01, as they are considered background the document images, where this domain the background is usually blank. This step ensures that only informative regions contribute the final relevance map. In Figure 5, show extreme case of this scenario, where the produced relevance map contain lot of background regions, after applying this postprocessing step, we can see that the quality of the map is significantly improved.
TOoo
FAOAANA AND IN
AREDANCA AND OFT
Predicted Mask (with bg regions)
Predicted Mask (without bg regions)
Original
Figure S. Visualization of the background removal step in methods. Question: "What is the estimated budget of 'conduct analysis of decision makers/ information targets' in research and development?". Model answer "S 7,000.00".
Bounding connected regions. After filtering background regions, the remaining mask effectively highlights relevant information, shown in Figure 5. However, the raw mask may still appear fragmented and less intuitive for users. To enhance interpretability, we apply additional postprocessing step that identifies connected components within the relevance map and encloses them in bounding boxes. We chose this approach because text in documents is best highlighted using bounding boxes, ensuring clear visibility and structured localization. Unlike other forms of heatmaps, which may appear vague or ambiguous, bounding boxes provide a precise, casily interpretable representation of the model's focus which facilitate processing by human users.
13

---

DocVXQA: Context-Aware Visual Explanations for Document Question Answering
After that, rank these regions based on their confidence scores and retain only the top l most relevant boxes. This step, shown in Figure 6, ensures that the final explanation is both compact and user-friendly. In our study, we experimented with different values of k and found that & provided the best balance between conciseness and completeness.
POLLOWI
Monto
Monba
Mosi
Predicted Mask
Original
Predicted Mask
(with full postprocessing)
Figure 6. Visualization of connecting the relevance regions in the heatmap and keeping only region with top confidence score (k 1). Question: "What is the estimated budget of 'conduct analysis of decision makers/ information targets' in research and development?". Model answer "S 7,000.00"..
A.2. Mask head
The implemented mask head a fully connected neural network (MLP-based decoder) designed to process an input consists of the concatenation of the encoded tokens (features). decoder attentions and the positional encoding. The mask head generates relevance score for each token, producing relevance map that highlights the most important regions in terms of influence the final answer. During training, the output of the mask head is passed through Sigmoid activation function to ensure that the values are normalized between 0 and 1, applied to the input image, and then passed to the Pix2Struct component for the masked image prediction.
A.3. Hyperparameters
We summarize the values of the hyperparameters used in this paper. We not that the Pix2Struct component is first fine-tuned on the VoA dataset (Mathew et al., 2021), then our proposed model is trained according to the settings presented in Table 4. Note that in our work, we use the base version of Pix2Struct with 282M parameters. The model is composed of 12 encoder and 12 decoder layers with a hidden size of 768.
HYPERPARAMETER
VALUE 1 10
LEARNING RATE
BATCH SIZE
5 ADAMW 0.5
OPTIMIZER
8
5
THRESHOLD (k) FOR POSTPROCESSING
3
Table HYPERPARAMETER SETTINGS
14

---

DocVXQA: Context-Aware Visual Explanations for Document Question Answering
B. Additional Results
B.1. Selection of
As illustrated in Figure 6, during the post-processing step, we retain only the top-k most relevant boxes. We conducted an ablation study to evaluate the impact of different i values and found that offers the best trade-off between conciseness and completeness. The full results across various values are reported in Table 5, As shown, k 3 is considered the best choice for good trade-off between interpretability and utility.
ACCURACY
ANLS
PIXEL RATIO
0.36
0.52
0.21
0.38
0.53
0.23
0.38
0.54
0.24
4
0.39
0.55
0.25
0.39
0.55
0.26
0.40
0.56
0.28
10
0.41
0.57
0.29
15
0.42
0.58
0.31
20
0.43
0.58
0.32
0.45
0.59
0.38
100
0.45
0.60
0.40
Table 5. A study varying the number of top relevant boxes during post-processing.
B.2. The effect of using token interactions.
As we discussed in Section 3.4, training the model using the objective function in Equation (1) (without the use of token interactivity loss), lead to overfitting to the regions that directly corresponding to the answer. This overfitting reduces the mask's 's utility, rendering it a mere visual reproduction the textual answer rather than providing meaningful, context-aware insights. This can seen in Figure 7, where we show qualitative results of our method.
15

---

DocVXQA: Context-Aware Visual Explanations for Document Question Answering
What is the total foreign exchange used for raw materials(Rs.lac)) Answer: "450.4"
Clear Answer: "450.4"
Clear Answer: "450.4"
Masked Answer: "450.4"
Masked Answer: "450.4"
yineg dunge
Haghi,
$.50
What is the equipment maintenance expenses?
Clear Answer: "6,000"
Clear Answer: "6,000"
Answer: "6,000"
Masked Answer: "6,000"
Masked Answer: "6,000"
Tagler
What is the delivery point mentioned in the form?
Clear Answer:
"philips bank"
Clear Answer: "miops Reid"
GT Answer: "phipps bend"
Masked Answer:
"philips bank"
Masked Answer: "shipping pound"
Figure 7. Visualization of the effect of using token interactions in method. Left: Original Image. Middle: Masked image with relevance map learned without token interactions loss. Right: Masked image with relevance map learned with token interactions loss.
16

---

DocVXQA: Context-Aware Visual Explanations for Document Question Answering
B.3. Additional qualitative results
In this section, we present the results of the applied relevant regions masks on the images (right) across diverse contextual scenarios, in comparison to the corresponding full images (left).
Complex layouts et al., 2022).
We test our approach on documents with complex layouts from the Infographic dataset (Mathew
Palestinlan Population Worldwide by Place of Residence the Beginning of Iyy thousands)
1,12
6.%
101111
Figure& 8. Question: What is the total percentage of Palestinians residing at places other than West Bank and Arab countries? Answer: 32.6 % (correct).
POVERTY
90%
15%
18%
11,8%
18%
11,8%
MVF
UNDRIP LEVEENT SIZE
BEAGONS
GT
&
Figure Question: What the percentage of women with disabilities in low income countries? Answer: 22.1 % (correct).
17

---

DocVXQA: Context-Aware Visual Explanations for Document Question Answering
Densely populated documents
We present results documents containing dense textual content.
Figure 10. Question: Under Private service how many patients were discharged in Neurology?
Answer: 696 (correct).
Office Scoerrs Deperee
Figure 11. Question: What is the estimated cost of Space and Rentals?
Answer: 700 (correct).
18

---

DocVXQA: Context-Aware Visual Explanations for Document Question Answering
Degraded quality
We present results documents with degraded visual quality.
This Elemon atlnk
with Seee Part
3NSTDO roreve sbrieed wAs
and inengrrii aeceriste coleenoe auch
MEDI skpr Loveis prbcice vouce
WiTACo HEOO,, cerstes necett we wnteene Agremmene end year tiea there
MOTI
Figure 12. Question: What is the page number?
Answer: 9 (correct).
SHIS TOASS IS
a/t #1 CDED #3
GALDURT CHDO N.
ARRIVAL NOTICE
CHICAGO, MERGWIDGE COMSIGED YO N DUSTIMS EVDER C. B. A. 4..4 5 BRSWOLS BATORUN NABEOOSE CO
1525 NEWBLRRY AE. CHICAGO EL
CHICADO, LLLNOCS
Marnhnin And Carage Charpe C. O.
DESTBUS BODER
GRISWOLD & BATEMAN WAREHOUSE CO.
Figure 13, Question: How many cases of artworks are there in the shipment?
Answer: 1 (correct).
OSY Lalllngg
Figure 14. Question: Which bank's check this?
Answer: The chase manhattan bank (correct).
19

---

DocVXQA: Context-Aware Visual Explanations for Document Question Answering
B.4. Results with a different backbone
In this section, we show the results of Doc VXQA using the Donut backbone (Kim al., 2021), and compare it with the Pix2Struct backbone. The obtained results are given in Figure 15, Figure 16 and Figure 17, demonstrating the model-agnostic capability of our method. These figures highlight the relevant regions produeed by Doc using two different backbones while answering the same questions.
Original Image
Result with Pix2Struct backbone
Result with Donut backbone
3:10
PuStect: NoOk SEBRANC
19) Nost
All
kithough
badty
Gines sccerned dree biat daaks The
en Staee
Clreetonn siaeratiie slmat all
CeC anrhire
431091289
Figure 15. Question: Who is this question addressed to?
Answer: bob bexon (correct).
Original Image
Result with Pix2Struct backbone
Result with Donut backbone
SYM
Figure 16. Question: What is the amount for publising one poster abstract?
Answer: 58,500 (correct).
20

---

DocVXQA: Context-Aware Visual Explanations for Document Question Answering
Original Image
Result with Pix2Struct backbone
Result with Donut backbone
MOMET ROYALTY RAYMENT FORN
SHOTS
Figure 17. Question: what type of communication is issued by Biomet, inc.?
Answer: royalty payment form (correct).
21


Reviews:
[{"review_id": "HdalaFaHlb", "reviewer": "Reviewer_9NT1", "summary": "First, it introduces DocVXQA, a novel self-explainable framework that not only answers questions about documents but also provides visual explanations highlighting relevant regions that justify the answers. \nSecond, it quantitatively formulates explainability principles (sufficiency and minimality) as explicit learning criteria using an information bottleneck approach, creating a balance between contextual relevance and representation efficiency.\nThird, it proposes an end-to-end training strategy that maintains compatibility with existing DocVQA architectures while adding minimal architectural changes.\n\n## Final recommendation and Justification (post rebuttal)\n\nThe authors have provided substantial details in response to my comments, particularly by incorporating new datasets to address concerns around generalization. I am changing my rating to weak accept and expect that the authors include these additional details and results in the final version.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "The authors thank Reviewer 9NT1 for their feedback. We appreciate the recognition of our approach’s novelty in addressing the challenge of transparency in DocVQA systems, and the acknowledgment that our approach provides a strong theoretical foundation through the quantitative formulation of explainability principles. Below, we address all of 9NT1’s concerns.\n\n1. **What is the performance on documents with complex layouts or multiple tones of writing? What are the limitations in handling highly complex or densely populated docs?**:\nIn the first case, we found that our model still performs well and produces good explanations. In densely populated documents, our model can provide fragmented explanations, which are hard to interpret. See qualitative results here: https://shorturl.at/nstj0. We will discuss these results in the final manuscript.\n\n2. **What are the computational costs compared to baseline approaches?**: Our method requires additional training compared to post-hoc approaches. However, at inference time, the additional computational cost is only the forward pass through the small mask proposal head (591361 parameters). All baseline comparisons are performed with respect to the Pix2Struct base model with 282M parameters, aside from ColPali, which itself uses 2,9B parameters.\n\n3. **Can the method be extended to other vision-language understanding tasks? Why using of different metrics than ColPali and Pix2struct?**:\nThe masking-based method can be readily generalized to other vision-language tasks. Nevertheless, the primary goal of our work is to provide explanations tailored for the DocVQA task, rather than aiming at making Pix2Struct itself explainable.\nFor the metrics used, we clarify that our specific focus is on DocVQA, thus retrieval metrics such as NDCG (used in ColPali) are not applicable.  We adopt standard DocVQA metrics that are also used by Pix2Struct for QA tasks. \n\n4. **What is the relation of your work with mutual information estimation?**:\nWe appreciate the reviewer’s reference to Doshi-Velez and Kim (2017). Our core motivation closely corresponds with both the data-driven methodology for uncovering interpretability factors (Section 4.1) and the functionally-grounded evaluation outlined in Section 3.3. We concur with the suggestion of adopting an application-grounded evaluation framework; we plan to apply our methodology to domain-specific DocVQA tasks in future research.\n\n5. **How robust is the method to variations in document image quality and why there is a degradation of performance on masked images?**:\nSPDocVQA contains some images of bad quality, thus our model is robust to these variations, some qualitative results here: https://shorturl.at/mqbzg.  Regarding the performance degradation from masked to unmasked models: the trade-off between accuracy and explainability is a well-documented phenomenon (Crook et al., REW'23, Bell et al., ICPS'22), and several approaches have been proposed to balance this trade-off (Luo et al., BJR Open'19, Mori et al., ESE'23). As illustrated in Fig. 4: lowering the threshold decreases explanation quality but improves utility. This shows that the observed performance gap is an inherent constraint in explainability rather than a fundamental limitation of our approach. We also want to clarify also that the 38% is the accuracy of masked images; for clear images, the accuracy is 51%.\n\n6. **Why using different thresholds and why there is a heavy reliance on post-processing**:\nWe adjust the thresholds so that each technique operates at its optimal setting, to ensure a fair comparison between competing methods.\nThe post-processing steps are primarily intended to enhance human interpretability of the masks, and do not significantly affect the core performance of our model. As shown in Figure 4, the observed gains are primarily attributable to the underlying learning approach rather than to the post-processing. \n\n7. **The paper lacks a proper ablation study showing how ColPali and Pix2Struct perform individually**:\nWe would like to clarify that it is impossible to evaluate each technique's individual impact, as they are designed to be used only in combination for XDocVQA.\n\n8. **The baseline numbers in the top row appear to outperform many of the tested approaches.**\nWe would like to clarify that these are not the baselines but rather upper bounds (unmasked, no explanation). We will make this distinction clear in the paper.\n\n9. **Human evaluation lacks statistical rigor with 26 participants evaluating only 10 examples**:\nIt is challenging to include more documents because we have to redo the study, but we have increased the number of participants to 42, and found that the results remained consistent as seen in this table:\n\n| Method | Context ↑ | Clarity ↑ |\n|-----|------|------|\n| Raw Attn (0.25)| 2.90 ± 0.48| 2.75 ± 0.42|\n| Raw Attn (0.50)| 2.26 ± 0.54| 2.34 ± 0.60|\n| ColPali+Pix2Struct| 3.97 ± 0.25| 3.02 ± 0.42|\n| Ours (0.7)| **4.49 ± 0.26**|**3.56 ± 0.41**|", "claims_and_evidence": "The submission makes several key claims:\n1. DocVXQA provides context-aware visual explanations for document question answering, going beyond simply highlighting answer locations.\n1. The framework delivers explanations that are both contextually sufficient yet representation-efficient, creating a balance between comprehensive context and minimizing irrelevant information.\n1. The method outperforms baseline explanation techniques in both utility (accuracy/ANLS) and explainability (clarity and conciseness).\n1. Human evaluators prefer DocVXQA's explanations for their context-awareness and clarity.", "methods_and_evaluation_criteria": "The proposed methods make sense for explaining document question answering. The approach balances showing enough information to understand the answer while keeping explanations simple. The testing approach has some weaknesses - they should have tested with more people and different types of documents. Also, the big drop in accuracy when using explanations suggests this might not work well in real-world settings where getting the right answer is critical.", "theoretical_claims": "The paper makes no theoretical claims", "experimental_designs_or_analyses": "which ones, and discuss any issues.\n1. The paper fails to justify why they use different thresholds for their method compared to previous approaches. This inconsistency raises questions about fair comparison methodology.\n1. The paper lacks a proper ablation study showing how ColPali and Pix2Struct perform individually, making it difficult to assess the actual contribution of their proposed integration.\n1. There is a confusing presentation in the results tables where higher Accuracy is stated as better, yet the baseline numbers in the top row appear to outperform many of the tested approaches.\n1. The authors inexplicably use different evaluation metrics than those presented in the original ColPali and Pix2Struct papers, hindering direct comparison with established benchmarks and raising concerns about cherry-picking favorable metrics.\n1. The results section is particularly difficult to follow since none of the tables are properly referenced in the text, leaving readers to guess the relationships between claims and supporting evidence.\n1. The paper fails to demonstrate generalizability by not evaluating performance on standard tasks used in previous works, such as Screenshot Parsing Pretraining, AI2D, and Screen2Words, which were core evaluation benchmarks in related literature.", "relation_to_broader_scientific_literature": "DocVXQA builds upon several key research areas. It extends beyond simple attention visualization techniques (Chefer et al., 2021, https://arxiv.org/abs/2103.15679) by developing contextually sufficient explanations rather than just highlighting answer locations. The token interaction approach connects to knowledge distillation in vision-language models (Faysse et al., 2024, https://arxiv.org/abs/2407.01449). While innovative, the work misses opportunities to relate their challenges with mutual information estimation to similar challenges in explainable AI research and would benefit from more rigorous human evaluation protocols (Doshi-Velez & Kim, 2017, https://arxiv.org/abs/1702.08608).", "essential_references_not_discussed": "I am not aware of any missing references.", "other_strengths_and_weaknesses": "## Strengths:\nThe paper's strengths lie in its novel approach to a significant problem in document AI - the lack of transparency in document visual question answering. The paper effectively identifies and addresses the limitations of conventional attention-based explanations that merely highlight answer regions without proper context. The quantitative formulation of explainability principles (sufficiency and minimality) provides a sound theoretical foundation.\n\n## Weaknesses:\nFirst, the performance degradation from unmasked (56% accuracy) to masked models (38% accuracy) represents a substantial trade-off that undermines real-world applicability. Second, the heavy reliance on post-processing (background removal, bounding boxes) obscures how much improvement comes from the learning approach versus these enhancement techniques. Fourth, the human evaluation lacks statistical rigor with 26 participants evaluating only 10 examples each.", "other_comments_or_suggestions": "Typo on Line 99- redundant are", "questions_for_authors": "1. How does the method perform on documents with complex layouts or multiple tones of writing?\n1. What are the computational costs compared to baseline approaches?\n1. Can the approach be extended to other vision-language understanding tasks?\n1. How robust is the method to variations in document image quality?\n1. What are the limitations in handling highly complex or densely populated documents?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Yv4W8faAjd", "reviewer": "Reviewer_ovki", "summary": "This paper proposes DocVXQA, a novel self-explainable framework for Document Visual Question Answering (DocVQA), designed to not only answer questions from document images but also provide context-aware visual explanations via learned relevance heatmaps. The core contribution lies in integrating explainability criteria—namely sufficiency, minimality, and context-awareness—directly into the training process, using an additional mask-generation module guided by external pretrained vision-language retrieval models. Experimental evaluations on the DocVQA dataset demonstrate that DocVXQA effectively balances interpretability and prediction performance, surpassing several baselines in both quantitative metrics and human evaluations.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "The authors sincerely appreciate Reviewer ovki’s insightful comments. We are pleased that the reviewer recognized the paper’s key strengths, as he mentioned its originality in integrating interpretability into DocVQA, its effective balance between interpretability and prediction performance, and the \"conceptual soundness\" of our theoretical formulations.\nWe have carefully considered all suggestions and made a dedicated effort to address each point thoroughly.\n\n1. **I cannot confirm the generalizability of your approach. Additional experiments on diverse datasets are needed**: We have extended our evaluation beyond the standard DocVQA benchmark SPDocVQA (Mathew et al., WACV'21)  by incorporating two additional datasets: InfographicVQA (Mathew et al., WACV'22) and PFL-DocVQA (Tito et al., ICDAR'24). These datasets pose greater challenges due to their complex and heterogeneous visual-textual structures, providing a more rigorous test of our model’s generalization capabilities.\nSome qualitative results on InfographicVQA are available here: https://anonymous.4open.science/r/RebuttalICML2-4D0C/README.md. The new results are demonstrating our model’s ability to handle these visually complex documents. As seen in the examples, our approach yields promising results, indicating strong potential for generalization.\nWe will provide the full results on these new datasets in the final manuscript.\n\n2. **The proposed approach was evaluated using only one model (Pix2Struct), so the claim of being \"model-agnostic\" is not supported**:  As we also commented in response to reviewer Y52T, our claim of model-agnostic nature relies on the independent, modular design of the explanation mechanism. Although currently integrated with Pix2Struct, its components, such as a learnable mask head, three loss terms of the explainability principles, and a postprocessing pipeline, are architecture independent. To empirically validate this, we are currently integrating our approach with Donut (Kim et al., ECCV'22), and results will be provided in the the final version of the paper\n\n3. **The performance drop (56%→51%) introduced by interpretability constraints is not sufficiently explained or analyzed.**: It can be attributed to the additional double-loop process employed by our approach. Even when the mask $M$ is set to all ones so that $X \\odot M = X$, the model still performs two passes through the encoder—one for mask generation and another for final prediction. Consequently, during backpropagation, gradients from both passes are accumulated. In theory, if both passes were identical, this would be equivalent to a single pass. However, in practice, non-linearities, normalization layers, and other factors introduce slight variations between the two passes. These differences can cause over-accumulation or interference in the gradients, perturbing the weight updates relative to the original training regime. Such deviations, along with potential numerical instabilities from the extra pass, likely contribute to the modest performance drop observed from 56% to 51%.\n\n4. **The choice of a fixed parameter (top-$k$=3 regions) in mask selection lacks experimental justification. Providing sensitivity analysis for different values of $k$ is recommended**: Following the reviewer’s suggestion, we conducted a sensitivity analysis on the hyperparameter $k$ that determines the number of regions selected during mask postprocessing. The results are summarized in the table below. As the table indicates, increasing $k$ leads to gradual improvements in both accuracy and ANLS, but it also results in a higher pixel ratio, which reflects a less minimal explanation. In terms of balancing utility and minimality, our experiments suggest that choosing $k=3$ or $k=4$ offers a good trade-off. That is, these values yield utility metrics comparable to higher $k$ settings while keeping the highlighted area compact. We will include the complete sensitivity analysis table in the Appendix of the revised manuscript.\n\n| k   | ACC  | ANLS | Pixel_Ratio |\n|----|------|------|-------------|\n| 1  | 0.36 | 0.52 | 0.21        |\n| 2  | 0.38 | 0.53 | 0.23        |\n| **3**  | **0.38** | **0.54** | **0.24**        |\n| 4  | 0.39 | 0.55 | 0.25        |\n| 5  | 0.39 | 0.55 | 0.26        |\n| 8  | 0.40 | 0.56 | 0.28        |\n| 10 | 0.41 | 0.57 | 0.29        |\n| 15 | 0.42 | 0.58 | 0.31        |\n| 20 | 0.43 | 0.58 | 0.32        |\n| 50 | 0.45 | 0.59 | 0.38        |\n| 100| 0.45 | 0.60 | 0.40        |", "claims_and_evidence": "Most claims in the submission are supported by clear and convincing evidence. Specifically, the authors convincingly demonstrate that their proposed DocVXQA framework improves interpretability for document question-answering through both quantitative (accuracy, ANLS metrics, pixel ratio) and qualitative (human evaluation) experiments.\n\nHowever, two key claims lack sufficient evidence:\n\n1. The paper claims that the proposed method is inherently model-agnostic, capable of integrating easily into various DocVQA architectures. However, the authors only demonstrate this with the Pix2Struct architecture, providing no experimental validation on other models. Without such validation, this claim remains theoretical and unsupported by evidence.\n\n2. The authors implicitly present DocVXQA as a general framework suitable for broader document visual question-answering scenarios. However, all experimental evaluations were performed on a single dataset (DocVQA). Without evaluation on multiple datasets or diverse document styles, the evidence provided does not sufficiently support the claim of broad generalizability.", "methods_and_evaluation_criteria": "The proposed methods and evaluation criteria are generally appropriate for the problem. The mask-generation strategy and the use of sufficiency, minimality, and context-aware loss terms effectively address the interpretability objectives in Document VQA. The evaluation metrics (accuracy, ANLS, pixel ratio, and human evaluation) are suitable for measuring both interpretability and prediction quality. \n\nHowever, the evaluation is limited by relying solely on the DocVQA dataset, raising questions about the method's generalizability to other document types or scenarios. Expanding evaluations to additional diverse datasets would strengthen confidence in the broader applicability of the approach.", "theoretical_claims": "The paper does not present complex theoretical proofs. The theoretical framing provided—such as the formulation of the explainability objectives based on the Information Bottleneck principle and mutual information—is straightforward, well-established, and reasonably applied. I reviewed these theoretical formulations and found them conceptually sound, without identifying any significant issues.", "experimental_designs_or_analyses": "The experiments presented are generally valid and sound. The selection of evaluation metrics (accuracy, ANLS, pixel ratio) and comparison baselines (raw attention, attention rollout, Grad-CAM, ColPali-based retrieval) are appropriate for the DocVQA task and clearly aligned with the authors' interpretability objectives.\n\nHowever, several issues are noted:\n\nAll experiments were conducted on only one dataset (DocVQA). The absence of experiments on additional datasets raises concerns about the generalizability of the results.\n\nThe authors observed a performance drop (56% → 51%) due to interpretability constraints but did not provide sufficient analyses or discussions exploring the underlying reasons or possible approaches to alleviate this trade-off. Addressing this point explicitly would further strengthen the experimental analysis.\n\nAdditionally, the authors use a fixed post-processing parameter (selecting the top-k=3 regions) for generating the explanation masks. However, the rationale behind choosing k=3 is not clearly explained or experimentally justified. Including sensitivity analyses or experiments to demonstrate how different values of k affect the interpretability and accuracy would strengthen the validity of this choice.", "relation_to_broader_scientific_literature": "The paper relates clearly to existing literature on document VQA and explainable AI, specifically extending OCR-free approaches (e.g., Pix2Struct) and self-explainable methods based on information theory. Unlike post-hoc techniques (e.g., Grad-CAM), this work integrates interpretability directly into model training, contributing a novel, practical approach to improve transparency in document understanding.", "essential_references_not_discussed": "All essential related works appear to be appropriately cited and discussed in the paper.", "other_strengths_and_weaknesses": "The main strengths and weaknesses have been discussed in earlier sections. In summary, the paper's key strengths include its clear originality in integrating interpretability into document VQA and its strong motivation addressing real-world transparency needs.\n\nHowever, there are four main weaknesses that should be addressed:\n\nExperiments were conducted on only one dataset (DocVQA), limiting evidence of generalizability to other datasets or document scenarios.\n\nThe proposed method was evaluated using only one model architecture (Pix2Struct), which leaves the \"model-agnostic\" claim unsupported by empirical evidence.\n\nThe observed accuracy drop (from 56% to 51%) resulting from enhanced interpretability lacks thorough analysis or discussion.\n\nThe choice of a fixed parameter (Top-k=3 regions) in post-processing lacks experimental justification or sensitivity analysis to confirm its optimality.\n\nAddressing these issues would strengthen the overall contribution and reliability of the findings.", "other_comments_or_suggestions": "Fix minor typos, such as:\n\nLine 073 :\"preserve aspect the ratios\" (should be \"preserve the aspect ratios\")\nLine 099 : \"models are are reaching good performances\" (duplicate \"are\")\nLine 103 : \"text is extrtacted with OCR\" (should be \"extracted\")", "questions_for_authors": "1. The experiments are performed only on the DocVQA dataset, so I cannot confirm the generalizability of your approach. Additional experiments on diverse datasets are needed.\n\n2. The proposed approach was evaluated using only one model (Pix2Struct), so the claim of being \"model-agnostic\" is not supported. Additional experiments with other architectures would help validate this claim.\n\n3. The performance drop (56%→51%) introduced by interpretability constraints is not sufficiently explained or analyzed. Additional analysis or experiments addressing this issue would strengthen the paper.\n\n4. The choice of a fixed parameter (top-k=3 regions) in mask selection lacks experimental justification. Providing sensitivity analysis for different values of k is recommended.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {"ethics_expertise_needed": ["Discrimination / Bias / Fairness Concerns"]}}, {"review_id": "qlZlnYjfnf", "reviewer": "Reviewer_FXJM", "summary": "- This paper presents DocVXQA, a novel self-explainable framework for document question answering that learns to provide context-aware visual explanations. It builds on the Pix2Struct model and incorporates a learnable mask to enhance transparency. The approach is based on the information bottleneck principle and integrates additional pretrained models for better generalization. Experimental results show the effectiveness of the method in terms of both accuracy and interpretability.\n - The research question is important as it addresses the lack of transparency in current DocVQA models, which is crucial in high-stakes domains.\n- The key methodological innovation is the use of a learnable mask to generate context-aware explanations, which is formulated based on the information bottleneck principle.\n- The main findings are that the proposed method outperforms baseline techniques in terms of both accuracy and interpretability, and that the generated explanations are more context-aware and human-understandable.\n\n## update after rebuttal\n\nDear authors,\n\nThese questions are written by real human, not generated by LLM. If you feel they are LLM-generated and choose not to answer them, then I have to respect your action. But I will not change my comments, or my evaluation.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "(1) The proposed DocVXQA framework is a novel approach to visually self-explainable document question answering.\n(2) The use of the information bottleneck principle to formulate explainability principles is a theoretically sound approach.\n(3) The integration of additional pretrained models to enhance the generalizability of the explanations is a practical and effective strategy.\n(4) The extensive experimental evaluations, including human evaluation, provide strong evidence of the effectiveness of the method.", "methods_and_evaluation_criteria": "Plz read Claims And Evidence.", "theoretical_claims": "Plz read Claims And Evidence.", "experimental_designs_or_analyses": "Plz read Claims And Evidence.", "relation_to_broader_scientific_literature": "N/A", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "N/A", "other_comments_or_suggestions": "N/A", "questions_for_authors": "(1) The paper could discuss the limitations of the proposed method and potential future research directions in more detail. How does the proposed method handle complex or ambiguous questions? What are the potential limitations of the proposed method in handling different types of documents?\n\n(2) The paper could provide more examples of the generated explanations to better illustrate the effectiveness of the method.\n\n (3) The paper could discuss the potential applications of the proposed method in different domains. What are the potential applications of the proposed method in different domains?  How does the proposed method handle different types of data and tasks?\n\n(4) The paper could provide more details on the scalability of the proposed method. How does the proposed method handle noisy or incomplete data? What are the potential challenges in deploying the proposed method in practice?\n    \n (5) The paper could provide more details on the relationship between the proposed method and other self-explainable models (e.g., Dib-x). How does the proposed method compare to other state-of-the-art methods in terms of performance and interpretability?\n\nUpdated:\n\nDear authors,\n\nThese questions are written by real human, not generated by LLM. If you feel they are LLM-generated and choose not to answer them, then I have to respect your action. But I will not change my comments, or my evaluation.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "l5SNXY9y8X", "reviewer": "Reviewer_Y52T", "summary": "The paper introduces DocVXQA, a method that generates visual explanations (in the form of a mask) that highlight parts of documents that are relevant for OCR-free document question answering. DocVXQA builds on the Pix2Struct model and learns a mask that, when combined to the input image, must lead the document QA model to give the same answer, while optimizing for mask minimality. A prior is also given by ColPali (that provides a relevance score for text-image matchings) to improve the mask. Experiments are conducted on the DocVQA dataset using a fine-tuned Pix2Struct model. The method is evaluated with accuracy and ANLS (to measure the quality of answers), and pixel ratio (to measure mask minimality). A user-study is also conducted.\n\n## update after rebuttal\n\nThe rebuttal clarifies several things and answers my questions. I am now more inclined to recommend acceptance of the paper.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "The authors sincerely appreciate Reviewer Y25T’s constructive feedback. We are pleased that the reviewer recognized the importance of explainability in Document VQA and the strong motivation behind our loss components. We also appreciate the positive remarks on the clarity and coherence of our writing and the value of our user study.\n\nWe have carefully addressed all the points raised in the following, and we are grateful for the detailed identification of typos and missing content, which will be corrected in the revised manuscript\n\n1. **\"Self-explanatory\" clarification**: We clarify that our intention is not to suggest that we reveal the DocVQA model internal workings. Rather, we use \"self-explanatory\" following established usage in explainable AI literature (Mersha et al., Neurocomputing'24), (Gautam, Boubekki et al., NeurIPS'22), describing models inherently designed to produce interpretable outputs during inference (L158–162, left). This contrasts with post-hoc methods, which retrospectively assess relevance between inputs and outputs on frozen models. We fully accept the reviewer’s recommendation and will clearly specify that self-explanatory refers exclusively to \"the model’s capacity to produce interpretable outputs by design,\" not exposing internal workings.\n\n2. **Model-agnostic claim is not supported**: As we also commented in response to reviewer ovki, our claim of model-agnostic nature relies on the independent, modular design of the explanation mechanism. Although currently integrated with Pix2Struct, its components, such as a learnable mask head, three loss terms of the explainability principles, and a postprocessing pipeline, are architecture independent. To empirically validate this, we are currently integrating our approach with Donut (Kim et al., ECCV'22), and results will be provided in the final manuscript.\n\n3. **The encoder was pretrained only on original documents, the reviewer suspects that a domain gap occurs when it is applied on the masked ones. Using S+M, the answer is generally visible, it should be easy to produce the correct one, even without ColPali prior**: The reviewer points out low utility scores in the sufficiency and minimality ablation (S+M) despite visible answers in masked documents. We respectfully disagree. Although answers are sometimes visible, this is not consistently true. The domain gap noted by the reviewer is also reduced with finetuning on original and masked inputs. We identify aggressive masking as the primary factor lowering utility scores, enforcing minimality at the cost of sufficiency. This lack of context impairs predictions, highlighting the importance of integrating the ColPali prior through token interactions (Table 2).\n\n4. **The paper does not explain why the decoder is kept frozen**: We freeze the decoder to preserve its pretrained text generation capabilities and maintain stable training outputs. Since the decoder is pretrained on original, unmasked documents, jointly finetuning it with the encoder and mask head introduces distributional shifts, causing instability. Instead, we focus learning efforts on the encoder and the new mask head. Noting that we did tried unfreezing the decoder, but it led to unstable optimization, slower convergence, and reduced overall performance. We will add these details to the final manuscript.\n\n5. **Using ColPali to have more context-aware explanations will increase pixel ratio without increasing utility.**: Table 2 shows that using only sufficiency and minimality losses (S+M) yields accuracy of 0.19 and ANLS of 0.36. Integrating ColPali through token interactions (S+M+TI) significantly improves these metrics to accuracy 0.38 and ANLS 0.54. Although ColPali increases the pixel ratio, this reflects the preservation of essential contextual cues rather than irrelevant areas. Practically, this additional context is critical for robust predictions, accuracy and mask learning guidance. Qualitative human evaluations further confirm that ColPali-enhanced explanations are clearer and more context-aware, supporting a balanced trade-off between minimality and sufficiency.\n\n6. **The proposed model yields results that are very close to the ColPali + Pix2Struct baseline**: Figure 3 illustrates that relying solely on quantitative metrics is insufficient for comprehensive evaluation. Although quantitatively our results appear similar to ColPali+Pix2Struct, qualitatively ours perform better. Following the reviewer’s suggestion, we conducted a human preference study comparing masks from both methods. In total, 12 participants evaluated 21 randomly selected question-answer pairs (252 trials). Our method was preferred in **163 trials (64.7%; 95% CI [58.4%, 70.6%], p << 0.001)**, with all participants (12/12) favoring our approach overall. Thus, our method is close to ColPali+Pix2Struct quantitatively, but it is offering a significantly more compact and interpretable explanations. This new study will be added to the revised manuscript.", "claims_and_evidence": "The explanations provided by the model are useful for users to verify the correctness of an answer by comparing it to the explanations. However, they do not genuinely explain the inner workings of the original model, Pix2Struct. Instead, they may give persuasive narratives that do not necessarily reflect the model's decision-making process, potentially leading to misleading interpretations. In the reviewer’s opinion, the final model cannot thus be considered fully \"self-explanatory\". The reviewer recommends adjusting the wording throughout the paper to prevent any misinterpretation regarding the model's explanatory capabilities. This concern is further supported by the design of the user study, which focuses on users' subjective satisfaction with the explanations rather than evaluating whether the explanations genuinely reveal the model's reasoning process, or help users to better understand the model (see “What I Cannot Predict, I Do Not Understand: A Human-Centered Evaluation Framework for Explainability Methods”, Colin et al, NeurIPS 2022)\nThe statement at the end of the introduction (\"our method is inherently model-agnostic, allowing seamless integration with various DocVQA architectures\") is not adequately supported by the experiments. Providing empirical evidence would make this claim more convincing.\n\nThe encoder of Pix2Struct is applied on original documents, and masked ones. Since it was pretrained only on original documents, the reviewer suspects that a domain gap occurs when it is applied on the masked ones, hence the low utility scores, when only sufficiency and minimality losses are included. Still, as illustrated, in this ablation, the answer is generally visible, it should be easy to produce the correct one, even without all the context that is integrated back thanks to the ColPali prior.\n\nThe paper does not explain why the decoder is kept frozen during training, while other parts (e.g., the encoder) are learned. What is the motivation here? Were other choices explored?", "methods_and_evaluation_criteria": "The proposed metrics of Table 2 seem not enough to completely evaluate the method. For instance, using ColPali to have more context-aware explanations will increase pixel ratio without increasing utility. From this set of metrics only, it looks that the integration of ColPali is detrimental, while it looks reasonable and is well motivated.", "theoretical_claims": "There are no theoretical claims", "experimental_designs_or_analyses": "Yes.", "relation_to_broader_scientific_literature": "NA", "essential_references_not_discussed": "The related work section is rather well written. Except that the organization between Sections 2.2 and 2.3 is unclear, with some overlap that appears arbitrary. A clearer distinction or restructuring would improve the flow of ideas.", "other_strengths_and_weaknesses": "Strengths:\n* The paper goal of explaining document VQA model is sound and important for the community.\n* The proposed model is reasonable, the loss components are well motivated.\n* The paper is rather well written and motivated\n* The user-study was appreciated.\n\nWeaknesses:\n* The proposed model yields results that are very close to the ColPali + Pix2Struct baseline. The empirical improvement is marginal, while the method is much more complex (and has to be trained).", "other_comments_or_suggestions": "* The mandatory “Impact Statement” section is missing\n* The itemizations L.294-309 and L.437-400 do not follow the template format, as the bullets are positioned in the margin.\n\nTypos:\n* “are are” L.99\n* “aspect the ratios” L.73\n* “i. e.,” L138\n* “Right” L.877\n* L.284, a space is missing before the citation\n* The use of bold in Table 1 is inconsistent\n* “Section 4” L.314 (x2) - “Section 4.3” L.377 - “Section 4.3” L.409 → Please revise latex labels / references.\n* In Table 2, mismatch between “I” and “TI”\n* “were each shown”, L.428", "questions_for_authors": "NA", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The LLM-generated review by FXJM was ignored by the authors and the AC.

The other three reviewers were more positive than negative. Reviewer ovki provided a substantive review. He asked similar questions, wanting experiments on more backbones and datasets. The authors provided these in the rebuttal, and he was satisfied on all counts.

Y52T was initially leaning towards rejection, but raised his score to weak accepted after the rebuttal. He had concerns that the explanation model was not truly self-explanatory, or faithful to the inference model. The rebuttal addressed this well by clarifying terminology. He had a question about the performance drop on masked images, as did some of the others.

9NT1 provided a very thorough review, and maintained his rating of weak reject. The authors provided two rounds of responses with explanations and experimental results, which addressed most of his concerns. His primary issue is whether the method generalizes to other vision-language tasks, which is not a concern raised by the others. The AC does not agree that the method needs to work on other tasks in order to be published, and recommends acceptance.
Decision: Accept (poster)