Medical Visual Textual Entailment for Numerical Understanding of Vision-and-Language Models
Year: 2023

Abstract:
Assessing the capacity of numerical understanding of vision-and-language models over images and texts is crucial for real vision-andlanguage applications, such as systems for automated medical image analysis. We provide a visual reasoning dataset focusing on numerical understanding in the medical domain. The experiments using our dataset show that current vision-and-language models fail to perform numerical inference in the medical domain. However, the data augmentation with only a small amount of our dataset improves the model performance, while maintaining the performance in the general domain.

heading: 1 Introduction
text: Vision-and-language models have made great progress on complex tasks, going beyond image recognition and towards reasoning over images and texts (Antol et al., 2015; Xie et al., 2019; Suhr et al., 2019). Following the success of pre-trained language models (Devlin et al., 2019, inter alia), recent advances in vision-and-language models have been made by the introduction of large-scale pretraining (Li et al., 2019; Kim et al., 2021; Singh et al., 2022). However, as with pre-trained language models, it is unclear what information pretrained vision-and-language models learn and use in their predictions, and what their limitations are.
While a large body of research (Naik et al., 2018; Rozen et al., 2019; Ravichander et al., 2019; Richardson et al., 2020) has provided challenging reasoning tasks to probe the reasoning ability of pre-trained language models, such work has been more limited for vision-and-language models. Furthermore, previous visual reasoning datasets are usually provided by the general domain of images, and analysis across different domains is desirable.
∗Equal Contribution. 1https://radiopaedia.org/cases/haemorrhagic-intracranial-
metastases-from-breast-cancer
Our focus is to investigate whether current vision-and-language models have the ability to infer numerical relationships between images and texts in the medical domain, which is crucial for real vision-and-language applications such as systems for automated medical image analysis. Consider the example of images and textual descriptions in a medical article presented in Figure 1. The lesion number affects the treatment strategy for diseases such as brain metastasis. If systems can automatically judge whether the lesion number in given images matches that in arbitrary query texts, they can support medical decision-making. Recently, a vision-and-language model focusing on the medical domain (Delbrouck et al., 2022) has begun to be provided but is not yet fully developed.
With this motivation, we provide a visual reasoning dataset focusing on numerical inference in the medical domain by adding annotations to the previous medical image and caption dataset MedICaT (Subramanian et al., 2020). We call our dataset MedVTE, which will be publicly available at https://github.com/ ynklab/MedVTE. Using MedVTE, we investigate the extent to which current pre-trained visionand-language models have the ability of numerical
8
understanding on visual reasoning tasks across images and texts in the medical domain. The experiments show that current models have much room to perform numerical inference in the medical domain.

heading: 2 Background

heading: 2.1 Vision-and-language understanding
text: Regarding standard vision-and-language understanding tasks, SNLI-VE (Xie et al., 2019) is a large general domain dataset for the Visual Textual Entailment (VTE) task. The dataset consists of image-sentence pairs annotated with a three-class label (entailment, contradiction, or neutral), indicating whether a premise image entails a hypothesis sentence. There have been studies investigating the counting ability of vision-and-language models on visual question-answering tasks and object detection tasks (Chattopadhyay et al., 2017; Zhang et al., 2018; Song and Qiu, 2018; Trott et al., 2018; Acharya et al., 2019; Parcalabescu et al., 2021). However, since previous studies only use datasets in the general domain, it is unclear the extent to which models can maintain the ability to understand numerical expressions in the medical domain.
For visual reasoning in the medical domain, Li et al. (2020) have compared the performance of four pre-trained vision-and-language models and traditional CNN-RNN models on two datasets of thoracic findings classification tasks: MIMICCXR (Johnson et al., 2019) and OpenI datasets. The results showed that the pre-trained models outperformed the traditional models. Our VTE dataset gives a fine-grained analysis of the capacity of the pre-trained vision-and-language models for numerical understanding in the medical domain.

heading: 2.2 Clinical NLP
text: Clinical NLP is one of the practical fields of NLP, and various reasoning tasks in the medical domain have been provided. For sentence-level language understanding tasks, emrQA (Pampari et al., 2018) is a large-scale QA dataset on electronic medical records, and MedSTS (Wang et al., 2020) is a resource for Semantic Textual Similarity (STS) tasks in the medical domain. The most related dataset to ours is MedNLI (Romanov and Shivade, 2018), a physician-annotated Natural Language Inference (NLI) dataset with premises extracted from clinical notes. However, a recent study has reported annotation artifacts in MedNLI (Herlihy and Rudinger,
2021). To avoid such undesired artifacts, we cover a variety of numerical expressions.

heading: 3 MedVTE Datasets
text: We introduce MedVTE, visual reasoning datasets in the medical domain involving numerical expressions. MedVTE is composed of pairs of medical images, captions, and three-class entailment labels (entailment, contradiction, or neutral). MedVTE focuses on the relationship between the number of lesions, such as cancer in an image and the numerical expression in a text.
We created MedVTE by selecting examples involving numerical expressions from MedICaT dataset (Subramanian et al., 2020). MedICaT contains 217,060 figure-caption pairs in medical articles, whose captions sometimes refer to the number of the depicted lesions (e.g., tumors or nodules). The selection is conducted by one medical expert.

heading: 3.1 Premise–hypothesis collection
text: In MedVTE, a premise is a MedICaT figure, and a hypothesis is one complete sentence containing one or more lesion numbers. We created 409 examples for the MedVTE dataset in total.
Step 1. Cleaning We removed 58 MedICaT figure–caption duplicate pairs. We also mitigated occasional errors in MedICaT captions, such as missing letters or interrupted sentences. Some MedICaT captions are provided in two versions, the one by the MedICaT authors and the other from the S2ORC dataset (Lo et al., 2020). In such cases, we always chose the longer one to avoid including incomplete sentences.
Step 2. Figure collection We collected MedICaT figure–caption pairs whose captions include lesion numbers in a rule-based approach. We assigned Penn Treebank part-of-speech (POS) tags (Marcus et al., 1993) to all captions. We then applied a spaCy rule-based matcher to accept only captions having a numeral followed by a noun suggesting lesions. This step left us 431 figure–caption pairs. See Appendix A for details.
Step 3. Hypothesis collection Every MedVTE hypothesis is a single sentence including one or more lesion numbers. We collected hypotheses by splitting captions into sentences and selecting sentences containing at least one lesion number.
Sentence selection was performed in a rulebased approach as in Step 2 followed by manual
reviews. In manual reviews, we removed erroneous lesion numbers where integers do not actually count lesions, such as cell line names Walker 256 tumor. We also excluded invalid premise figure-hypothesis sentence pairs meeting the below criteria:
• the figure file contains multiple article figures
• the hypothesis is not a single sentence
• the hypothesis does not make sense due to ungrammaticality.
When multiple hypothesis sentences corresponded to a single premise figure, we treated each premise figure-hypothesis sentence pair as an independent sample. We obtained 409 premisehypothesis pairs for 373 premise figures, where 430 lesion numbers appear in total.

heading: 3.2 Labeling
text: We assigned two types of entailment labels, strict labels and loose labels, to premise-hypothesis pairs on MedVTE.
Strict labels follow the common practice of annotating visual reasoning datasets to compare all the
information, not only numerical one but also medical background knowledge, of a premise figure and a hypothesis sentence. However, we found that the considerable number of strict labels became neutral under given images because out-of-figure information in hypothesis sentences (i.e., information that is not acquired from images), such as “this image was obtained six months after surgery,” is necessary to judge their labels as entailment.
To realize separate assessments of the numerical reasoning abilities of models under only given
images, we add loose label annotations rather than editing hypothesis sentences. Loose labels only compare numerical information of a premise figure and a hypothesis sentence. This approach provides an option to focus on numerical reasoning abilities with loose labels, or to fully measure medical reasoning abilities with strict labels, which requires expert knowledge to recognize out-of-figure information.
The following is the definition of loose labels. Details are available in Appendix C.
• entailment: All lesion numbers are consistent with the premise figure
• contradiction: One or more lesion numbers are smaller than those depicted in the premise figure
• neutral: Either of the following is satisfied: (i) one or more lesion numbers are larger than those depicted in the premise figure although the others are consistent, (ii) the number of lesion numbers cannot be determined only from the premise figure, or (iii) no clauses remain after removing out-of-figure information from the hypothesis.
Figure 2 shows MedVTE examples. In the top and middle examples, their loose labels are the same as their strict labels. In the bottom example, its loose label is different from its strict label with the consideration of out-of-figure information. The distribution of loose labels in MedVTE is (entailment, neutral, contradiction) = (310, 95, 4), and that of strict labels is (entailment, neutral, contradiction) = (208, 197, 4).

heading: 4 Experiments and Analysis

heading: 4.1 Experimental setup
text: Models Vision-and-language models are categorized into three broad types based on their encoding style, fusion encoder, dual encoder, and a combination of both. We used two vision-andlanguage models for our experiments: a Vision-andLanguage Transformer model (ViLT) (Kim et al., 2021) and a Foundational Language And Vision Alignment model (FLAVA) (Singh et al., 2022). ViLT is a fusion-encoder style model which has 112M parameters. FLAVA is a fusion-encoder plus
dual-encoder style model which has 243M parameters. See details of pre-training datasets for each model in Appendix D.
Training For baseline models, we use visionand-language models fine-tuned with the training set of SNLI-VE. We split the MedVTE dataset as train:test=306:103 and evaluate the performance of the models on the MedVTE test set. To investigate whether a small portion of additional training data in the medical domain contributes to knowledge transfer for visual reasoning, we evaluate models fine-tuned with the SNLI-VE training set mixed with the MedVTE training set. We fine-tune the models for three epochs for each dataset and use F1-macro scores for evaluation metrics. Details on the hyperparameters can be found in Appendix D.

heading: 4.2 Baseline results
text: Table 1 shows baseline results. While both ViLT and FLAVA models trained with SNLI-VE achieved around 75% on in-domain SNLI-VE, their performance was very low on MedVTE.
When we evaluated models trained with SNLIVE mixed with a subset of MedVTE, the performance on MedVTE was improved while maintaining the performance on SNLI-VE. However, the overall performance on MedVTE was still lower than 50%. This indicates that numerical inference in the medical domain is challenging for visionand-language models even when they train with a subset of MedVTE. Regarding the difference between loose labels and strict labels with a subset of MedVTE, the performance improvement on MedVTE strict labels was lower than that on loose labels. This suggests that the ability to use out-offigure information is difficult to obtain from the data augmentation.

heading: 5 Conclusion
text: We created the visual reasoning dataset MedVTE, focusing on numerical understanding in the medical domain. The experiments using MedVTE showed that current vision-and-language models struggled with performing numerical inference in the medical domain. However, the data augmentation with only a small amount of our MedVTE dataset improved the model performance, while maintaining the performance in the general domain. In future work, we increase the size of our MedVTE dataset and make further analysis of visionand-language models to investigate the extent to
which the size of a fine-tuning dataset affects the performance of numerical inference in the medical domain. Improving automated numerical visionand-language understanding in the medical domain could aid therapeutic decision-making that depends on lesion numbers.

heading: 6 Limitation
text: Since hypothesis sentences were created and labeled by medical experts, the size of our current dataset is small. In particular, the number of examples of contradiction is small because the hypothesis sentences were created based on captions to efficiently construct our dataset. However, we can increase the number of examples of contradiction by rewriting phrases in the hypothesis sentences. The claim of this study is that we can relatively efficiently create a VTE dataset in the medical domain from the existing image caption dataset, and can empirically demonstrate the challenges of current vision-and-language models on the VTE dataset. Although increasing the data size is an important next step, it is beyond the scope of this paper.

heading: Acknowledgements
text: We thank the three anonymous reviewers for their helpful comments and suggestions, which improved this paper. This work was supported by JST, PRESTO grant number JPMJPR21C8, Japan.

heading: A Sample selection rules
text: This section explains detailed MedVTE sample selection rules.
We employed rule-based approaches to select figure-caption pairs from the MedICaT dataset so that all sampled captions refer to the number of lesions.
We selected sentences in the MedICaT captions containing LESION-NUMBER-EXPRESSIONs. We defined a LESION-NUMBER-EXPRESSION as any token subsequence of a single sentence of a caption that satisfies all of the following Rules 1 to 3:
• Definition 1. COMPARATIVE is a string whose lowercase form is either at least, at most, more than, or less than.
• Definition 2. NUMBER is a single token whose Penn Treebank part-of-speech (POS) tag (Marcus et al., 1993) is CD (cardinal number).
• Definition 3. LESION-NOUN is a single token whose POS tag is either NN (noun, singular or mass) or NNS (noun, plural).
• Rule 1. A LESION-NUMBER-EXPRESSION must be a concatenation of COMPARATIVE, NUMBER, and LESION-NOUN in this order, or a concatenation of NUMBER and LESIONNOUN in this order.
• Rule 2. The lemma of LESION-NOUN must be either cancer, lesion, mass, metastasis, nodule, or tumor.
• Rule 3. A LESION-NUMBER-EXPRESSION must not appear immediately after a token whose lowercase form is either figure, figures, fig, figs, patient, case, day, sample, type, category, group, grade, level, stage, rads, pirads, birads, cin, score, likert, c, t, l, s, segment, gs, suv, +, +1, +2, +3, +4, +5, mm, cm, mm2, cm2, mm3, or cm3.
In our implementation, we first assigned POS tags to all MedICaT captions using Berkeley Neural Parser (Stern et al., 2017; Kitaev and Klein, 2018; Kitaev et al., 2019). We then built a spaCy rule-based matcher and applied it to all parsing results.

heading: B Dataset statistics
text: Of the 409 MedVTE premise-hypothesis pairs, 300 (73.3%) have radiological premise figures, twelve (2.9%) have scopic premise figures, and the remaining have other various types of premise figures including histopathological images.
MedVTE contains 430 lesion numbers in total because three of the 409 hypotheses (0.7%) contain three lesion numbers, fifteen hypotheses (3.7%) contain two lesion numbers, while the remaining 391 hypotheses (95.6%) contain one lesion number.
Six of the 430 lesion numbers (1.4%) include comparative expressions, four of which are associated with “at least” and the others are accompanied by “more than.” Figure 3 shows the distribution of the remaining 424 lesion numbers. The most frequent lesion number is two, occurring 223 times in the dataset (52.6%). 398 lesion numbers (92.6%) are between one and five, and fourteen lesion numbers (3.3%) are greater than ten.

heading: C Details of labeling
text: C.1 Loose labels
Each MedVTE premise image consists of one or more subfigures that are often excerpts of a vast series of radiological, pathological, or endoscopic images. Therefore, it must be considered that the premise image may not reflect the entire patient and may contain only a subset of the lesions that are actually present, or conversely, the same lesion
may repeatedly appear across multiple subfigures as in Figure 4. This phenomenon is prevalent not only in the medical articles from which MedVTE originates but also in the real-world clinical practice that we target for application.
We regard each hypothesis as a set of propositions. For each proposition addressing the lesion number in the hypothesis sentence, the following procedure was employed to determine the veracity or falsity.
(a) If the medical expert determines that the quantities are equal in the premise figure and the hypothesis sentence, the proposition is supported.
(b) When the lesion number in the hypothesis sentence apparently exceeds that in the premise figure, the medical expert is requested to carefully review the premise figure and determine if the gap can be explained by the following reason:
• The original caption is correct, but the medical expert initially missed some lesions due to subtle image findings.
If so, the hypothesis is supported. Otherwise, the loose label is neutral because it is impossible to judge which of the following is happening:
• The original caption is correct, but the premise figure does not show all the lesions
• The original caption has overcounted the lesions.
(c) When the lesion number in the hypothesis sentence appears to be smaller than the premise figure, the medical expert is asked to examine the premise figure again and determine which of the following is the most convincing:
• The original caption is correct, but the medical expert initially overcounted the lesions due to equivocal image findings
• The original caption is correct, but the medical expert initially overcounted the lesions due to the same lesion repeatedly appearing across multiple subfigures
• The original caption has undercounted the lesions.
In the first or second case, the hypothesis is supported. In the last case, the loose label is contradiction.
C.2 Strict labels When a hypothesis contains propositions that cannot be judged true or false from the premise im-
age alone, we consider it out-of-figure information. The following are examples of propositions that we regard as out-of-figure information:
• Mention to other figures than the premise figure (e.g., “show no enhancement on arterial phase images of MRI and on the arterial phase of CT scan (not shown)”)
• Numerical values for elapsed time, such as days, months, or years (e.g., “Axial contrastenhanced CT six weeks pre-RF ablation (a) demonstrates two lesions”)
• Specific lesion size numbers (e.g., “The two nodules were 1.2 cm in diameter”).
If the hypothesis sentence includes out-of-figure information, we set the strict label to neutral regardless of the loose label. Otherwise, the strict label is the same as the loose label.

heading: D Model details
text: ViLT is pre-trained on MSCOCO (Lin et al., 2014)+VG (Krishna et al., 2017)+CC (Sharma et al., 2018)+SBU (Ordonez et al., 2011). FLAVA is pre-trained on filtered YFCC100M (Thomee et al., 2015)+CC12M (Changpinyo et al., 2021)+WIT (Srinivasan et al., 2021)+RedCaps (Desai et al., 2021)+LN (Pont-Tuset et al., 2020)+MSCOCO+VG+CC+SBU.
We basically adopted models and parameters implemented in transformers2. We attached a 2-layer classifier head ourselves for FLAVA since there was no model implementation for classification tasks in the library. Table 2 and Table 3 show hyperparameters in ViLT and FLAVA models, respectively.
2https://huggingface.co/docs/transformers/v4.20.1/en/index

heading: Future Work
text: In future work, we increase the size of our MedVTE dataset and make further analysis of visionand-language models to investigate the extent to
which the size of a fine-tuning dataset affects the performance of numerical inference in the medical domain. Improving automated numerical visionand-language understanding in the medical domain could aid therapeutic decision-making that depends on lesion numbers