SciMMIR: Benchmarking Scientific Multi-modal Information Retrieval (2024)

Abstract: Multi-modal information retrieval (MMIR) is a rapidly evolving field where significant progress has been made through advanced representation learning and cross-modality alignment research, particularly in image-text pairs. However, current benchmarks for evaluating MMIR performance on image-text pairs overlook the scientific domain, which has characteristics that are distinct from generic data, as the captions of scientific charts and tables usually describe experimental results or scientific principles, rather than human activity or scenery. To bridge this gap, we develop a scientific domain-specific MMIR benchmark (SciMMIR) by leveraging corpora of openaccess research papers to extract data relevant to the scientific domain. This benchmark comprises 530K meticulously curated image-text pairs extracted from figures and tables with detailed captions from scientific documents. We further annotate the image-text pairs with a twolevel subset-subcategory hierarchy to facilitate a more comprehensive evaluation of baseline retrieval systems. We conduct zero-shot and finetuned evaluations on prominent multi-modal image-captioning and visual language models, such as CLIP, BLIP, and BLIP-2. Additionally, we perform optical character recognition (OCR) on the images and exploit this text to improve the capability of VLMs on the SciMMIR task. Our findings offer useful insights for MMIR in the scientific domain, including the influence of pre-training and fine-tuning settings, the effects of different visual and textual encoders, and the impact of OCR information. All our data and code are made publicly available.1

heading: 1 Introduction
text: Information retrieval (IR) systems are expected to provide a relevant piece of information from a vast,
*Corresponding authors. 1https://github.com/Wusiwei0410/SciMMIR
yet organised, collection of data, according to given user queries. With the advancement of representation learning (Bengio et al., 2013), the methodological paradigm of IR systems has evolved from using lexical matching to retrieve textual data (Luhn, 1957; Jones et al., 2000; Robertson et al., 2009) to a mixture of similarity matching approaches in a learned representation space, consequently supporting additional modalities such as images and audio, alongside text (Karpukhin et al., 2020; Chen et al., 2020b; Koepke et al., 2022).
In scientific domains, offering users a finegrained multi-modal retrieval service presents considerable practical significance. Although previous studies have evaluated the image-text retrieval task across a range of general topics on large-scale datasets such as Wikipedia (Young et al., 2014; Lin et al., 2014; Srinivasan et al., 2021; Goldsack et al., 2023; Luo et al., 2023a), there is a notable research gap in comprehensively assessing MMIR models within scientific domains, specifically. Integrating both in-domain and out-of-domain data in the pre-training phase significantly boosts the performance of visual language models (VLMs) on downstream tasks. However, the training of most VLMs has focused exclusively on common generic topics concerning the mundane events of daily life (Luo et al., 2023b), such as images depicting scenery and human activities. As a result, this pre-training overlooks data pertinent to scientific domains such as elements related to model architectures, illustrations of scientific principles, and the results of experiments.
Due to the substantial differences in the data distribution characteristics between generic data and scientific data, many VLMs may not have an adequate ability to perform MMIR for scientific domains. Additionally, existing table-related works, such as table generation tasks, have mainly focused on textual representations of tables, while overlooking image-based representations of tabular data.
12560
This presents problems for human-computer interaction, as users may desire to input information in the form of screenshots and expect an interactive system to present results in a graphical format.
To address the aforementioned research gap, we introduce SciMMIR, a Scientific Multi-Modal Information Retrieval benchmark. SciMMIR (outlined in Figure 1) is the first benchmark to comprehensively evaluate a model’s MMIR ability in the scientific domain. To build our data collection, we retrieve images of figures and tables, and their associated captions, from scholarly documents available on arXiv, an open-access archival corpus, to construct image-text pairs. In order to comprehensively evaluate the cross-modality aligned representations learned by models, our SciMMIR benchmark defines the retrieval task as bi-directional, involving searching for the correct textual caption in a candidate pool from a given image (img→txt) and finding the corresponding figure or table image from a textual caption (txt→img).
Given the disparity among various data types, we contend that achieving uniform model performance across diverse data formats is challenging. For example, a model may excel at retrieving data related to experimental results but demonstrate average performance regarding data related to model architectures. If an overall improvement is sought for the performance of VLMs, this improvement may not be observed in specific sub-domains of information. Consequently, such improvements do not necessarily translate into observable boosts to a VLM’s performance for a specific use case. As a result, we annotate and categorise the imagetext pairs into three figure-caption and two tablecaption subcategories based on the type of content they describe (such as experimental results, model architectures, and scientific principles). We then conduct fine-grained evaluation on each subset. By analysing performance across subcategories, we are better able to carry out targeted improvements to a model for a specific subcategory of interest.
To explore the MMIR capabilities of existing image captioning models and VLMs in scientific domains, as well as different subcategories, we conduct extensive experiments in both zero-shot and fine-tuned settings across various subcategories. Furthermore, we extract OCR-text data from the images and investigate its influence on the performance of VLMs. We present our key insights as follows:
1. We reveal that MMIR tasks in the scientific domain pose significant challenges for current VLMs, which usually do not demonstrate adequate performance in scientific domains. Furthermore, after fine-tuning VLMs with data specific to scientific domains, there is a marked performance improvement, underlining the effectiveness of domain-specific adaptation.
2. The results suggest a distinction between tasks involving the figure and table subsets, with performance on the figure subset being more easily improved through domainspecific adaptation. Furthermore, by leveraging text data extracted through OCR, we are able to substantially boost the performance of VLMs in MMIR tasks within scientific domains. This suggests that character recognition is a key weakness of standalone VLMs in the performance of SciMMIR.
3. Regardless of parameter size, the BLIP-2 series of models generally perform better on SciMMIR than other pre-trained VLMs. This improved zero-shot capability may be the result of distinct pre-training tasks including image-text matching and image-text contrastive learning, rather than standard language modelling.
These findings underscore the importance of tailored approaches for different data types within the scientific MMIR framework. A more in-depth exploration of these findings is given in §5.

heading: 2 Related Work
text: General Information Retrieval. Information Retrieval is a fundamental task within NLP and has recently been facilitated by dense representation learning (Reimers and Gurevych, 2019; Karpukhin et al., 2020). More recently, the desire for unified representations across tasks has become significant, with this line of research proposing to understand and evaluate task-agnostic representations in a single representation space (Muennighoff et al., 2023; Asai et al., 2022; Su et al., 2022; Wei et al., 2023). In another vein, domain generalisation has always been seen as a key weakness of IR models (Thakur et al., 2021). Through the subpar performance of general image-text models on SciMMIR, we evidence that scientific IR, especially when multi-modal, remains an out-of-
Retrieval Model
Caption: Product/Service Codes Classification Methodology Caption: Synchronization order parameter S is depicted as the heat map ……
Caption: Observed fk versus z for the multiplylensed systems that serve as constraints on .….
Fig Result
Caption: Results comparison of the NTU RGB+D dataset under a well-established fall detection backbone, ….. Caption: The galario best-fit parameters of the gaussian plus power law model, for each ALMA Band
Table ParameterTable Result
Task1: Txt -> Img Input: Product/Service Codes Classification Methodology
Retrieval Model
Task2: Img -> Txt Input:
The galario best-fit parameters of the gaussian plus power law model …..
Fig Architecture
Figure 1: An illustration of the SciMMIR framework.
domain (OOD) task despite advancements in general information retrieval.
Multi-modal Information Retrieval. In earlier multi-modal representation learning research, small-scale cross-modal retrieval datasets including MSCOCO (Lin et al., 2014) and Flickr30k (Plummer et al., 2015) have facilitated the alignment between visual and linguistic representations. Efforts have since shifted towards large-scale visionlanguage pretraining (Radford et al., 2021; Kim et al., 2021; Li et al., 2021; Jia et al., 2021; Yu et al., 2022), with these small-scale retrieval datasets, in turn, becoming the standard evaluation approach for such systems. Advancements in multi-modal representation alignment have also facilitated multimodal retrieval-augmented generation (Chen et al., 2022; Yasunaga et al., 2022; Hu et al., 2023; Lin et al., 2023), and more recently, evaluating the unified cross-modal representations across diverse tasks has emerged as a prevalent trend (Wei et al., 2023).
Scientific Document Retrieval. Scientific information retrieval has received moderate attention in NLP, with SciFact (Wadden et al., 2020) and SCIDOCS (Cohan et al., 2020) commonly incorporated in popular zero-shot information retrieval benchmarks (Thakur et al., 2021). More complex tasks have been proposed in this area, such as DORISMAE, a task to retrieve documents in response to complex, multifaceted scientific queries (Wang et al., 2023a). In the multi-modal area, VQA (Antol et al., 2015) presents another major approach in evaluating vision-language systems, concerning
in-depth visual grounding, rather than the use of distributional priors (Agrawal et al., 2018). It is in this area that work with a similar scope to ours in the scientific domain, such as PlotQA (Methani et al., 2020) and ChartQA (Masry et al., 2022), is seen. Our proposed SciMMIR benchmark distinguishes itself from these existing works by offering extensive coverage across annotations of figure and table subcategories, a larger dataset size, and the use of real-world data that is naturally paired and therefore not reliant on costly human annotation.

heading: 3 Dataset Construction
text: Data Collection. We collect PDF files from a 6-month period (i.e. papers submitted between May and October 2023) from arXiv using the official API2. We use an open-source tool (Clark and Divvala, 2016) to locate non-textual elements (i.e., figures and tables) in the papers and extract their corresponding caption text. All tables and figures are stored in the form of images, and we remove the figure/table entries that have empty captions. The aforementioned collection process results in
2https://info.arxiv.org/help/api
the SciMMIR dataset that comprises 530K imagecaption samples, with an average caption length of 43.19 words as shown in Table 1. The dataset is split into training, validation, and testing sets with 498, 279, 16, 433, and 16, 263 samples, respectively. As shown in Figure 2, the SciMMIR benchmark covers a wide range of scientific disciplines, including those that require complex reasoning (such as Mathematics, Physics, and Computer Science), which attests to the presence of comprehensive and intricate scientific knowledge within the dataset.
Subset and Subcategory Structure. To better understand the performance of VLMs across various data types within the scientific domain, we define a hierarchical architecture with two subsets and five subcategories for the SciMMIR benchmark. We divide the data into two subsets: one for tables and one for figures, as they possess distinct data distributions. Tables contain ample textual information, whereas figures predominantly utilise geometric shapes to elucidate scientific principles or reveal patterns within data. For tabular data, we further categorise them into two subcategories, Table-Parameter and Table-Result. TableResult data primarily presents experimental results, whereas Table-Parameter data provides explanations of parameters or specific numerical values (i.e., learning rates and physical coefficients), and consequently both have different data type distributions. As for Figures, we consider those depicting
experimental results, explaining model architectures (e.g. a figure describing each module in a deep learning model), and illustrating scientific theories (e.g. a figure illustrating an event related to double-slit interference, aiming to elucidate the underlying scientific principle), as they encompass different types of scientific knowledge. Therefore, the performance of models on these distinct data types may vary, leading us to categorise them into three separate subcategories. Specifically, our finegrained categorisation is derived based on the statistics in Table 2.
Data Annotation. For data annotation, we use manually constructed key phrases to classify imagetext sample pairs. Firstly, we acquire keywords based on the unique words that emerge in captions under different subcategories and conduct an initial categorisation of the data based on this keyword set. Subsequently, to ensure the quality of our statistical analysis, we randomly select 2000 images from the test set and hire three graduate students experienced in natural language processing to manually review the results of the keyword-based classification on the criteria of whether the image within the imagecaption pairs conform to the expected characteristic of the corresponding subcategory. We then construct new keywords and remove low-quality ones by analysing which words in the caption result in misclassified examples. Finally, we refine the keyword list iteratively, enhancing the quality until the manual evaluation’s accuracy on the 2,000 extracted samples reached 80%. The subset and subcategory classification results are shown in Table 1, providing a structured and standardised basis for subsequent experiments.

heading: 4 Experiment

heading: 4.1 Retrieval Baseline
text: To investigate the capabilities of current VLMs on the SciMMIR task and to assess whether data from different categories influences their performance, we evaluate a wide range of baseline models. Furthermore, we collect information regarding the pretraining strategy for each baseline model in Table 3 and present additional details in Appendix A, in order to explore the potential factors that cause performance differences between VLMs.
Image Captioning Models. As our baselines, we present image-captioning models, including CLIPbase (Radford et al., 2021) and BLIP-base (Li et al., 2022), that have learned the pairing relationship between images and the corresponding text via a strong supervision signal. We evaluate these image captioning models trained on general domain datasets (such as images related to scenery and daily life events) in both zero-shot and finetuned settings to investigate the need for scientific domain adaptation. We also introduce BERT (Devlin et al., 2018) as an alternative text encoder for captioning (denoted "+BERT" in the tables), where such ensemble baselines may reveal the influence of the text encoders.
Visual Language Models. Additionally, we select large visual language models (VLMs) trained for multi-modal tasks such as Visual Question Answering (VQA) to examine their zero-shot and finetuned MMIR performance in the scientific domain. Additional details of the benchmarked VLMs are given in Appendix B.
OCR Based Method. We perform OCR on the images in our SciMMIR benchmark to extract textual content. To improve the performance of VLMs on the SciMMIR task, we combine the OCR text embeddings generated by the text encoder of the VLMs with the image embeddings produced by the VLMs’ visual encoder.

heading: 4.2 Evaluation Protocol
text: Task Definition. The SciMMIR benchmark presents a bi-directional MMIR task:
• txt→img: The forward direction retrieval task, where for a given text, the model retrieves the correct corresponding image from a candidate set. • img→txt: The inverse direction retrieval task, where given an image, the model retrieves the correct corresponding text from a candidate set.
Given an image imgi and a text textj , the relevance score R in the retrieval ranking is defined as the dot product between the visual and textual representations of imgi and textj (i.e. R = Eimgi · Etextj ). In addition to assessing the models’ performance on the overall test set (denoted “ALL”), we evaluate the models’ retrieval capability on different subsets and subcategories to scrutinise their abilities. Specifically, we assess the models’ performance on five fine-grained subcategories (shown in Table 2) of the test set, as well as the performance on the Figure and Table subsets overall.
Metrics. In this paper, we use the Mean Reciprocal Rank (MRR) and Hits@K metrics to assess the
IR models’ performance on the SciMMIR benchmark. These metrics are calculated based on the ranking of the golden answer within the entire set of candidates provided by the IR models. The details of these metrics are described in Appendix D.
Zero-shot We provide a zero-shot (ZS) setting in the evaluation for all baselines. For imagecaptioning models, the features extracted by the visual encoder and textual encoder are directly used, since they have been aligned to the same representation space. For the visual language models, the visual representation remains unchanged, but the representations from the textual module are used depending on their architectures. Specifically, for the encoder-decoder textual models such as BLIP2FLAN-T5s, we use the output features from the textual encoder as the text features, whereas for decoder-only textual models like BLIP2-OPTs, we perform mean pooling on the outputs from the last decoder layer.
Fine-tuning. We also provide an evaluation of fine-tuned (FT) versions of the relatively small models (CLIP-base and BLIP-base) and a large VLM (BLIP2-FLAN-T5-XL) that were trained with our data. During fine-tuning, we employ standard contrastive learning (Chen et al., 2020a) to maximize the relevance score between positive textimage pairs and minimise the relevance score between negative text-image pairs within a batch of samples. In addition to training the models on the entire training set, we also train them on different subsets (e.g., Figure-Result and Table-Parameter) of the training data to investigate the modelling abilities in a fine-grained manner.

heading: 5 Result Analysis

heading: 5.1 Overall Evaluation
text: Following the evaluation protocol shown in Table 4, we report the baseline performances on the universal set (ALL), Figure set, and Table set.
For both the forward (txt→img) and inverse (img→txt) tasks, we find that small models (e.g. CLIP and BLIP) fine-tuned with our in-domain scientific image-text data generally demonstrate superior performance in all settings of the SciMMIR benchmark. Specifically, the MRR of finetuned CLIP and BLIP models are over 6% in all settings. This underscores the necessity of domain adaption for improvement in the SciMMIR task. Our designed tasks remain challenging for most of
the models. For tasks across both directions, the zero-shot capabilities of most large VLMs demonstrate relatively poor performance, with both the MRR and Hits@10 metrics falling below 0.23% in the ALL setting. It is worth mentioning that the CLIP-base model’s zero-shot performance is the best across all VLMs with its MMR being over 0.3%, which suggests CLIP maybe encounter some image-caption pair related to the scientific domain during the pre-training.
The performance of the fine-tuned multi-modal models in information retrieval involving both figures and tables is promising overall. However, in the non-OCR setting, the performance of the models is significantly higher on the Figure subset than on the Table subset, suggesting that the table retrieval task is more challenging. Conversely, when fine-tuning with the OCR-text data, there is not an explicit gap between the models’ performance on the Figure and Table subsets. We hypothesise that the lower performance on the table subset without OCR-text data may be due to the scarcity of table-style images in the pre-training datasets and the lack of textual perception ability in the visual encoders.
Experimental results based on our SciMMIR benchmark demonstrate the limitations of existing VLMs for MMIR in scientific domains. However, by employing the high-quality data of SciMMIR for fine-tuning, the performance of VLMs can be effectively improved. Additionally, our experiments show that retrieving visual tables is challenging and requires thoroughly mining the semantic relations between caption information and textual data within tables.

heading: 5.2 Zero-Shot Analysis
text: To provide a more thorough analysis, we present the zero-shot performance of the baselines across different subcategories in Table 8 and Table 9 in Appendix F.
As for the txt→img direction, the selected large pre-trained VLMs (e.g. BLIP2-OPT-6.7B and LLaMA-Adapter2-7B) demonstrate poor performance across various subcategories in both the Figure and Table subsets. For the subcategories within the Table subset, all models, except CLIP-base, are ineffective. In the Figure subset, the BLIP2-FLANT5 series models show slightly better performance across all subcategories. This may be attributed to the fact that the textual encoder part of encoderdecoder architecture could better capture textual
features. Conversely, as for the img→txt direction, on the Figure subset, the performance of all VLMs in the reverse direction is marginally lower than in the forward direction. This proves that the image feature captured by visual encoder of current VLMs is unable to model effective relation with the relevant text.

heading: 5.3 Analysis in Fine-tuning Setting
text: Overall Analysis. As shown in Table 11 in Appendix E, we fine-tune the models using data of different categories. The results indicate that training the model only with data from a specific subcategory leads to a significant performance gap compared to the models fine-tuned on all the data.
There are two main factors contributing to this. Firstly, the dataset size of a specific subcategory is relatively small. Secondly, there are significant differences in data distributions among different subcategories.
Among all the models, BLIP-base+BERT performs the best across all fine-tuned settings, while the performance of the CLIP model drops when its text encoder is replaced with BERT. Notably, merely fine-tuning the Q-Former parameters of BLIP2-FLAN-T5-XL to adapt the large VLM to the scientific domain did not yield as effective results as the smaller models. Consequently, there remains a need for efficiently fine-tuning small models to construct robust connections between the representations of the visual and textual modalities.
The Impact of Subcategory Training Data. As shown in Table 5 and Table 6, we report the result on testing samples of specific subcategories, for the sake of comprehensively investigating the impact of different subcategory training data.
For BLIP, training on a certain subcategory results in performance improvements on the corresponding part of the test set, whilst its performance on other subcategories remains relatively poor. This demonstrates the distribution gap among our labelled subcategories and proves the rationality of our subset-subcategory hierarchy. As for CLIP, the models trained on different subcategories consistently perform best in the Fig-Architecture subcategory. This may be because CLIP has been trained on data with a more similar distribution to our data.
The model trained on Figure-Results data demonstrates the best performance across the entire Figure subset. One reason could be that the Figure-Result subset has the largest training proportion (54.02%) and contains text documents with a relatively longer average length (52.93 words compared to the dataset’s overall average length of 43.23 words) in the training dataset. This highlights the impact of training dataset size and its length coverage of text (Xiao et al., 2023a) on the performance and generalisability of retrieval models.
The Impact of OCR. After fine-tuning the CLIP model with OCR-extracted text data, we observe a notable improvement on the Table subset compared to the Figure subset. Furthermore, as for the subcategories related to results (i.e., the Table Result and Fig Result subcategories), the VLMs
achieve their best performance, with MRR exceeding 20%, compared to other models and subcategories. These findings indicate that the OCRextracted text data can provide textual information from the images, which may not be completely captured by the VLMs. This underscores the value of incorporating OCR data to enhance the text perception ability of VLMs, particularly in the domain of scientific multi-modal information retrieval.

heading: 5.4 Text Encoder Generalisability
text: To investigate the impact of text encoders on SciMMIR, we substitute the text encoders in both BLIPbase and CLIP-base models with BERT-base. As shown in Table 11 in Appendix E, replacing the text encoder of BLIP with BERT results in a significant improvement, while that of CLIP experiences a decline. The reason for the performance changes after replacing the text encoder with BERT in both CLIP and BLIP may be as follows:
CLIP. With effective contrastive learning (Wang and Isola, 2020), the textual and visual embeddings are well-aligned in an isotropic space in the pretraining phase of CLIP, which is demonstrated by the zero-shot setting experiments. However, replacing the text encoder with a highly anisotropic vanilla text encoder (e.g., BERT) hinders the stable alignment with the already learned vision encoder (Xiao et al., 2023b). We hypothesise that freezing the vision encoder in early fine-tuning may help guide the replaced language model.
BLIP. Unlike CLIP, BLIP incorporates BERT as its text encoder from the pre-training phase. This structural consistency significantly contributes to the model’s enhanced adaptation to the scientific domain. Besides, employing BERT as the text encoder may facilitate the learning of more effective text representations. This is particularly advantageous in establishing associations between images and text, especially since tables, a common element in scientific documents, are rich in textual information.

heading: 5.5 Error Analysis
text: For better analysis of the performance, we conduct experiments on test data spanning different subcategories and calculate the ratio of all subcategories within the top 10 answers predicted by the fine-tuned and vanilla CLIP models. Predictions matching the test subcategory were considered correct.
As shown in Table 7, due to the larger volume of samples in our dataset are labelled as Fig-Results and Table-Results (58.00% and 26.16%, calculated through Table 1, respectively), the models tend to predict samples from these categories as answers. When comparing zero-shot and fine-tuned models, it can be observed that fine-tuning helps reduce incorrect predictions across almost all categories.
Compared with zero-shot results, the fine-tuned models show the largest improvement in prediction accuracy on the Figure-Architecture and FigureResult test data. However, the increase in prediction accuracy on the Table subset after fine-tuning is not obvious, indicating that retrieving information from Tables still poses significant challenges.

heading: 6 Conclusion
text: In summary, we introduce a novel benchmark and a corresponding dataset designed to address the gap in evaluating multi-modal information retrieval (MMIR) models in the scientific domain. Additionally, we categorise the images into fine-grained subcategories based on the characteristics of the figures and tables to facilitate a more comprehensive evaluation and analysis. Our evaluation of zeroshot and fine-tuned approaches, which we conduct on extensive baselines within various subsets and subcategories, offers valuable insights for future research.
Limitations
Due to computational resource constraints, we only fine-tune BLIP2-FLAN-T5-XL on our SciMMIR dataset and did not investigate the fine-tuning effects of other large VLMs on our benchmark. In this work, we find that BLIP+BERT could improve the model’s ability in our benchmark, specifically for the Table subset. However, we do not design experiments to explore which kind of models would be better suited to the replacement of the textual encoder with BERT or other language models.Despite
our best efforts to ensure data quality, given the sheer volume of data, we cannot guarantee that all results and content within the scientific domain dataset are accurate. This inherent limitation could potentially lead to models generating misleading or deceptive outputs in future use, necessitating further filtering in future work.
Ethics Statement
The dataset used in our research is constructed using publicly available data sources, ensuring that there are no privacy concerns or violations. We do not collect any personally identifiable information, and all data used in our research is obtained following legal and ethical standards. In the stage of designing keywords and human evaluation classification of image-text pair, we employed three graduate students experienced in natural language processing for human evaluation. We paid the graduate students approximately $13 per hour, well above the local average wage, and engaged in constructive discussions if they had concerns about the process.

heading: A The Baseline Pre-training Datasets
text: Enhancing model performance through additional knowledge has garnered considerable attention (Cheng et al., 2023a,b,c; Wang et al., 2023b) making it is essential to boost model capabilities in the scientific domain through multi-modal retrieval. Besides, retrieving the related specific domain knowledge could significantly relieve the hallucination of LLM and VLMs (Choi et al., 2023) and improve the interpretability of them (Wang et al., 2022, 2024). To this end, we have designed a series of baselines and provided a reference list for the pre-training image-text datasets mentioned in Table 3. COCO (Lin et al., 2014), consists of over 200,000 images across various categories including people, animals, everyday objects, and indoor scenes. The VG dataset (Krishna et al., 2017) consists of over 100,000 images and covers a diverse range of visual concepts, including objects, scenes, relationships between objects, and other contextual information within images. CC3M (Sharma et al., 2018) contains over 3.3 million of images paired with descriptive captions, covering a wide range of topics and scenes. CC12M (Changpinyo et al., 2021) contains 12.4 million image-text pairs, which is 3 times larger in scale compared to CC3M with a higher diversity degree containing more instances of out-of-domain (OOD) visual concepts. SBU (Ordonez et al., 2011) contains over 1 million images with visually relevant captions. The dataset is designed to be large enough for reasonable image-based matches to a query and the captions are filtered to ensure they are visually descriptive and likely to refer to visual content. LAION400M (Schuhmann et al., 2021) is an open dataset that consists of 400 million image-text pairs, their CLIP embeddings, and KNN indices for efficient similarity search. It includes image URLs, corresponding metadata, CLIP image embeddings, and various KNN indices for quick search. LAION5B (Schuhmann et al., 2022) is an open, large-scale dataset that consists of 5.85 billion image-text pairs, with 2.32 billion pairs in English. COYO (Byeon et al., 2022) is a large-scale dataset containing 747M image-text pairs as well as many other metaattributes to increase the usability to train various models. MMC4 (Zhu et al., 2023) consists of 101.2 million documents with 571 million images interleaved with 43 billion English tokens. It covers a wide range of everyday topics such as cooking, travel, and technology. GRIT (Peng et al., 2023) is
a large-scale dataset of Grounded Image-Text pairs that consists of approximately 91 million images, 115 million text spans, and 137 million associated bounding boxes. DataCamp (Gadre et al., 2023) is a participatory benchmark that focuses on dataset curation for large image-text datasets. It provides a new candidate pool of 12.8 billion image-text pairs. The dataset size in DataComp is a design choice and not predetermined.

heading: B Used Visual Language Models
text: • BLIP-2 (Li et al., 2023) series models use a querying transformer module to address the modality gap. We choose the models grounded in large language models (LLMs), BLIP2-OPT-2.7B, BLIP2-OPT-6.7B, BLIP2FLAN-T5-XL and BLIP2-FLAN-T5-XXL, as our baselines.
• LLaVA-V1.5-7B (Liu et al., 2023) use two simple methods, namely, an MLP cross-modal connector incorporating academic task related data such as VQA to improve the ability of the LLaVA.
• LLaMA-Adapter2-7B (Gao et al., 2023) efficiently fine-tunes additional parameters based on the LLaMA model (Touvron et al., 2023), where the extra expert models further boost its image understanding capability.
• Kosmos-2 (Peng et al., 2023) aligns perception with language and adds the ability to recognise and understand images based on its multi-turn dialogue and reasoning capabilities. Specifically, it achieves the capability of grounding images, allowing it to interact with inputs at the object level.
• mPLUGw-OWL2 (Ye et al., 2023) introduces a Modality-Adaptive Module (MAM) into the large language model. By adding a small number of parameters during the attention process, it further learns a shared space for both vision and language representations.

heading: C Effects of Visual Encoder Resolution
text: In Table 4 for overall results, we compare the finetuned BLIP with the default image preprocessing dimensions of 384 and fine-tuned CLIP with the default image preprocessing dimensions of 224, where the results are relatively close. To make a fairer comparison, we decrease the image dimensions of BLIP-base model from 384 to 224 to be
the same as CLIP-base to conduct SciMMIR evaluation, as described in Table 10.
It can be seen that the granularity of image processing has a significant impact on model performance. When using a lower preprocessing dimension, the performance of BLIP is significantly decreased in both txt→img and img→txt tasks, using all training data settings. The performance of the CLIP model, which uses the same image processing dimension, is almost double that of BLIP.
Furthermore, although replacing the text encoder of BLIP with BERT during training on lowerdimensional (224) image preprocessed data im-
proved the performance of the model, there was still a significant gap compared to CLIP. However, when the text encoder of BLIP was replaced with BERT during training on higher-dimensional image preprocessed data, the performance of the model was far superior to both CLIP and CLIP+BERT. This suggests that certain image-text shared interactive information is stored in the visual representations, and higher image quality can help the models better establish the connection between image and text representations.

heading: D MRR and Hit@K
text: • MRR stands for Mean Reciprocal Rank and is calculated as the reciprocal of the golden label’s ranking in candidates. A higher MRR score indicates better performance.
• Hits@K assesses the accuracy of the retrieval system by checking whether the golden label is present within the top-k ranked results. Hits@10 are used in our measurements.

heading: E Fine-tuning Analysis
text: The effect of text-image matching task. As shown in Table 11, the BLIP-2 series of models outperform other large VLMs in both Figure and Table subcategories, especially in the forward direction task. We believe that this is because BLIP-2 incorporates the text-image matching task and the image-grounded text generation task during its pretraining process to better align textual and visual information. The experimental results demonstrate that other models solely relying on image-grounded text generation tasks may not yield effective representations for multi-modal retrieval. Therefore, dedicated pre-training for multi-modal retrieval still requires a primary focus on the text-image matching task.

heading: F Zero-shot Analysis
text: CLIP-base and BLIP-base. As shown in Table 8 and Table 9, the CLIP-base captioning baseline, which is specifically designed for image-text matching, shows certain generalisability in both forward and inverse retrieval across all subcategories within the Figure and Table subsets. In contrast, the BLIP-base model shows nearly no signs of effective learning on the scientific domain multi-modal
data. These models have strong MMIR abilities for generic topic data, such as BLIP achieving an IR@1 of 86.7% on the Flicker dataset in the zeroshot setting, whilst BLIP does not surpass 0.05% MMR. This further demonstrates the challenges presented for MMIR in scientific domains.

heading: Future Work
text: Our evaluation of zeroshot and fine-tuned approaches, which we conduct on extensive baselines within various subsets and subcategories, offers valuable insights for future research. Limitations
Due to computational resource constraints, we only fine-tune BLIP2-FLAN-T5-XL on our SciMMIR dataset and did not investigate the fine-tuning effects of other large VLMs on our benchmark. In this work, we find that BLIP+BERT could improve the model’s ability in our benchmark, specifically for the Table subset. However, we do not design experiments to explore which kind of models would be better suited to the replacement of the textual encoder with BERT or other language models. Despite
our best efforts to ensure data quality, given the sheer volume of data, we cannot guarantee that all results and content within the scientific domain dataset are accurate. This inherent limitation could potentially lead to models generating misleading or deceptive outputs in future use, necessitating further filtering in future work