AISciVision: A Framework for Specializing Large Multimodal Models in Scientific Image Classification

Authors: Brendan Hogan Rappazzo, Anmol Kabra, Felipe Siqueira Pacheco, Laura Greenstreet, Joshua Fan, Aaron M Ferber, MARTA EICHEMBERGER UMMUS, Alecsander Gomes Brito, Olivia J. Graham, Lillian R. Aoki, C. Drew Harvell, Alexander Flecker, Carla P Gomes
Primary area: applications to physical sciences (physics, chemistry, biology, etc.)
Keywords: computer vision, large language models, scientific application, retrieval augmented generation, agent systems

Abstract:
Trust and interpretability are crucial for the use of Artificial Intelligence (AI) in scientific research, but current models often operate as black boxes offering limited transparency and justifications for their outputs. Motivated by this problem, we introduce AiSciVision, a framework that specializes Large Multimodal Models (LMMs) into interactive research partners and classification models for image classification tasks in niche scientific domains. Our framework uses two key components: (1) Visual Retrieval-Augmented Generation (VisRAG) and (2) domain-specific tools utilized in an agentic workflow. To classify a target image, AiSciVision first retrieves the most similar positive and negative labeled images as context for the LMM. Then the LMM agent actively selects and applies tools to manipulate and inspect the target image over multiple rounds, refining its analysis before making a final prediction. These VisRAG and tooling components are designed to mirror the processes of domain experts, as humans often compare new data to similar examples and use specialized tools to manipulate and inspect images before arriving at a conclusion. Each inference produces both a prediction and a natural language transcript detailing the reasoning and tool usage that led to the prediction. We evaluate AiSciVision on three real-world scientific image classification datasets: detecting the presence of aquaculture ponds, diseased eelgrass, and solar panels. Across these datasets, our method outperforms fully supervised models in low and full-labeled data settings. AiSciVision is actively deployed in real-world use, specifically for aquaculture research, through a dedicated web application that displays and allows the expert users to converse with the transcripts. This work represents a crucial step toward AI systems that are both interpretable and effective, advancing their use in scientific research and scientific discovery.

Decision: reject
Reviewer scores: 3, 6, 6, 3

Reviews and discussion:
<thread>
creation date:
1729351290188

thread_type:
official_review

<thread_object>
creator: reviewer_cxn6

<content>
summary:
This paper proposes a method that uses an LMM, such as ChatGPT, in combination with VisRAG and domain-specific tools to classify scientific images. At test time, given an image, the system retrieves similar images and prompts the LMM to utilize the tools (explaining how to use each one) to better understand and then classify the image. In addition to the classification result, the method also provides the confidence score and the conversation with the LMM to ensure transparency for the user.

strengths:
- The authors try to increase the interpretability of classification task
- Make use of LMMs to solve real world problems
- Does not require any additional model training

weaknesses:
- Comparing new data to similar data carries a high risk of introducing bias into the process.
- Defining domain-specific tools is challenging, as it requires domain experts, and even experts may overlook some tools.
- Using multiple rounds of prediction increases both inference time and cost.
- Describing domain-specific tools in natural language is not always straightforward.
- The testing datasets are very small and may not cover a broad range of test samples.

questions:
- How do you use a hardcoded prompt template for the response?
- How does your model output a probability score indicating its confidence?
- How do you determine how many conversation turns are enough?
- In the paper, you state: "We provide text descriptions of the available tools as prompts to the LMM, and instruct how to request tool use" How do you ensure that the LMM reliably follows your instructions behind the scenes? We know LMMs can sometimes misunderstand prompts, requiring users to read the answer, refine their prompt, and ask again.
- In the paper, you mention: "You ask the LMM to have 4 conversation turns." How do you determine that this number (4) is appropriate?
- In the paper, you state: "All embeddings for VisRAG are computed via CLIP." How do you ensure that these images are not out of distribution for CLIP?
- In the paper, you claim that "AISciVision consistently outperforms all baselines on all three datasets." What about the descriptive text your framework returns? Did you conduct any analysis on the quality of those texts?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730235748066

thread_type:
official_review

<thread_object>
creator: reviewer_q2eg

<content>
summary:
The paper introduces a method for scientific image classification using Visual RAG and external tools with Large Multimodal Models (LLMs)/. For classification, the proposed method tries to mimic an 'expert human' by first retrieving relevant examples from the training to to compare and then using different tools to aid in the classification. They show the effectiveness of their method on 3 real world datasets and also show the reasoning ability through the multi-round dialog component. They also claim that the model is deployed as a web application and is being used by ecologists and scientists.

strengths:
1. A very creative application of a) Visual RAG, b) utilizing external tools and 3) multi-round dialog with LMMs for scientific image classification.

2. The fact that the model is actively deployed and is used by ecologists for aquaculture research is quite impressive.

weaknesses:
1. This kind of framework of using Visual RAG + external tools with Large models is not that novel. However, I do agree that their application in scientific images is quite novel.

2. The method provides reasoning of its predictions through it's transcripts during the multi-round dialog process. Although the transcript makes the model more interpretable and diverse, there is no verification/human evaluation of the correctness of the model's reasoning. A human evaluation study of the correctness of the reasoning would have been great.

questions:
1. The method says that the most similar positive and negative images are retrieved for a test image. However, in the example transcripts, I can't seem to find where this step is happening. Could you explain how this retrieval is happening through an example?

2. The naive baseline of k-nn itself is very strong baseline. Did you find any particular reason for that?

3. It is said that the evaluation are done in two settings: low label (20%) and full labeled (100%). Where are these labels used? to train CLIP+MLP? or to retrieve +ve and -ve examples during Visual RAG?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730531887882

thread_type:
official_review

<thread_object>
creator: reviewer_fk76

<content>
summary:
The paper presents an approach for interpretable image classification tailored to scientific domains. It introduces a framework that enables Large Multimodal Models (LMMs) to perform more accurate and explainable reasoning by integrating Visual Retrieval-Augmented Generation (VisRAG) and domain-specific tool usage. This approach can be effective for niche scientific tasks involving binary image classification, where transparency and understanding of the decision-making process are essential.

strengths:
1.	Ablations to understand importance of VisRAG and Tools
2.	Exact prompts used were provided making reproduceability easier

weaknesses:
1.	Compute heavy multi turn conversation.
2.	Limited to binary classification: The approach in its current form is limited to binary classification. Extending the approach to multiclass classification do not appear to be straightforward. If the authors can possible ways to extend beyond binary classification that would be helpful.
3.	Bias from the prediction of supervised model. As discussed in the supplementary, the framework can be heavily influenced by the supervised model.

questions:
Some suggestions and questions,

1.	It would be interesting to perform an experiment without supervised model in the toolset, considering that the LMM can be heavily influenced by the model.
2.	There is no mention of the train test split for each dataset. It can be relevant information
3.	There is a drop in performance of kNN going from 20% to 100% on Aquaculture dataset. Any explanation for why this is happening?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730716790126

thread_type:
official_review

<thread_object>
creator: reviewer_bmj1

<content>
summary:
The paper introduces AISciVision, a framework designed to enhance the capability of large multimodal models (LMMs) for scientific image classification tasks. The framework comprises two main components: (1) Visual RAG (or VisRAG), which retrieves contextually similar positive and negative labeled images from the training set that serves as an initial prompt to the LMM or VLM model along with the test image that needs to be classified. (2) AI agentic workflow that engages with domain-specific tools in a multi-round conversation to classify input test image. The goal is to accurately classify the image and also generate transparent reasoning transcripts that detail the decision-making process. The paper also introduces a web application for real-time aquaculture pond detection.

strengths:
- The paper is well-written and easy to understand.
- The two-stage process of first using VisRAG followed by an interactive AI agentic workflow with domain-specific tools is innovative and well-integrated.
- The framework demonstrates applicability across different domains and generates reasoning transcripts for enhanced interpretability of image classification.
- The paper introduces a web application for aquaculture pond detection which can serve as a data collection tool to build better algorithms based on user feedback.

weaknesses:
- The paper could benefit from a stronger motivation section that elaborates on the significance of the selected scientific classification tasks and their scientific impact. While generating reasoning transcripts is valuable, the specific implications of correct classification and the benefit of LMMs/VLMs over supervised classification models are still unclear.
- Although the authors claim AISciVision is model-agnostic, experiments are limited to GPT-4o. Including evaluations with other LMMs/VLMs, such as LLaVA [A], BLIP [B], etc. would strengthen this claim and demonstrate broader applicability.
- The paper asserts that it outperforms several fully supervised models; however, Table 1 displays a weak baseline of k-NN as a supervised approach. While CLIP + MLP is also included as a supervised baseline, it performs similarly to, and in some cases better than, AISciVision on two of the three datasets (Eelgrass and Solar). To more effectively justify the advantages of AISciVision, it would be helpful to include stronger and more diverse supervised baselines.
- The Tools for the Aquaculture dataset leverage additional geospatial location metadata and utilize Google Maps APIs to fetch images of areas beyond the input test image. Depending on the selected tool—such as panning left/right/up/down, or zooming out—these tools can access a wider context. This makes it an unfair comparison for baseline models that solely rely on the input images and have not seen the images of the surrounding area.
- The domain-specific tools form a significant part of the AISciVision framework but most of the details are missing in the manuscript and are discussed in the appendix.




**REFERENCES**

[A] Liu, Haotian, et al. "Visual instruction tuning." Advances in neural information processing systems 36 (2024).

[B] Li, Junnan, et al. "Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation." International conference on machine learning. PMLR, 2022.

questions:
- The inclusion of both positive and negative images as initial prompts is interesting. What specific advantage does this approach offer? How would the model’s performance change if only one positive or negative image is provided?
- Table 2 provides ablations with GPT-4o and different components of AISciVision.In a zero-shot setting, GPT-4o demonstrates strong performance across all datasets. For the Eelgrass and Solar datasets, the majority of performance improvements are attributed to the combination of GPT-4o and VisRAG. What additional benefits do these tools provide in these scenarios?
- In the aquaculture dataset, the most notable improvements come from using GPT-4o along with tools that utilize the Google Maps API and supplemental images. How does the model perform when it is limited to tools similar to those used for the Eelgrass and Solar datasets?
- For the aquaculture dataset, when using images of surrounding areas through the Google Maps API, how do you account for changes in the landscape over time? How does this affect the analysis?
- Why do the initial user prompts to the LMM differ for various datasets when providing positive and negative images?
- Why is the model encouraged to use at least three tools? Does performance vary with fewer or more tools?
- What are the benefits of using VisRAG as a standalone component? Could it be incorporated as one of the tools within the AI agent's workflow?
- How are the domain-specific tools chosen? The same tools are used for two different datasets: the Eelgrass dataset, which contains plant images, and the Solar dataset, made up of satellite images. Why do we refer to them as domain-specific tools?
- The AISciVision framework for Aquaculture pond detection relies on geospatial metadata. What happens if a user does not have the geospatial location metadata for a specific test image while using the web application? Is the framework adaptable for datasets that lack this metadata, or are there alternative features that can be utilized?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732064799511

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Rebuttal

comment:
We thank the reviewers for their thoughtful feedback and for engaging deeply with our work. We are particularly pleased that the reviewers found our approach creative (q2eg), well-written (BMJ1), innovative (BMJ1) and an effective application of LMMs to real-world scientific problems (q2eg, fK76, cXN6). The feedback has provided many valuable suggestions for improving both the clarity and comprehensiveness of our work. We have addressed the raised concerns and are excited to share several significant updates:

**Additional Baselines and Models**
* We now include evaluations with additional LMMs/VLMs - Claude-3.5-Sonnet [1] and Llama-3.2-11B [2,3], to demonstrate the model-agnostic nature of AISciVision (Table 4 in Appendix E), as suggested by reviewer BMJ1. Additional experiments followed expected patterns across LMMs: VisRAG and domain-specific tools deliver complementary benefits in performance metrics with any choice of LMM (Table 5 in Appendix E with Llama-3.2-11B). These experiments validate the efficacy of our proposed AISciVision framework. 
* We have also added a Resnet50 baseline across all datasets (Table 1), addressing BMJ1's concern about the need for stronger supervised baselines. While Resnet50 performed well, particularly on the eelgrass dataset, AISciVision with GPT-4o outperformed it for aquaculture and solar datasets. Resnet50 particularly struggled because of the high class imbalance and difficulty of these datasets [4] - highlighting the strength of our framework for scientific image classification.
* Reviewer BMJ1 also raised the valid concern that AISciVision might have an unfair advantage on the aquaculture dataset because it accesses additional satellite data through its tools. We view the ability to flexibly access additional data to be a key strength of AISciVision. While we can devise model-specific approaches and training paradigms to select how to query additional data, AISciVision readily handles this in a model-agnostic manner. Nevertheless, it is an interesting question to understand how a standard neural classifier performs when given additional data. To address this, we conducted an experiment where Resnet50 was given the same set of images accessed by our method and made predictions via a majority vote (Table 7 in Appendix E). Even under this fairer comparison, AISciVision outperformed Resnet50, highlighting the robustness of our framework. We acknowledge that accessing additional data could be seen as an advantage, and this is a question we grappled with internally. On one hand, incorporating more context could be viewed as giving our method an edge; on the other hand, the ability to intelligently query and integrate multiple data sources is a fundamental strength of AISciVision, designed to address the complexities of scientific classification. We fully agree with the reviewer’s concern and have now included results under a fairer comparison to address this point.

**Additional Experiments**
We performed several new ablations to further analyze AISciVision's components:
* VisualRAG Ablation: As suggested by BMJ1 and q2eg, we ran experiments returning only a single positive or single negative image in the VisualRAG process (Table 6 in Appendix E). These experiments revealed that using both positive and negative examples, as in our full method, yields the best results, emphasizing the complementary value of both.
* Aquaculture Dataset Without Tools: Addressing fK76's concern about supervised model reliance, we evaluated the aquaculture dataset without the supervised tool and, separately, without satellite tools like Google Maps (Table 8 in Appendix E). Both removing the supervised tool, and removing the satellite tools resulted in a slight performance dip - consistent with our expectation.

**All additional experiments can be found in Appendix E.**

**Anonymized Open Source Code**

We share an anonymized version of our code here: https://anonymous.4open.science/r/AiSciVision-B2B5/

**References**

[1] https://www-cdn.anthropic.com/fed9cc193a14b84131812372d8d5857f8f304c52/Model_Card_Claude_3_Addendum.pdf 

[2] Touvron, Hugo, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, et al. ‘LLaMA: Open and Efficient Foundation Language Models’. arXiv [Cs.CL], 2023. arXiv. http://arxiv.org/abs/2302.13971.

[3] https://www.llama.com/docs/model-cards-and-prompt-formats/llama3_2/#-llama-3.2-vision-models-(11b/90b)-  

[4] Johnson, Justin M., and Taghi M. Khoshgoftaar. "Survey on deep learning with class imbalance." Journal of Big Data (2019)
</content>
</thread_object>

</thread>


Metareview:
metareview:
The AC appreciates the authors' detailed responses and the exploration of applying the AISciVision framework to new domains. However, the paper falls short in several critical areas. First of all, AC questions the usefulness of focusing on classification as the task. AC also recognizes that the domain-specific tools used for the eelgrass and solar panel datasets are identical, undermining the claim of domain specificity. For the aquaculture domain, the reliance on Google Maps APIs to provide additional images does not convincingly demonstrate domain-specificity.  Additional issues include the high risk of bias when comparing new data to similar datasets, the challenges in defining domain-specific tools without sufficient expert input, and the added inference time and cost caused by multiple prediction rounds, etc. etc.  Given these substantial limitations, and AC shares strong negative views with two reviewers, the submission does not meet the standards for acceptance.